Use an existing Alertmanager
SLO-based alerting requires you to configure Service Mesh Manager to use an Alertmanager deployment. This procedure describes how to configure Service Mesh Manager to use an existing Alertmanager deployment.
Note: If you don’t have and existing Alertmanager deployment, or you want to use a separate deployment, see Deploy a new Alertmanager.
Prerequisites
Prometheus Alertmanager installed and configured.
Note: We recommend configuring alert grouping in the following way:
route: receiver: 'slack-notifications' group_by: [ service, severity ]
That way you can route notifications based on severity, using the
severity
label of Service Mesh Manager-generated alerts.
Steps
To configure Service Mesh Manager to use an existing Alertmanager deployment, complete the following steps.
-
Download the following configuration snippet as
enable-alertmanager.yaml
. -
Replace the
your-alert-manager-X-host
part with your Alertmanager’s fully qualified domain name, and theyour-alert-manager-X-port
with the port Alertmanager is listening on. -
Service Mesh Manager is controlled by a
ControlPlane
custom resource found in the Service Mesh Manager’s namespace (default:smm-system
) namedsmm
.The following command changes the
spec.smm.prometheus.alertmanager
value to connect to the existing Alertmanagers. Run the following command:kubectl patch controlplane --type=merge --patch "$(cat enable-alert-manager.yaml)" smm
If you are using Service Mesh Manager in operator mode, skip this step.
Otherwise, execute a reconciliation so Service Mesh Manager updates your Kubernetes cluster to the desired state described by the ControlPlane Custom Resource. Run the following command:
smm operator reconcile
- In case your Alertmanagers are not part of your service mesh setup, create destination rules in the
smm-system
namespace to allow communication with your Alertmanagers.