Use an existing Alertmanager

SLO-based alerting requires you to configure Service Mesh Manager to use an Alertmanager deployment. This procedure describes how to configure Service Mesh Manager to use an existing Alertmanager deployment.

Note: If you don’t have and existing Alertmanager deployment, or you want to use a separate deployment, see Deploy a new Alertmanager.

Prerequisites

Prometheus Alertmanager installed and configured.

Note: We recommend configuring alert grouping in the following way:

   route:
     receiver: 'slack-notifications'
     group_by: [ service, severity ]

That way you can route notifications based on severity, using the severity label of Service Mesh Manager-generated alerts.

Steps

To configure Service Mesh Manager to use an existing Alertmanager deployment, complete the following steps.

  1. Download the following configuration snippet as enable-alertmanager.yaml.

    
    
  2. Replace the your-alert-manager-X-host part with your Alertmanager’s fully qualified domain name, and the your-alert-manager-X-port with the port Alertmanager is listening on.

  3. Service Mesh Manager is controlled by a ControlPlane custom resource found in the Service Mesh Manager’s namespace (default: smm-system) named smm.

    The following command changes the spec.smm.prometheus.alertmanager value to connect to the existing Alertmanagers. Run the following command:

    kubectl patch controlplane --type=merge --patch "$(cat enable-alert-manager.yaml)" smm
    
  4. If you are using Service Mesh Manager in operator mode, skip this step.

    Otherwise, execute a reconciliation so Service Mesh Manager updates your Kubernetes cluster to the desired state described by the ControlPlane Custom Resource. Run the following command:

    smm operator reconcile
    
  1. In case your Alertmanagers are not part of your service mesh setup, create destination rules in the smm-system namespace to allow communication with your Alertmanagers.