Set up High Availability for the Monitoring Stack
To ensure that Prometheus is always available, enable the High Availability mode of Service Mesh Manager. That way the monitoring stack uses multiple Prometheus instances and Thanos Query to ensure metric consistency and high availability. Complete the following steps.
Enable high availability. Service Mesh Manager is controlled by a
ControlPlanecustom resource found in the Service Mesh Manager’s namespace (default:
smm. The following commands change the
true, indicating that you want to have a Highly Available deployment.
Run the following commands:
cat > enable-ha.yaml <<EOF spec: smm: highAvailability: enabled: true EOF kubectl patch controlplane --type=merge --patch "$(cat enable-ha.yaml)" smm
If you are using Service Mesh Manager in operator mode, skip this step.
Otherwise, execute a reconciliation so Service Mesh Manager updates your Kubernetes cluster to the desired state described by the ControlPlane Custom Resource. Run the following command:
smm operator reconcile
Verify that High Availability is enabled. To verify that Prometheus is running in Highly Available mode, check the pods running in the
smm-systemnamespace for multiple Prometheus instances. Run the following command.
kubectl get pods -l app=prometheus -n smm-system NAME READY STATUS RESTARTS AGE prometheus-smm-prometheus-0 5/5 Running 1 21m prometheus-smm-prometheus-1 5/5 Running 1 21m
If two running pods are present then the High Availability setup has been completed.