Set up Persistent Volumes for Prometheus

Each Prometheus instance needs a persistent volume (PV) to store the recorded metrics.

To set up such persistent volumes, modify the Service Mesh Manager control plane custom resource to allow for acquiring such volumes. Complete the following steps.

Find which storage classes does your Kubernetes cluster support. Log in to your cluster, then run the following command:

kubectl get storageclass                                    (kubernetes-admin@turip-host/smm-system)
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  7h17m

Select the appropriate storage class from the output list.

The previous example shows the default AWS settings for EKS or PKE clusters, in this case it is safe to use gp2 volumes.

Set up the Persistent Volume.

To enable Persistent Volumes for Prometheus, first customize this YAML patch file:

spec:
  smm:
    prometheus:
      enabled: true
      storage:
        apiVersion: v1
        kind: PersistentVolumeClaim
        spec:
          storageClassName: gp2
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: "50Gi"

Save this patch file as persistent-storage.patch.yaml.
Enter the Storage Class name obtained in the previous step into the storageClassName property, and set the storage space required by updating the value of spec.smm.prometheus.storage.spec.resources.requests.storage.

Service Mesh Manager is controlled by a ControlPlane custom resource found in the Service Mesh Manager’s namespace (default: smm-system) named smm. To update the Custom Resource with the patch, run the following command.
```
kubectl patch controlplane --type=merge --patch "$(cat persistent-storage.patch.yaml)" smm
controlplane.smm.cisco.com/smm patched
```
If you are using Service Mesh Manager in operator mode, skip this step.

Otherwise, execute a reconciliation so Service Mesh Manager updates your Kubernetes cluster to the desired state described by the ControlPlane Custom Resource. Run the following command:
```
smm operator reconcile
```

Verify that Persistent Volumes exist. To verify that Prometheus has the physical volumes attached, run the following command:

kubectl get pv -n smm-system
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                                                   STORAGECLASS   REASON   AGE
pvc-79cc1a86-bcfa-46a9-a0e7-24e38d001d29   50Gi       RWO            Delete           Bound    smm-system/prometheus-smm-prometheus-db-prometheus-smm-prometheus-0   gp2                     7m10s

The list shows that a Physical Volume (named smm-system/prometheus-smm-prometheus-db-prometheus-smm-prometheus-0 in the example) has been created and attached (Bound status).

Next steps

If you haven’t done yet, Set up High Availability for the Monitoring Stack.