Storage settings

Streaming Data Manager deploys Prometheus with the following default storage settings. These settings create a Persistent Volume of 10Gi for Prometheus to store historical metrics. Prometheus keeps historical records for 10 days (retention: 10d) or 9.5GB (retentionSize: 9.5GiB), whichever occurs first.

prometheusSpec:
  retentionSize: 9.5GiB
  retention: 10d
  storageSpec:
    volumeClaimTemplate:
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi

Customize Prometheus storage settings

You can override the default Prometheus storage settings using the Streaming Data Manager ApplicationManifest CR. For example:

prometheusOperator:
     ...
     valuesOverride: |-
       prometheus:
         prometheusSpec:
           retentionSize: 19GiB # should be less than storage size, for example: storage size * 0.95
           retention: 10d
           storageSpec:
             volumeClaimTemplate:
               spec:
                 accessModes:
                  - ReadWriteOnce
                 resources:
                   requests:
                     storage: 20Gi

Note: The Prometheus operator doesn’t support resizing an existing storage out of the box.

Change storage size of an existing Prometheus deployment

To change storage size of an existing Prometheus deployment, complete the following steps.

  1. Make sure that the storage class you use has allowVolumeExpansion: true

  2. Update the storage settings in the ApplicationManifest custom resource.

  3. Reconcile with Streaming Data Manager.

  4. Update the size of the PersistentVolumeClaim (PVC) used by Prometheus to match what you specified in the ApplicationManifest custom resource (for example, 20Gi).

  5. Verify that the size of the PVC has been updated:

    kubectl get pvc -n supertubes-system -lapp=prometheus