Streaming Data Manager deploys Prometheus with the following default storage settings. These settings create a Persistent Volume of 10Gi for Prometheus to store historical metrics. Prometheus keeps historical records for 10 days (retention: 10d) or 9.5GB (retentionSize: 9.5GiB), whichever occurs first.
prometheusSpec: retentionSize: 9.5GiB retention: 10d storageSpec: volumeClaimTemplate: spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi
Customize Prometheus storage settings
You can override the default Prometheus storage settings using the Streaming Data Manager ApplicationManifest CR. For example:
prometheusOperator: ... valuesOverride: |- prometheus: prometheusSpec: retentionSize: 19GiB # should be less than storage size, for example: storage size * 0.95 retention: 10d storageSpec: volumeClaimTemplate: spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi
Note: The Prometheus operator doesn’t support resizing an existing storage out of the box.
Change storage size of an existing Prometheus deployment
To change storage size of an existing Prometheus deployment, complete the following steps.
Make sure that the storage class you use has allowVolumeExpansion: true
Update the storage settings in the ApplicationManifest custom resource.
Reconcile with Streaming Data Manager.
Update the size of the PersistentVolumeClaim (PVC) used by Prometheus to match what you specified in the ApplicationManifest custom resource (for example, 20Gi).
Verify that the size of the PVC has been updated:
kubectl get pvc -n supertubes-system -lapp=prometheus