Upgrading SMM and SDM

The procedure to upgrade Service Mesh Manager depends on whether you have installed Service Mesh Manager in imperative mode or in operator mode.

  • If you have installed Service Mesh Manager in imperative mode, upgrade it using the CLI.
  • If you have installed Service Mesh Manager in operator mode, upgrade the operator.
  • If you have installed Service Mesh Manager using our GitOps guide, upgrade the operator chart.

CAUTION:

Supported upgrade paths

Service Mesh Manager supports upgrades from the prior minor release and patch releases. The current supported upgrade path: v1.10.x to v1.11.x

Before upgrading

If you have cert-manager installed on your Service Mesh Manager cluster, optionally complete the following step.

Before upgrading Service Mesh Manager 1.10 to 1.11, apply the following patch to your Service Mesh Manager v1.10 cluster to modify the spec field of a job that cleans up the cert-manager-startupapicheck job after 100sec when completed. If you skip this step, you might see a “cert-manager-startupapicheck” related error during the upgrade. The error is non-blocking and doesn’t stop the upgrade process. Alternatively, you can apply the patch after you have upgraded the cluster.

kubectl patch jobs.batch -n cert-manager cert-manager-startupapicheck -p '{"spec":{"ttlSecondsAfterFinished":100}}' --type=merge

Using the CLI

In case your Service Mesh Manager deployment is managed using the Service Mesh Manager CLI it should be used to upgrade to the new version.

For an example of upgrading Service Mesh Manager from 1.10.0 to 1.11.0 in a multi-cluster setup, see Multi-cluster upgrade from 1.10.0 to 1.11.0.

  1. Download the Service Mesh Manager command-line tool for version 1.11.0. The archive contains the smm and supertubes binaries. Extract these binaries and update your local copy on your machine. For details, see Accessing the Service Mesh Manager binaries.

  2. Deploy a new version of Service Mesh Manager.

    The following command upgrades the Service Mesh Manager control plane. It also installs the new Istio control plane (version 1.15.x), but the applications keep using the old control plane until you restart your workloads.

    In the following examples, smm refers to version 1.11.0 of the binary.

    • If you want to upgrade only Service Mesh Manager:

      smm install -a
      
    • If you want to upgrade both Service Mesh Manager and Streaming Data Manager use below command

      smm install -a --install-sdm
      
      • In case you want to have custom settings for your Istio control plane, you can provide that during the installation:

        smm install -a --istio-cr-file <custom-istio-cr-file.yaml>
        
  3. Check that the Service Mesh Manager control plane is upgraded and already uses the new Istio control plane.

    • If you are upgrading only Service Mesh Manager upgrade, run the following command to verify that the installation is complete.

      kubectl get pods -n=smm-system -L istio.io/rev
      

      The output should be similar to:

      NAME                                              READY   STATUS    RESTARTS   AGE   REV
      istio-operator-v113x-64bc574fdf-mdtwj             2/2     Running   0          21m
      istio-operator-v115x-8558dbb88c-6r6fx             2/2     Running   0          21m
      mesh-manager-0                                    2/2     Running   0          21m
      prometheus-node-exporter-76jwv                    1/1     Running   0          18m
      prometheus-node-exporter-ptbwk                    1/1     Running   0          18m
      prometheus-node-exporter-w86lc                    1/1     Running   0          18m
      prometheus-smm-prometheus-0                       4/4     Running   0          19m   cp-v115x.istio-system
      smm-6b5575474d-l88lg                              2/2     Running   0          19m   cp-v115x.istio-system
      smm-6b5575474d-wp727                              2/2     Running   0          19m   cp-v115x.istio-system
      smm-als-6b995458c-z8jt9                           2/2     Running   0          19m   cp-v115x.istio-system
      smm-authentication-78d96d6fc9-hg89p               2/2     Running   0          19m   cp-v115x.istio-system
      smm-federation-gateway-7c7d9b7fb5-xgv5t           2/2     Running   0          19m   cp-v115x.istio-system
      smm-federation-gateway-operator-ff8598cb7-xj7pk   2/2     Running   0          19m   cp-v115x.istio-system
      smm-grafana-7bcf9f5885-jhwpg                      3/3     Running   0          19m   cp-v115x.istio-system
      smm-health-56896f5b9b-r54w8                       2/2     Running   0          19m   cp-v115x.istio-system
      smm-health-api-665d4787-pw7z4                     2/2     Running   0          19m   cp-v115x.istio-system
      smm-ingressgateway-b6d5b5b84-l5llx                1/1     Running   0          17m   cp-v115x.istio-system
      smm-kubestatemetrics-5455b9697-5tbgq              2/2     Running   0          19m   cp-v115x.istio-system
      smm-leo-7b64559786-2sj4c                          2/2     Running   0          19m   cp-v115x.istio-system
      smm-prometheus-operator-66dbdb499d-sz6t8          3/3     Running   1          19m   cp-v115x.istio-system
      smm-sre-alert-exporter-668d9cbd68-926t5           2/2     Running   0          19m   cp-v115x.istio-system
      smm-sre-api-86cf44fbbb-lxvxd                      2/2     Running   0          19m   cp-v115x.istio-system
      smm-sre-controller-858b984df6-6b5r6               2/2     Running   0          19m   cp-v115x.istio-system
      smm-tracing-76c688ff6f-7ctjk                      2/2     Running   0          19m   cp-v115x.istio-system
      smm-vm-integration-5df64bdb4b-68xgh               2/2     Running   0          19m   cp-v115x.istio-system
      smm-web-677b9f4f5b-ss9zs                          3/3     Running   0          19m   cp-v115x.istio-system
      
    • If you are upgrading both Service Mesh Manager and Streaming Data Manager, run the following command to verify that the installation is complete.

      kubectl get pods -A -L istio.io/rev
      

      The output should be similar to:

      NAMESPACE                  NAME                                                      READY   STATUS      RESTARTS        AGE     REV
      cert-manager               cert-manager-67575448dd-8qbws                             1/1     Running     0               5h56m
      cert-manager               cert-manager-cainjector-79f8d775c7-ww7fw                  1/1     Running     0               5h56m
      cert-manager               cert-manager-webhook-5949cc4b67-gwknv                     1/1     Running     0               5h56m
      cluster-registry           cluster-registry-controller-b86f8857c-44jh8               1/1     Running     0               5h57m
      csr-operator-system        csr-operator-5955b44674-bvl9p                             2/2     Running     0               5h56m
      istio-system               istio-meshexpansion-v115x-d8555488f-btdx6              1/1     Running     0               37m     v115x.istio-system
      istio-system               istiod-v115x-555749b797-dcwwm                          1/1     Running     0               5h55m   v115x.istio-system
      istio-system               istiod-sdm-iv115x-6c8cfb5fc5-85w2d                     1/1     Running     0               5h55m   sdm-iv115x.istio-system
      kafka                      kafka-operator-operator-76df6db8d4-l4kkq                  3/3     Running     2 (5h52m ago)   5h53m   sdm-iv115x.istio-system
      smm-registry-access        imagepullsecrets-controller-6c45b46459-qb9j8              1/1     Running     0               6h1m
      smm-system                 istio-operator-v113x-6fb944b86b-xgpbd                     2/2     Running     0               5h55m
      smm-system                 istio-operator-v115x-68dcbc59c8-vt2mp                     2/2     Running     0               5h55m
      smm-system                 mesh-manager-0                                            2/2     Running     0               5h56m
      smm-system                 prometheus-node-exporter-74dcm                            1/1     Running     0               5h53m
      smm-system                 prometheus-node-exporter-8s458                            1/1     Running     0               5h59m
      smm-system                 prometheus-node-exporter-vmth4                            1/1     Running     0               5h59m
      smm-system                 prometheus-node-exporter-xsk8j                            1/1     Running     0               5h59m
      smm-system                 prometheus-smm-prometheus-0                               4/4     Running     0               5h55m   v115x.istio-system
      smm-system                 smm-656d45f7cc-c2kd6                                      2/2     Running     0               5h55m   v115x.istio-system
      smm-system                 smm-656d45f7cc-xrx9n                                      2/2     Running     0               5h55m   v115x.istio-system
      smm-system                 smm-als-855c6878b7-55gvd                                  2/2     Running     0               5h55m   v115x.istio-system
      smm-system                 smm-authentication-666547f79f-hwt6t                       2/2     Running     0               5h55m   v115x.istio-system
      smm-system                 smm-federation-gateway-fd4bbb4f8-4nql8                    2/2     Running     1 (5h54m ago)   5h55m   v115x.istio-system
      smm-system                 smm-federation-gateway-operator-bd94d8444-nbvjz           2/2     Running     0               5h55m   v115x.istio-system
      smm-system                 smm-grafana-59c54f67f4-tft2h                              3/3     Running     0               5h55m   v115x.istio-system
      smm-system                 smm-health-86b8dbdf68-k8bfr                               2/2     Running     0               5h55m   v115x.istio-system
      smm-system                 smm-health-api-69bc97d89-gkdp5                            2/2     Running     0               5h55m   v115x.istio-system
      smm-system                 smm-ingressgateway-9875bc895-v95m9                        1/1     Running     0               37m     v115x.istio-system
      smm-system                 smm-kubestatemetrics-86c6f96789-cxsrb                     2/2     Running     0               5h55m   v115x.istio-system
      smm-system                 smm-leo-8446486596-2w7fc                                  2/2     Running     0               5h55m   v115x.istio-system
      smm-system                 smm-prometheus-operator-77cd64556d-ghz5r                  3/3     Running     1 (5h55m ago)   5h55m   v115x.istio-system
      smm-system                 smm-sre-alert-exporter-5dd8b64d58-ccrnh                   2/2     Running     0               5h55m   v115x.istio-system
      smm-system                 smm-sre-api-998fc554b-lpvsq                               2/2     Running     0               5h55m   v115x.istio-system
      smm-system                 smm-sre-controller-68c974c9db-grb44                       2/2     Running     0               5h55m   v115x.istio-system
      smm-system                 smm-tracing-5886d59dd-7k6kt                               2/2     Running     0               5h55m   v115x.istio-system
      smm-system                 smm-vm-integration-5cb96cdd78-mh5lh                       2/2     Running     0               5h55m   v115x.istio-system
      smm-system                 smm-web-55f45cc8c5-gd894                                  3/3     Running     0               5h55m   v115x.istio-system
      supertubes-control-plane   supertubes-control-plane-5bdbfcf5b6-85bw7                 2/2     Running     0               5h57m
      supertubes-system          prometheus-operator-grafana-5fd88bcf86-55kgg              4/4     Running     0               5h53m   sdm-iv115x.istio-system
      supertubes-system          prometheus-operator-kube-state-metrics-5dbf8656db-wlzfw   2/2     Running     2 (5h53m ago)   5h53m   sdm-iv115x.istio-system
      supertubes-system          prometheus-operator-operator-7bdc575546-b4n94             2/2     Running     1 (5h53m ago)   5h53m   sdm-iv115x.istio-system
      supertubes-system          prometheus-operator-prometheus-node-exporter-69cmx        1/1     Running     0               5h53m
      supertubes-system          prometheus-operator-prometheus-node-exporter-75b7q        1/1     Running     0               5h53m
      supertubes-system          prometheus-operator-prometheus-node-exporter-skksk        1/1     Running     0               5h53m
      supertubes-system          prometheus-operator-prometheus-node-exporter-v2pll        1/1     Running     0               5h53m
      supertubes-system          prometheus-prometheus-operator-prometheus-0               3/3     Running     0               5h53m   sdm-iv115x.istio-system
      supertubes-system          supertubes-6f6b86b497-c5zqf                               3/3     Running     1 (5h54m ago)   5h54m   sdm-iv115x.istio-system
      supertubes-system          supertubes-ui-backend-c97564f84-c2vd6                     2/2     Running     2 (5h54m ago)   5h54m   sdm-iv115x.istio-system
      zookeeper                  zookeeper-operator-6ff85cf58d-6kxhk                       2/2     Running     1 (5h54m ago)   5h54m   sdm-iv115x.istio-system
      zookeeper                  zookeeper-operator-post-install-upgrade-qq4kf             0/1     Completed   0               5h54m
      
  4. Restart your workloads to move your workloads to v115x mesh.

In operator mode

In case the deployment is managed in operator mode the upgrade procedure only consists of installing a newer version of the operator helm chart and allowing it to reconcile the cluster.

SMM upgrade

  1. Uninstall the previous version (1.10.0) of the smm-operator chart.

    helm uninstall smm-operator --namespace smm-registry-access
    
  2. Install the new version (1.11.0) of the smm-operator chart.

    helm install \
      --namespace=smm-registry-access \
      --set "global.ecr.enabled=false" \
      --set "global.basicAuth.username=<your-username>" \
      --set "global.basicAuth.password=<your-password>" \
      smm-operator \
      oci://registry.eticloud.io/smm-charts/smm-operator --version 1.11.0
    

    Note: If the system uses helm for deploying the chart (and not some other CI/CD solution such as Argo CD), then the CustomResourceDefinitions (CRDs) will not be automatically upgraded. In this case, fetch the helm chart locally using the helm pull command and apply the CRDs in the crds folder of the helm chart manually.

  3. After the operator has been started, monitor the status of the ControlPlane resource until it finishes the upgrade (reconciliation). Run the following command:

    kubectl describe cp
    

    After the upgrade is finished, the output should be similar to the following. The Status: Succeeded line shows that the deployment has been upgraded. In case of any errors, consult the Kubernetes logs of the operator (installed by Helm) for further information.

    ...
    Status:
      Components:
        Cert Manager:
          Status:  Available
        Cluster Registry:
          Status:  Available
        Mesh Manager:
          Status:  Available
        Node Exporter:
          Status:  Available
        Registry Access:
          Status:  Available
        Smm:
          Status:  Available
      Status:      Succeeded
    
  4. Restart your workloads to move your workloads to the v115x mesh.

SDM upgrade

  1. Uninstall previous version of the sdm-operator chart (if Streaming Data Manager is installed).

    helm uninstall --namespace supertubes-control-plane sdm-operator
    
  2. Install the new version 1.8.0 of the sdm-operator chart.

    helm install \
      --namespace supertubes-control-plane \
      --set imagePullSecrets={smm-pull-secret} \
      --set operator.image.repository="registry.eticloud.io/sdm/supertubes-control-plane" \
      sdm-operator \
      oci://registry.eticloud.io/sdm-charts/supertubes-control-plane --version 1.8.0
    
  3. After the operator has been started, monitor the status of the applicationmanifest resource until it finishes the upgrade (reconciliation). Run the following command:

    kubectl describe applicationmanifests.supertubes.banzaicloud.io -n supertubes-control-plane sdm-applicationmanifest
    

    The output should be similar to:

    ...
    Status:
      Components:
        Cluster Registry:
          Status:  Removed
        Csr Operator:
          Status:  Available
        Imps Operator:
          Image Pull Secret Status:  Unmanaged
          Status:                    Removed
        Istio Operator:
          Status:  Removed
        Kafka Operator:
          Status:  Available
        Monitoring:
          Status:  Available
        Supertubes:
          Status:  Available
        Zookeeper Operator:
          Status:  Available
      Status:      Succeeded
    

In a GitOps scenario

If you have installed Service Mesh Manager using our GitOps guide, complete the following steps to upgrade the operator chart.

  1. Check your username and password on the download page.

  2. Download the smm-operator chart from registry.eticloud.io into the charts directory of your Service Mesh Manager GitOps repository and extract it. Run the following commands:

    export HELM_EXPERIMENTAL_OCI=1 # Needed prior to Helm version 3.8.0
    
    echo "${CALISTI_PASSWORD}" | helm registry login registry.eticloud.io -u "${CALISTI_USERNAME}" --password-stdin
    

    Expected output:

    Login Succeeded
    
    helm pull oci://registry.eticloud.io/smm-charts/smm-operator --destination ./charts/ --untar --version 1.11.0
    

    Expected output:

    Pulled: registry.eticloud.io/smm-charts/smm-operator:latest-stable-version
    Digest: sha256:someshadigest
    
  3. Commit the changes and push the repository.

    git add .
    git commit -m "Update smm-operator chart"
    git push origin
    
  4. Restart your workloads to move your workloads to the v115x mesh.

Restarting workloads

After the upgrade has completed, the Pods running in applications' namespaces are still running the old version of Istio proxy sidecar.

  1. To obtain the latest security patches, restart these Controllers (Deployments, StatefulSets, and so on) either using the kubectl rollout command, or by instructing the CI/CD systems enabled on the cluster. For example, to restart the deployments in a namespace, you can run:

    kubectl rollout restart deployment --namespace <name-of-your-namespace>
    
  2. If the upgrade also involved a minor or major version upgrade of Istio, the kubectl rollout command will only ensure that the latest patch level is being used on the Pods.

    For example: Service Mesh Manager 1.8.2 comes with Istio 1.11, while Service Mesh Manager 1.9.0 is bundled with Istio 1.12. By upgrading from Service Mesh Manager 1.8.2 to 1.9.0, and then restarting the Controllers will only result in the latest 1.11 Istio sidecar proxy to be started in the Pods.

    To upgrade to the new minor/major version of Istio on your workloads, complete the Canary control plane upgrades procedure.