Recovery procedure

This document details how to change and revoke all keys related to a Service Mesh Manager installation when the cluster where it is running is compromised. We’ll assume that an attacker could have had access to various Service Mesh Manager related service account tokens or keys.

Disclaimer: This is not a full guide for Kubernetes security recovery, only covers parts related to Service Mesh Manager.

Service Account Tokens

Service account tokens provide access to specific actions related to the Kubernetes API. Service Mesh Manager is using multiple service account tokens to access the Kubernetes API service. If you think some, or all of these are compromised, and an attacker could have gained access, delete these tokens to revoke them, and restart the pods in the Service Mesh Manager related namespaces. Complete the following steps:

1. Delete all secrets holding service account tokens.

   for namespace in smm-system smm-demo cert-manager cluster-registry istio-system smm-canary smm-registry-access
   do
     kubectl delete secret -n $namespace $(kubectl get secret -n smm-system --field-selector type=kubernetes.io/service-account-token -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}')
   done
   

2. Restart all pods.

   for namespace in smm-system smm-demo cert-manager cluster-registry istio-system smm-canary smm-registry-access
   do
     kubectl delete pods --all
   done
   

Istio root CAs

Istio root CAs are used to sign certificates provided to workloads in the service mesh. If an attacker gets access to your CA keys, they may be able to set up a man-in-the-middle attack and eavesdrop encrypted network traffic.

Using the default, generated CA

If you are using the default, generated CA, delete the secret that holds the generated root CA and restart all pods in the istio-system namespace.

k delete secret -n istio-system istio-ca-secret
k delete pods -n istio-system --all

After Istiod is restarted, and the CA is regenerated, you restart all pods with a sidecar in the system.

Using your own certificates

If using your own certificates with Istio, revoke them, create new ones, and configure those in the cluster.

This guide doesn’t cover how exactly you’ll need to revoke the certificates, or create new ones. It depends on how the original certificates were created, what is the signing root CA for them, and where else are they used. You’ll probably need to add them to the Certificate Revocation List (CRL) of the issuing Certificate Authority, and follow the issuing Certificate Authority’s guidelines to create a new one.

Once you’ve revoked the CA and generated a new one, set the new CA in Service Mesh Manager.

1. If you’re storing the CAs in secrets on the cluster, update the contents of the configured secret specified in spec.citadel.caSecretName. To properly set up the secret with the certificates, follow the Istio documentation
2. If you’ve configured Vault using istiod.ca.vault in the IstioControlPlane CR, update the CAs in Vault.

Once your CA is changed, restart all pods in the istio-system namespace:

kubectl delete pods -n istio-system --all

After Istiod is restarted, restart all pods with a sidecar in the system.

JWT backend key

A JWT backend key is generated during Service Mesh Manager installation to sign and validate JWT tokens for the GraphQL API authentication service. If a malicious attacker acquires this key, they could use it to forge a JWT token and gain unauthorized access to the SMM API using that key to retrieve or even change Kubernetes data.

The JWT backend key is stored in the smm-authentication Kubernetes secret in the smm-system namespace. If you suspect that this key is compromised, delete the Kubernetes secret:

kubectl delete secret -n smm-system smm-authentication

To regenerate a new key:

• If you’re using the Service Mesh Manager operator, it will automatically regenerate the key and the secret.
• Otherwise, rerun the smm install command.

IMPS

Service Mesh Manager can hold access key pairs in Kubernetes secrets to be able to access images in ECR (default) or other image registries. It’s always a good idea to restrict the permissions of these keys to image access, but in case a malicious attacker gets access to these secrets, revoke these keypairs, and create new ones instead. To be able to revoke the old keys and create new ones, you should follow the process of the image registry you’re using.

For ECR, you should be able to revoke and recreate credentials on the AWS console, or by contacting your AWS administrator.

To check the credentials configured in the environment, list secrets in the smm-registry-access namespace. Credentials are base64 encoded within the secrets:

k get secret -n smm-registry-access

Once you have new image registry credentials, use the smm registry commands of the Service Mesh Manager CLI to configure them in the cluster.

• smm registry list lists the currently configured image registry settings.
• smm registry remove remove the configured registry entry.
• smm registry add configures your image registry with new access credentials.
• smm registry reconcile updates the remote cluster’s image pull secrets from the local settings.