Recovery procedure

This document details how to change and revoke all keys related to a Streaming Data Manager installation when the cluster where it is running is compromised. We’ll assume that an attacker could have had access to various Streaming Data Manager related service account tokens or keys.

Disclaimer: This is not a full guide for Kubernetes security recovery, only covers parts related to Streaming Data Manager.

Service account tokens

Service account tokens provide access to specific actions related to the Kubernetes API. Streaming Data Manager is using multiple service account tokens to access the Kubernetes API service. If you think some, or all of these are compromised, and an attacker could have gained access, delete these tokens to revoke them, and restart the pods in the Streaming Data Manager related namespaces.

Complete the following steps:

1. Delete all secrets holding service account tokens.

   for namespace in supertubes-system zookeeper cert-manager kafka istio-system csr-system
   do
     kubectl delete secret -n $namespace --field-selector type=kubernetes.io/service-account-token
   done
   

2. Restart all pods.

   for namespace in supertubes-system zookeeper cert-manager kafka istio-system csr-system
   do
     kubectl delete pods --all
   done
   

Istio root CAs

Istio root CAs are used to sign certificates provided to workloads in the service mesh. If an attacker gets access to your CA keys, they may be able to set up a man-in-the-middle attack and eavesdrop encrypted network traffic.

Using the default, generated CA

If you are using the default, generated CA, delete the secret that holds the generated root CA and restart all pods in the istio-system namespace.

kubectl delete secret -n istio-system istio-ca-secret
kubectl delete pods -n istio-system --all

After Istiod is restarted, and the CA is regenerated, you must restart all pods with a sidecar in the system.

Using your own certificates

If you are using your own certificates with Istio, revoke them, create new ones, and configure those in the cluster.

This guide doesn’t cover how exactly you’ll need to revoke the certificates, or create new ones. It depends on how the original certificates were created, what is the signing root CA for them, and where else are they used. You’ll probably need to add them to the Certificate Revocation List (CRL) of the issuing Certificate Authority, and follow the issuing Certificate Authority’s guidelines to create a new one.

Once you’ve revoked the CA and generated a new one, set the new CA in Streaming Data Manager.

• If you’re storing the CAs in secrets on the cluster, update the contents of the configured secret specified in spec.citadel.caSecretName. To properly set up the secret with the certificates, follow the Istio documentation
• If you’ve configured Vault using istiod.ca.vault in the IstioControlPlane CR, update the CAs in Vault.

Once your CA is changed, restart all pods in the istio-system namespace:

kubectl delete pods -n istio-system --all

After Istiod is restarted, restart all pods with a sidecar in the system.

Using CSR-operator

If you are using the CSR-operator as a certificate issuer in your Istio cluster, simply delete the CA certificate secret generated by the CSR-operator named csr-operator-cacerts in the namespace CSR-operator is installed. The default value for this namespace is csr-operator-system.

kubectl delete secret csr-operator-cacerts -n <csr-operator-namespace>

After that, restart the CSR-operator pod, so it creates a new CA secret.

Delete and create a new external-ca-cert secret for istio, so it uses the new external CA:

kubectl create -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: external-ca-cert
  namespace: istio-system
data:
  root-cert.pem: <ca_crt.pem-from-csr-operator-cacerts>
EOF

Restart all the pods in the istio-system namespace:

kubectl delete pods -n istio-system --all

Finally, after Istiod is restarted, restart all pods with a sidecar in the system.