Fetch CA certificate from Vault

Fetching CA certificate overview

Istio uses X.509 certificates to create strong identities to every workload. These certificates are used to authenticate the service-to-service communication in the mesh. The Istio agent of the workload obtains these certificates from istiod and provides them to the Envoy proxy of the workload. istiod acts as a Certificate Authority, and signs the certificates with the signing certificate of the Certificate Authority (CA) certificate.

How Istio provides certificates for the workloads How Istio provides certificates for the workloads

By default, the Certificate Authority has a self-signed root certificate and key and uses them to sign the workload certificates. The private key of the CA is stored in Kubernetes secrets. In production environments these are not recommended, and you should instead:

  • Manage the CA independently from Istio, in a solution that offers strong security protection.
  • Use a production-ready CA, such as Hashicorp Vault.

You can configure Service Mesh Manager to use Vault as a secret store and fetch the signing certificate of the Certificate Authority from a Vault instance. That way you can securely store the CA certificate and its private key outside Istio.

The following high-level procedure shows the main steps of configuring Service Mesh Manager to use Vault.

Note: Service Mesh Manager works with any Vault solution. For the following procedures and examples we used Bank-Vaults, which is an open-source solution to run and manage Vault in Kubernetes.

  1. Configure Vault so Istio (the istiod service account) can access the CA certificate.
  2. Configure Istio to use the CA from Vault. If you are using Bank-Vaults, then the following examples show how to configure Service Mesh Manager to inject the bank-vaults sidecar into Istio, so the istiod can access the CA certificate.
  3. (Optional) Test your setup to verify that it works properly.

Prerequisites

Before configuring Istio and Vault, make sure these prerequisites are met:

  • Make sure you have a Vault instance with CA certificate and private key ready to pull.

  • If you want to use Bank-Vaults to create and manage Vault, install Vault, the Vault operator, and the Vault Secrets Webhook using the Helm charts of the Bank-Vaults project. For details, see the Bank-Vaults documentation).

Configure Vault

Create a policy so that istiod can access the Vault secrets.

To give istiod the access right to Vault secrets, a policy needs to be created for authentication purpose. The Vault is not limited to Bank-Vaults, it can be any other vault solutions. But we will use bank-vaults as an example on how to set the authentication policy:

Edit Vault CR by running kubectl edit vault.

spec:
  caNamespaces:
  - istio-system
  externalConfig:
    auth:
    - roles:
      # Make sure the role name 'istiod' corresponds to the role name we set in Istio
      - bound_service_account_names:
        - istiod-service-account-cp-v115x
        bound_service_account_namespaces:
        - istio-system
        name: istiod
        policies:
        - allow_istiod_secrets
      - bound_service_account_names:
        - default
        - vault-secrets-webhook
        - vault
        bound_service_account_namespaces:
        - default
        name: default
        policies:
        - allow_secrets
        - allow_pki
        ttl: 1h
      type: kubernetes
    policies:
    # Policy created for istiod to read secrets under this specific path
    - name: allow_istiod_secrets
      rules: path "secret/data/pki/istiod" { capabilities = ["read", "list"] }
    - name: allow_secrets
      rules: path "secret/*" { capabilities = ["create", "read", "update", "delete",
        "list"] }
    - name: allow_pki
      rules: path "pki/*" { capabilities = ["create", "read", "update", "delete",
        "list"] }

Note: Remember to add the istio-system namespace under .spec.caNamespaces so that istiod has the required TLS certificate to access Vault.

Configure SMM

In order for istiod pod to access the Vault instance, you need to set values of .istiod.ca.vault.

  1. Collect the parameters you need to configure the Istio custom resource:

    • The address of your Vault instance (For example, https://vault.default:8200).
    • The path to the CA certificate in Vault. Istio will use this certificate to sign the workload certificates with. (For example, vault:secret/data/pki/istiod#certificate).
    • The path to the private key of the certificate in Vault. (For example, vault:secret/data/pki/istiod#privateKey).
    • The name of the role that Istio can use to access the CA certificate and its private key in Vault. (For example, istiod).

  2. Edit the Istio custom resource. You can use the following command to edit the Istio custom resource (adjust the name of the controlplane to your Istio version: kubectl edit istio cp-v115x -n istio-system

  3. Add the parameters of Vault under the .istiod.ca.vault key, for example:

      istiod:
    ca:
      vault:
        address: https://vault.default:8200
        certPath: vault:secret/data/pki/istiod#certificate
        enabled: true
        keyPath: vault:secret/data/pki/istiod#privateKey
        role: istiod
        vaultEnvImage: ghcr.io/banzaicloud/vault-env:1.13.1

  4. Set the .istiod.ca.vault.enabled key to true, so that vault-env sidecar will be injected into istiod

Test your setup

If you are using Bank-Vaults and followed the steps on this page, you should see that the vault-env sidecar has been injected into the istiod pod which pulls the certificate from the Vault instance.

Sidecar container injected in istiod Sidecar container injected in istiod

If not, delete the istiod pod and wait for it to restart with the latest Istio configuration.

Additional Info

Add secrets in Vault

For testing purposes, you may need this step-by-step guide to put some secrets in the specified path defiend in Istio CR.

  1. Port-forward into the pod: 
    kubectl port-forward vault-0 8200 &
  2. Set the address of the Vault instance: 
    export VAULT_ADDR=https://127.0.0.1:8200
  3. Import the CA certificate and private key of the Vault instance by running the following commands (otherwise, you’ll get x509: certificate signed by unknown authority errors): 
    kubectl get secret vault-tls -o jsonpath="{.data.ca\.crt}" | base64 --decode > $PWD/vault-ca.crt
kubectl get secret vault-tls -o jsonpath="{.data.ca\.key}" | base64 --decode > $PWD/vault-ca.key
export VAULT_CACERT=$PWD/vault-ca.crt
  4. To authenticate to Vault, you can access its root token by running: 
    export VAULT_TOKEN=$(kubectl get secrets vault-unseal-keys -o jsonpath={.data.vault-root} | base64 --decode)
  5. Now you can interact with Vault and add a secret by running: 
    vault kv put secret/pki/istiod privateKey=@vault-ca.key certificate=@vault-ca.crt
  6. To verify the secret has been added to Vault: 
    vault kv get secret/pki/istiod