Setup Istio to use CSR operator as external CA

Streaming Data Manager allows you to use a separate CSR operator to provide certificates to Istio.

  1. Install CSR-operator. CSR-operator runs in privateCA signer mode and it generates the CA key and CA certificate into a secret by default. The name of the secret is “csr-operator-cacerts” and it can be found in the namespace where the CSR-operator has been installed.

    smm sdm csr install --namespace <csr-operator-namespace>
    
  2. Configure Istio to use CSR-operator as an external CA. The CSR-operator signs certificate signing requests which are generated by Istio.

    1. From the secret generated automatically by the CSR-operator (“csr-operator-cacerts” in the “csr-operator-system” namespace), create a new secret into the namespace where Istio is installed (by default, it is “istio-system”), because Istio requires that secret in another format without the CA private key.

      kubectl apply -f - <<EOF
      apiVersion: v1
      kind: Secret
      metadata:
        name: sdm-istio-external-ca-cert
        namespace: istio-system
      data:
        root-cert.pem: $(kubectl --namespace csr-operator-system get secret csr-operator-cacerts -o 'jsonpath={.data.ca_crt\.pem}')
      EOF
      

      Expected output:

      secret/sdm-istio-external-ca-cert created
      
    2. Deploy the IstioControlPlane CR into your cluster.

      For OpenShift:

      kubectl apply -f - <<EOF
      apiVersion: servicemesh.cisco.com/v1alpha1
      kind: IstioControlPlane
      metadata:
        labels:
          banzaicloud.io/managed-by: supertubes
        name: sdm-icp-v115x
        namespace: istio-system
      spec:
        containerImageConfiguration:
          imagePullPolicy: Always
          imagePullSecrets:
          - name: smm-pull-secret
        distribution: cisco
        istiod:
          deployment:
            env:
              # Skip validating that the peer is from the same trust domain when mTLS is enabled in authentication policy
              - name: PILOT_SKIP_VALIDATE_TRUST_DOMAIN
                value: "true"
              # Indicate to Istiod that we use an external signer (likely to be removed and added to mesh config - from upstream Istio)
              - name: EXTERNAL_CA
                value: ISTIOD_RA_KUBERNETES_API
              # Kubernetes CA signer type (likely to be removed and added to mesh config - from upstream Istio)
              - name: K8S_SIGNER
                value: csr.banzaicloud.io/privateca
              - name: ISTIO_MULTIROOT_MESH
                value: "true"
            image: 033498657557.dkr.ecr.us-east-2.amazonaws.com/banzaicloud/istio-pilot:v1.15.3-bzc.1
        k8sResourceOverlays:
          - groupVersionKind:
              group: apps
              kind: Deployment
              version: v1
            objectKey:
              name: istiod-sdm-icp-v115x
            patches:
              - parseValue: true
                path: /spec/template/spec/volumes/-
                type: replace
                value: |
                  name: external-ca-cert
                  secret:
                    secretName: sdm-istio-external-ca-cert
                    optional: true
              - parseValue: true
                path: /spec/template/spec/containers/name=discovery/volumeMounts/-
                type: replace
                value: |
                  name: external-ca-cert
                  mountPath: /etc/external-ca-cert
                  readOnly: true
          # Amend ClusterRole to add permission for istiod to approve certificate signing by custom signer
          - groupVersionKind:
              group: rbac.authorization.k8s.io
              kind: ClusterRole
              version: v1
            objectKey:
              name: istiod-sdm-icp-v115x-istio-system
            patches:
              - parseValue: true
                path: /rules/-
                type: replace
                value: |
                  apiGroups:
                  - certificates.k8s.io
                  resourceNames:
                  - csr.banzaicloud.io/privateca
                  resources:
                  - signers
                  verbs:
                  - approve
        meshConfig:
          defaultConfig:
            proxyMetadata:
              PROXY_CONFIG_XDS_AGENT: "true"
          enableAutoMtls: true
          protocolDetectionTimeout: 5s
        meshID: sdm
        mode: ACTIVE
        proxy:
          image: 033498657557.dkr.ecr.us-east-2.amazonaws.com/banzaicloud/istio-proxyv2:v1.15.3-bzc-kafka.0
        proxyInit:
          cni:
            binDir: /var/lib/cni/bin
            chained: false
            confDir: /etc/cni/multus/net.d
            confFileName: istio-cni-sdm-icp-v115x-istio-system.conf
            daemonset:
              image: 033498657557.dkr.ecr.us-east-2.amazonaws.com/banzaicloud/istio-install-cni:v1.15.3-bzc.1
              securityContext:
                privileged: true
            enabled: true
          image: 033498657557.dkr.ecr.us-east-2.amazonaws.com/banzaicloud/istio-proxyv2:v1.15.3-bzc-kafka.0
        telemetryV2:
          enabled: true
        version: 1.15.3
      EOF
      

      For Kubernetes:

      kubectl create -f - <<EOF
      apiVersion: servicemesh.cisco.com/v1alpha1
      kind: IstioControlPlane
      metadata:
        labels:
          banzaicloud.io/managed-by: supertubes
        name: sdm-icp-v115x
        namespace: istio-system
      spec:
        containerImageConfiguration:
          imagePullPolicy: Always
          imagePullSecrets:
          - name: smm-pull-secret
        distribution: cisco
        istiod:
          deployment:
            env:
              # Skip validating the peer is from the same trust domain when mTLS is enabled in authentication policy
              - name: PILOT_SKIP_VALIDATE_TRUST_DOMAIN
                value: "true"
              # Indicate to Istiod that we use an external signer (likely to be removed and added to mesh config - from upstream Istio)
              - name: EXTERNAL_CA
                value: ISTIOD_RA_KUBERNETES_API
              # Kubernetes CA signer type (likely to be removed and added to mesh config - from upstream Istio)
              - name: K8S_SIGNER
                value: csr.banzaicloud.io/privateca
              - name: ISTIO_MULTIROOT_MESH
                value: "true"
            image: 033498657557.dkr.ecr.us-east-2.amazonaws.com/banzaicloud/istio-pilot:v1.15.3-bzc.1
        k8sResourceOverlays:
          - groupVersionKind:
              group: apps
              kind: Deployment
              version: v1
            objectKey:
              name: istiod-sdm-icp-v115x
            patches:
              - parseValue: true
                path: /spec/template/spec/volumes/-
                type: replace
                value: |
                  name: external-ca-cert
                  secret:
                    secretName: sdm-istio-external-ca-cert
                    optional: true
              - parseValue: true
                path: /spec/template/spec/containers/name=discovery/volumeMounts/-
                type: replace
                value: |
                  name: external-ca-cert
                  mountPath: /etc/external-ca-cert
                  readOnly: true
          # Amend ClusterRole to add permission for istiod to approve certificate signing by custom signer
          - groupVersionKind:
              group: rbac.authorization.k8s.io
              kind: ClusterRole
              version: v1
            objectKey:
              name: istiod-sdm-icp-v115x-istio-system
            patches:
              - parseValue: true
                path: /rules/-
                type: replace
                value: |
                  apiGroups:
                  - certificates.k8s.io
                  resourceNames:
                  - csr.banzaicloud.io/privateca
                  resources:
                  - signers
                  verbs:
                  - approve
        meshConfig:
          defaultConfig:
            proxyMetadata:
              PROXY_CONFIG_XDS_AGENT: "true"
          enableAutoMtls: true
          protocolDetectionTimeout: 5s
        meshID: sdm
        mode: ACTIVE
        proxy:
          image: 033498657557.dkr.ecr.us-east-2.amazonaws.com/banzaicloud/istio-proxyv2:v1.15.3-bzc-kafka.0
        telemetryV2:
          enabled: true
        version: 1.15.3
      EOF
      

      Expected output:

      istiocontrolplane.servicemesh.cisco.com/sdm-icp-v115x created
      
    3. Check for the IstioControlPlane and pods to be available.

      kubectl get istiocontrolplanes.servicemesh.cisco.com -n istio-system sdm-icp-v115x
      

      Expected output:

      NAME            MODE     NETWORK    STATUS      MESH EXPANSION   EXPANSION GW IPS   ERROR   AGE
      sdm-icp-v115x   ACTIVE   network1   Available                                               5m21s
      
      kubectl get pods -n istio-system
      

      Expected output:

      For OpenShift:

      NAME                                    READY   STATUS    RESTARTS   AGE
      istio-cni-node-sdm-icp-v115x-2mkxn      1/1     Running   0          2m37s
      istio-cni-node-sdm-icp-v115x-6t6lx      1/1     Running   0          2m37s
      istio-cni-node-sdm-icp-v115x-7nxqs      1/1     Running   0          2m37s
      istio-cni-node-sdm-icp-v115x-htgzw      1/1     Running   0          2m37s
      istio-cni-node-sdm-icp-v115x-mdrvj      1/1     Running   0          2m37s
      istio-cni-node-sdm-icp-v115x-mk6vh      1/1     Running   0          2m37s
      istio-cni-node-sdm-icp-v115x-mstzx      1/1     Running   0          2m37s
      istio-cni-node-sdm-icp-v115x-qwlvz      1/1     Running   0          2m37s
      istio-cni-node-sdm-icp-v115x-rjlrz      1/1     Running   0          2m37s
      istio-cni-node-sdm-icp-v115x-tk5xv      1/1     Running   0          2m37s
      istio-cni-node-sdm-icp-v115x-x88ls      1/1     Running   0          2m37s
      istio-cni-node-sdm-icp-v115x-xht6n      1/1     Running   0          2m37s
      istio-cni-node-sdm-icp-v115x-xv6gw      1/1     Running   0          2m37s
      istio-operator-5d7cb59c9-g2htw          2/2     Running   0          7m29s
      istiod-sdm-icp-v115x-54f6c69775-nfs64   1/1     Running   0          2m42s
      

      For Kubernetes:

      istio-operator-5d7cb59c9-5q6dx          2/2     Running   0          114m
      istiod-sdm-icp-v115x-54f6c69775-786bj   1/1     Running   0          5m43s
      
    4. Create the istiomesh-ca-trust-extension-script ConfigMap.

      kubectl apply -f - <<EOF
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: istiomesh-ca-trust-extension-script
        namespace: supertubes-control-plane
      data:
        run.sh: |-
          #!/bin/sh
          # Fill these fields properly----------------------
          export CA_SECRET_NAMESPACE="istio-system"
          export CA_SECRET_NAME="sdm-istio-external-ca-cert"
          # ------------------------------------------------
          export ICP_NAME="sdm-icp-v115x"
          export ICP_NAMESPACE="istio-system"
          export CA_CERT=\$(kubectl get secret -n \$CA_SECRET_NAMESPACE \$CA_SECRET_NAME -o jsonpath='{.data.root-cert\.pem}' | base64 -d | sed '\$ ! s/\$/\\\n/' | tr -d '\n')
          read -r -d '' PATCH << EOF
          {"spec": {"meshConfig": {"caCertificates": [{"pem": "\$CA_CERT"}]}}}
          EOF
          read -r -d '' INSERT_PATCH << EOF
          [{"op": "add", "path": "/spec/meshConfig/caCertificates/-", "value": {"pem": "\$CA_CERT"}}]
          EOF
          kubectl patch istiocontrolplanes.servicemesh.cisco.com \$ICP_NAME -n \$ICP_NAMESPACE --type json --patch="\$INSERT_PATCH" || kubectl patch istiocontrolplanes.servicemesh.cisco.com \$ICP_NAME -n \$ICP_NAMESPACE --type merge --patch="\$PATCH"
      EOF
      

      Expected output:

      configmap/istiomesh-ca-trust-extension-script created
      
    5. Create the istiomesh-ca-trust-extension Job.

      kubectl apply -f - <<EOF
      apiVersion: batch/v1
      kind: Job
      metadata:
        name: istiomesh-ca-trust-extension
        namespace: supertubes-control-plane
      spec:
        completions: 1
        template:
          metadata:
            name: istiomesh-ca-trust-extension
          spec:
            containers:
            - command:
              - /scripts/run.sh
              image: lachlanevenson/k8s-kubectl:v1.16.10
              imagePullPolicy: IfNotPresent
              name: istio-trust-extension-job
              volumeMounts:
              - mountPath: /scripts
                name: run
                readOnly: false
            dnsPolicy: ClusterFirst
            restartPolicy: Never
            serviceAccount: supertubes-control-plane
            serviceAccountName: supertubes-control-plane
            volumes:
            - configMap:
                defaultMode: 365
                name: istiomesh-ca-trust-extension-script
              name: run
      EOF
      

      Expected output:

      job.batch/istiomesh-ca-trust-extension created
      

      Check if the job ran successfully.

      kubectl get pods -n supertubes-control-plane
      

      Expected output:

      NAME                                        READY   STATUS      RESTARTS   AGE
      istiomesh-ca-trust-extension-4r9sr          0/1     Completed   0          19s
      supertubes-control-plane-549f55595f-8pd2z   2/2     Running     0          3h46m