Setup Istio to use CSR operator as external CA

Streaming Data Manager allows you to use a separate CSR operator to provide certificates to Istio.

  1. Install CSR-operator. CSR-operator runs in privateCA signer mode and it generates the CA key and CA certificate into a secret by default. The name of the secret is “csr-operator-cacerts” and it can be found in the namespace where the CSR-operator has been installed.

    smm sdm csr install --namespace <csr-operator-namespace>

  2. Configure Istio to use CSR-operator as an external CA. The CSR-operator signs certificate signing requests which are generated by Istio.

    1. From the secret generated automatically by the CSR-operator (“csr-operator-cacerts” in the “csr-operator-system” namespace), create a new secret into the namespace where Istio is installed (by default, it is “istio-system”), because Istio requires that secret in another format without the CA private key.

      kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: sdm-istio-external-ca-cert
  namespace: istio-system
data:
  root-cert.pem: $(kubectl --namespace csr-operator-system get secret csr-operator-cacerts -o 'jsonpath={.data.ca_crt\.pem}')
EOF

      Expected output:

      secret/sdm-istio-external-ca-cert created

    2. Deploy the IstioControlPlane CR into your cluster.

      For OpenShift:

      kubectl apply -f - <<EOF
apiVersion: servicemesh.cisco.com/v1alpha1
kind: IstioControlPlane
metadata:
  labels:
    banzaicloud.io/managed-by: supertubes
  name: sdm-icp-v115x
  namespace: istio-system
spec:
  containerImageConfiguration:
    imagePullPolicy: Always
    imagePullSecrets:
    - name: smm-pull-secret
  distribution: cisco
  istiod:
    deployment:
      env:
        # Skip validating that the peer is from the same trust domain when mTLS is enabled in authentication policy
        - name: PILOT_SKIP_VALIDATE_TRUST_DOMAIN
          value: "true"
        # Indicate to Istiod that we use an external signer (likely to be removed and added to mesh config - from upstream Istio)
        - name: EXTERNAL_CA
          value: ISTIOD_RA_KUBERNETES_API
        # Kubernetes CA signer type (likely to be removed and added to mesh config - from upstream Istio)
        - name: K8S_SIGNER
          value: csr.banzaicloud.io/privateca
        - name: ISTIO_MULTIROOT_MESH
          value: "true"
      image: 033498657557.dkr.ecr.us-east-2.amazonaws.com/banzaicloud/istio-pilot:v1.15.3-bzc.1
  k8sResourceOverlays:
    - groupVersionKind:
        group: apps
        kind: Deployment
        version: v1
      objectKey:
        name: istiod-sdm-icp-v115x
      patches:
        - parseValue: true
          path: /spec/template/spec/volumes/-
          type: replace
          value: |
            name: external-ca-cert
            secret:
              secretName: sdm-istio-external-ca-cert
              optional: true
        - parseValue: true
          path: /spec/template/spec/containers/name=discovery/volumeMounts/-
          type: replace
          value: |
            name: external-ca-cert
            mountPath: /etc/external-ca-cert
            readOnly: true
    # Amend ClusterRole to add permission for istiod to approve certificate signing by custom signer
    - groupVersionKind:
        group: rbac.authorization.k8s.io
        kind: ClusterRole
        version: v1
      objectKey:
        name: istiod-sdm-icp-v115x-istio-system
      patches:
        - parseValue: true
          path: /rules/-
          type: replace
          value: |
            apiGroups:
            - certificates.k8s.io
            resourceNames:
            - csr.banzaicloud.io/privateca
            resources:
            - signers
            verbs:
            - approve
  meshConfig:
    defaultConfig:
      proxyMetadata:
        PROXY_CONFIG_XDS_AGENT: "true"
    enableAutoMtls: true
    protocolDetectionTimeout: 5s
  meshID: sdm
  mode: ACTIVE
  proxy:
    image: 033498657557.dkr.ecr.us-east-2.amazonaws.com/banzaicloud/istio-proxyv2:v1.15.3-bzc-kafka.0
  proxyInit:
    cni:
      binDir: /var/lib/cni/bin
      chained: false
      confDir: /etc/cni/multus/net.d
      confFileName: istio-cni-sdm-icp-v115x-istio-system.conf
      daemonset:
        image: 033498657557.dkr.ecr.us-east-2.amazonaws.com/banzaicloud/istio-install-cni:v1.15.3-bzc.1
        securityContext:
          privileged: true
      enabled: true
    image: 033498657557.dkr.ecr.us-east-2.amazonaws.com/banzaicloud/istio-proxyv2:v1.15.3-bzc-kafka.0
  telemetryV2:
    enabled: true
  version: 1.15.3
EOF

      For Kubernetes:

      kubectl create -f - <<EOF
apiVersion: servicemesh.cisco.com/v1alpha1
kind: IstioControlPlane
metadata:
  labels:
    banzaicloud.io/managed-by: supertubes
  name: sdm-icp-v115x
  namespace: istio-system
spec:
  containerImageConfiguration:
    imagePullPolicy: Always
    imagePullSecrets:
    - name: smm-pull-secret
  distribution: cisco
  istiod:
    deployment:
      env:
        # Skip validating the peer is from the same trust domain when mTLS is enabled in authentication policy
        - name: PILOT_SKIP_VALIDATE_TRUST_DOMAIN
          value: "true"
        # Indicate to Istiod that we use an external signer (likely to be removed and added to mesh config - from upstream Istio)
        - name: EXTERNAL_CA
          value: ISTIOD_RA_KUBERNETES_API
        # Kubernetes CA signer type (likely to be removed and added to mesh config - from upstream Istio)
        - name: K8S_SIGNER
          value: csr.banzaicloud.io/privateca
        - name: ISTIO_MULTIROOT_MESH
          value: "true"
      image: 033498657557.dkr.ecr.us-east-2.amazonaws.com/banzaicloud/istio-pilot:v1.15.3-bzc.1
  k8sResourceOverlays:
    - groupVersionKind:
        group: apps
        kind: Deployment
        version: v1
      objectKey:
        name: istiod-sdm-icp-v115x
      patches:
        - parseValue: true
          path: /spec/template/spec/volumes/-
          type: replace
          value: |
            name: external-ca-cert
            secret:
              secretName: sdm-istio-external-ca-cert
              optional: true
        - parseValue: true
          path: /spec/template/spec/containers/name=discovery/volumeMounts/-
          type: replace
          value: |
            name: external-ca-cert
            mountPath: /etc/external-ca-cert
            readOnly: true
    # Amend ClusterRole to add permission for istiod to approve certificate signing by custom signer
    - groupVersionKind:
        group: rbac.authorization.k8s.io
        kind: ClusterRole
        version: v1
      objectKey:
        name: istiod-sdm-icp-v115x-istio-system
      patches:
        - parseValue: true
          path: /rules/-
          type: replace
          value: |
            apiGroups:
            - certificates.k8s.io
            resourceNames:
            - csr.banzaicloud.io/privateca
            resources:
            - signers
            verbs:
            - approve
  meshConfig:
    defaultConfig:
      proxyMetadata:
        PROXY_CONFIG_XDS_AGENT: "true"
    enableAutoMtls: true
    protocolDetectionTimeout: 5s
  meshID: sdm
  mode: ACTIVE
  proxy:
    image: 033498657557.dkr.ecr.us-east-2.amazonaws.com/banzaicloud/istio-proxyv2:v1.15.3-bzc-kafka.0
  telemetryV2:
    enabled: true
  version: 1.15.3
EOF

      Expected output:

      istiocontrolplane.servicemesh.cisco.com/sdm-icp-v115x created

    3. Check for the IstioControlPlane and pods to be available.

      kubectl get istiocontrolplanes.servicemesh.cisco.com -n istio-system sdm-icp-v115x

      Expected output:

      NAME            MODE     NETWORK    STATUS      MESH EXPANSION   EXPANSION GW IPS   ERROR   AGE
sdm-icp-v115x   ACTIVE   network1   Available                                               5m21s

      kubectl get pods -n istio-system

      Expected output:

      For OpenShift:

      NAME                                    READY   STATUS    RESTARTS   AGE
istio-cni-node-sdm-icp-v115x-2mkxn      1/1     Running   0          2m37s
istio-cni-node-sdm-icp-v115x-6t6lx      1/1     Running   0          2m37s
istio-cni-node-sdm-icp-v115x-7nxqs      1/1     Running   0          2m37s
istio-cni-node-sdm-icp-v115x-htgzw      1/1     Running   0          2m37s
istio-cni-node-sdm-icp-v115x-mdrvj      1/1     Running   0          2m37s
istio-cni-node-sdm-icp-v115x-mk6vh      1/1     Running   0          2m37s
istio-cni-node-sdm-icp-v115x-mstzx      1/1     Running   0          2m37s
istio-cni-node-sdm-icp-v115x-qwlvz      1/1     Running   0          2m37s
istio-cni-node-sdm-icp-v115x-rjlrz      1/1     Running   0          2m37s
istio-cni-node-sdm-icp-v115x-tk5xv      1/1     Running   0          2m37s
istio-cni-node-sdm-icp-v115x-x88ls      1/1     Running   0          2m37s
istio-cni-node-sdm-icp-v115x-xht6n      1/1     Running   0          2m37s
istio-cni-node-sdm-icp-v115x-xv6gw      1/1     Running   0          2m37s
istio-operator-5d7cb59c9-g2htw          2/2     Running   0          7m29s
istiod-sdm-icp-v115x-54f6c69775-nfs64   1/1     Running   0          2m42s

      For Kubernetes:

      istio-operator-5d7cb59c9-5q6dx          2/2     Running   0          114m
istiod-sdm-icp-v115x-54f6c69775-786bj   1/1     Running   0          5m43s

    4. Create the istiomesh-ca-trust-extension-script ConfigMap.

      kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: istiomesh-ca-trust-extension-script
  namespace: supertubes-control-plane
data:
  run.sh: |-
    #!/bin/sh
    # Fill these fields properly----------------------
    export CA_SECRET_NAMESPACE="istio-system"
    export CA_SECRET_NAME="sdm-istio-external-ca-cert"
    # ------------------------------------------------
    export ICP_NAME="sdm-icp-v115x"
    export ICP_NAMESPACE="istio-system"
    export CA_CERT=\$(kubectl get secret -n \$CA_SECRET_NAMESPACE \$CA_SECRET_NAME -o jsonpath='{.data.root-cert\.pem}' | base64 -d | sed '\$ ! s/\$/\\\n/' | tr -d '\n')
    read -r -d '' PATCH << EOF
    {"spec": {"meshConfig": {"caCertificates": [{"pem": "\$CA_CERT"}]}}}
    EOF
    read -r -d '' INSERT_PATCH << EOF
    [{"op": "add", "path": "/spec/meshConfig/caCertificates/-", "value": {"pem": "\$CA_CERT"}}]
    EOF
    kubectl patch istiocontrolplanes.servicemesh.cisco.com \$ICP_NAME -n \$ICP_NAMESPACE --type json --patch="\$INSERT_PATCH" || kubectl patch istiocontrolplanes.servicemesh.cisco.com \$ICP_NAME -n \$ICP_NAMESPACE --type merge --patch="\$PATCH"
EOF

      Expected output:

      configmap/istiomesh-ca-trust-extension-script created

    5. Create the istiomesh-ca-trust-extension Job.

      kubectl apply -f - <<EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: istiomesh-ca-trust-extension
  namespace: supertubes-control-plane
spec:
  completions: 1
  template:
    metadata:
      name: istiomesh-ca-trust-extension
    spec:
      containers:
      - command:
        - /scripts/run.sh
        image: lachlanevenson/k8s-kubectl:v1.16.10
        imagePullPolicy: IfNotPresent
        name: istio-trust-extension-job
        volumeMounts:
        - mountPath: /scripts
          name: run
          readOnly: false
      dnsPolicy: ClusterFirst
      restartPolicy: Never
      serviceAccount: supertubes-control-plane
      serviceAccountName: supertubes-control-plane
      volumes:
      - configMap:
          defaultMode: 365
          name: istiomesh-ca-trust-extension-script
        name: run
EOF

      Expected output:

      job.batch/istiomesh-ca-trust-extension created

      Check if the job ran successfully.

      kubectl get pods -n supertubes-control-plane

      Expected output:

      NAME                                        READY   STATUS      RESTARTS   AGE
istiomesh-ca-trust-extension-4r9sr          0/1     Completed   0          19s
supertubes-control-plane-549f55595f-8pd2z   2/2     Running     0          3h46m