How to run SDM with upstream Istio in the same Kubernetes cluster

Streaming Data Manager deploys and manages its own Istio service mesh (see Apache Kafka over Istio for more details). In some cases, you might want to have an upstream Istio mesh to run in the same Kuberentes cluster - this page guides you to accomplish this setup.

Requirements

Streaming Data Manager supports Istio 1.13.x, therefore, the CRDs that upstream Istio uses need to be compatible with Istio 1.13.

For example, since there are no breaking changes within the Istio CRDs between Istio 1.11.x and 1.13.x, the users cun run upstream Istio 1.11 with Streaming Data Manager without issues

For other older versions of Istio, the users need to check if ther are breaking changes in the CRDs between the older version and Istio 1.13. If there are, the users need to upgrade the upstream Istio to the compatible version

The Streaming Data Manager Istio must be installed in a namespace different from the upstream Istio. It should also have fullNameOverride specified as clusterRoles may clash between the two Istio versions. One way to accomplish this is to modify the ApplicationManifest as follows, and then install the Streaming Data Manager via Operator mode:

kubectl apply -n supertubes-control-plane -f- <<EOF
apiVersion: supertubes.banzaicloud.io/v1beta1
kind: ApplicationManifest
metadata:
  name: applicationmanifest
spec:
    ...
    istioOperator:
      enabled: true
      # Assuming the upstream Istio is not installed in the sdm-istio-system namespace
      namespace: sdm-istio-system
      valuesOverride: |-
        fullnameOverride: sdm-istio-operator
    ...
EOF

When running Streaming Data Manager with upstream Istio, there are potentially two istio-operator instances running in the Kuberentes cluster - one for each Istio mesh. If this is the case, you must make sure that:
- The ownership of all Istio CRDs belongs to only one of these operators, otherwise upgrading Istio can lead to errors.
- The other operator must not modify the Istio CRDs.

Managing Istio CRDs

Managing Istio CRDs can be tricky when users try to run Streaming Data Manager with upstream Istio because the Istio CRDs are cluster-wide and shared resources between the two Istios.

There are two scenarios when it comes to deploying the Streaming Data Manager with upstream Istio:

Upstream Istio is deployed first
Streaming Data Manager is deployed first

Upstream Istio is deployed before Streaming Data Manager

In this case, upstream Istio can be installed as usual (either via istioctl or istio-operator). Installing Streaming Data Manager does not modify or change the ownership of the existing Istio CRDs.

If you want Streaming Data Manager to manage the Istio CRDs, you can transfer the CRDs ownership from upstream Istio to Streaming Data Manager’s istio-operator by completing the following steps:

Remove the following labels from the CRDs:
- install.operator.istio.io/owning-resource-namespace
- install.operator.istio.io/owning-resource (this label is only present if the upstream Istio is deployed using istio-operator)
- operator.istio.io/managed
- operator.istio.io/component (removing this would prevent istioctl x uninstall --purge from deleting the CRDs)

Add the app.kubernetes.io/managed-by: istio-operator label to the Istio CRDs, for example:

Name:  authorizationpolicies.security.istio.io
Namespace:
Labels:
  ...
  app: istio-pilot
  app.kubernetes.io/managed-by: istio-operator
  ...
...

Streaming Data Manager is deployed before upstream Istio

In this case, you need to configure the upstream Istio to not modify the existing Istio CRDs, and to keep them under the ownership of Streaming Data Manager:

If upstream Istio is deployed with istioctl, add the --set values.base.enableIstioConfigCRDs=false flag for Istio installation. For example, the following command deploys upstream Istio without modifying the existing CRDs:
istioctl install --set values.base.enableIstioConfigCRDs=false

If upstream Istio is deployed with the istio-operator, use the following configuration to prevent the upstream istio-operator from modifying the existing Istio CRDs created by Streaming Data Manager:

kubectl apply -f - <<EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: example-istiocontrolplane
spec:
  ...
  values:
    base:
      # Tell upstream istio-operator to not create/modify Istio CRDs
      enableIstioConfigCRDs: false
  ...
EOF

Deployment and configuration

To ensure that the two Istio deployments run side-by-side without interfering with each other when injecting sidecar containers into workloads:

The Istio Control Planes must be configured with different revisions so the revision uniquely identifies them.
The namespaces or workloads (pods) should be marked with the istio.io/rev=<Istio Control Plane revision> label instead of the istio-injection label.

For Streaming Data Manager to use Istio Control Plane revisions, the following must be present in the istiocontrolplanes.servicemesh.cisco.com custom resource, which represents the Istio Control Plane deployed by Streaming Data Manager:

apiVersion: servicemesh.cisco.com/v1alpha1
kind: IstioControlPlane
metadata:
  name: icp-v115x
  namespace: sdm-istio-system
spec:
  ...
  version: "1.15.3"
  mode: ACTIVE
  distribution: cisco
  meshID: sdm
  meshConfig:
    defaultConfig:
      proxyMetadata:
        # To have CA certs changes to be published dynamically to Istio proxies
        PROXY_CONFIG_XDS_AGENT: "true"
  ...

The Istio Control Plane deployed by Streaming Data Manager is automatically configured with the <icp-cr-name>.<icp-cr-namespace> revision label (for example, icp-v115x.sdm-istio-system). Similarly, the namespaces where Streaming Data Manager deploys its components are automatically labeled with istio.io/rev=<istio-cr-name>.<istio-cr-namespace> (for example, istio.io/rev=icp-v115x.sdm-istio-system). That way the workloads running in the Istio mesh are bound to a specific Istio Control Plane.

In order to allow workloads from the two meshes to communicate with each other seamlessly and securely, the two meshes have to be configured to trust each other. For this, the Istio Control Planes must be aware of each others CA certificate (or certificate chain in case intermediate CA is used).

On the Streaming Data Manager side, the istiocontrolplanes.servicemesh.cisco.com custom resource must have the following setting:

apiVersion: servicemesh.cisco.com/v1alpha1
kind: IstioControlPlane
metadata:
  name: icp-v115x
  namespace: sdm-istio-system
spec:
  ...
  istiod: 
    deployment:
      env:
      - name: ISTIO_MULTIROOT_MESH
        value: "true"
  meshConfig:
    defaultConfig:
      proxyMetadata:
        PROXY_CONFIG_XDS_AGENT: "true"
    caCertificates:
      - pem: |
          # CA certificate of the upstream Istio Control Plane running in PEM format
  ...

Note: If there are more CAs to trust just add a separate - pem: item for each of them.

On the Upstream Istio Control Plane side, assuming that the istiooperators.install.istio.io custom resource was used to deploy the Istio Control Plane, the following configuration is needed:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
...
spec: 
  ...
  components: 
    ...
    pilot: 
      k8s:  
        env: 
        - name: ISTIO_MULTIROOT_MESH
          value: "true"
    ...
    meshConfig: 
      caCertificates: 
      - pem: |
          <ca-cert of the Banzai Cloud Istio Control Plane in PEM format - for example, the external-ca-cert in the sdm-istio-system namespace>
      defaultConfig: 
        proxyMetadata: 
          PROXY_CONFIG_XDS_AGENT: "true" # enable config distribution through XDS

Note: If you configure the upstream Istio to use the CSR operator - which comes with Streaming Data Manager - as an external CA, setting up the trust between the two meshes may become even simpler.