How to run SDM with upstream Istio in the same Kubernetes cluster
Streaming Data Manager deploys and manages its own Istio service mesh (see Apache Kafka over Istio for more details). In some cases, you might want to have an upstream Istio mesh to run in the same Kuberentes cluster - this page guides you to accomplish this setup.
- Streaming Data Manager supports Istio 1.13.x, therefore, the CRDs that upstream Istio uses need to be compatible with Istio 1.13.
For example, since there are no breaking changes within the Istio CRDs between Istio 1.11.x and 1.13.x, the users cun run upstream Istio 1.11 with Streaming Data Manager without issues
For other older versions of Istio, the users need to check if ther are breaking changes in the CRDs between the older version and Istio 1.13. If there are, the users need to upgrade the upstream Istio to the compatible version
The Streaming Data Manager Istio must be installed in a namespace different from the upstream Istio. It should also have
fullNameOverridespecified as clusterRoles may clash between the two Istio versions. One way to accomplish this is to modify the ApplicationManifest as follows, and then install the Streaming Data Manager via Operator mode:
kubectl apply -n supertubes-control-plane -f- <<EOF apiVersion: supertubes.banzaicloud.io/v1beta1 kind: ApplicationManifest metadata: name: applicationmanifest spec: ... istioOperator: enabled: true # Assuming the upstream Istio is not installed in the sdm-istio-system namespace namespace: sdm-istio-system valuesOverride: |- fullnameOverride: sdm-istio-operator ... EOF
When running Streaming Data Manager with upstream Istio, there are potentially two istio-operator instances running in the Kuberentes cluster - one for each Istio mesh. If this is the case, you must make sure that:
- The ownership of all Istio CRDs belongs to only one of these operators, otherwise upgrading Istio can lead to errors.
- The other operator must not modify the Istio CRDs.
Managing Istio CRDs
Managing Istio CRDs can be tricky when users try to run Streaming Data Manager with upstream Istio because the Istio CRDs are cluster-wide and shared resources between the two Istios.
There are two scenarios when it comes to deploying the Streaming Data Manager with upstream Istio:
Upstream Istio is deployed before Streaming Data Manager
In this case, upstream Istio can be installed as usual (either via
istio-operator). Installing Streaming Data Manager does not modify or change the ownership of the existing Istio CRDs.
If you want Streaming Data Manager to manage the Istio CRDs, you can transfer the CRDs ownership from upstream Istio to Streaming Data Manager’s istio-operator by completing the following steps:
Remove the following labels from the CRDs:
install.operator.istio.io/owning-resource(this label is only present if the upstream Istio is deployed using istio-operator)
operator.istio.io/component(removing this would prevent
istioctl x uninstall --purgefrom deleting the CRDs)
app.kubernetes.io/managed-by: istio-operatorlabel to the Istio CRDs, for example:
Name: authorizationpolicies.security.istio.io Namespace: Labels: ... app: istio-pilot app.kubernetes.io/managed-by: istio-operator ... ...
Streaming Data Manager is deployed before upstream Istio
In this case, you need to configure the upstream Istio to not modify the existing Istio CRDs, and to keep them under the ownership of Streaming Data Manager:
If upstream Istio is deployed with
istioctl, add the
--set values.base.enableIstioConfigCRDs=falseflag for Istio installation. For example, the following command deploys upstream Istio without modifying the existing CRDs:
istioctl install --set values.base.enableIstioConfigCRDs=false
If upstream Istio is deployed with the
istio-operator, use the following configuration to prevent the upstream
istio-operatorfrom modifying the existing Istio CRDs created by Streaming Data Manager:
kubectl apply -f - <<EOF apiVersion: install.istio.io/v1alpha1 kind: IstioOperator metadata: namespace: istio-system name: example-istiocontrolplane spec: ... values: base: # Tell upstream istio-operator to not create/modify Istio CRDs enableIstioConfigCRDs: false ... EOF
Deployment and configuration
To ensure that the two Istio deployments run side-by-side without interfering with each other when injecting sidecar containers into workloads:
- The Istio Control Planes must be configured with different revisions so the revision uniquely identifies them.
- The namespaces or workloads (pods) should be marked with the istio.io/rev=<Istio Control Plane revision> label instead of the
For Streaming Data Manager to use Istio Control Plane revisions, the following must be present in the
istiocontrolplanes.servicemesh.cisco.com custom resource, which represents the Istio Control Plane deployed by Streaming Data Manager:
apiVersion: servicemesh.cisco.com/v1alpha1 kind: IstioControlPlane metadata: name: icp-v115x namespace: sdm-istio-system spec: ... version: "1.15.3" mode: ACTIVE distribution: cisco meshID: sdm meshConfig: defaultConfig: proxyMetadata: # To have CA certs changes to be published dynamically to Istio proxies PROXY_CONFIG_XDS_AGENT: "true" ...
The Istio Control Plane deployed by Streaming Data Manager is automatically configured with the <icp-cr-name>.<icp-cr-namespace> revision label (for example,
icp-v115x.sdm-istio-system). Similarly, the namespaces where Streaming Data Manager deploys its components are automatically labeled with istio.io/rev=<istio-cr-name>.<istio-cr-namespace> (for example,
That way the workloads running in the Istio mesh are bound to a specific Istio Control Plane.
In order to allow workloads from the two meshes to communicate with each other seamlessly and securely, the two meshes have to be configured to trust each other. For this, the Istio Control Planes must be aware of each others CA certificate (or certificate chain in case intermediate CA is used).
On the Streaming Data Manager side, the
istiocontrolplanes.servicemesh.cisco.com custom resource must have the following setting:
apiVersion: servicemesh.cisco.com/v1alpha1 kind: IstioControlPlane metadata: name: icp-v115x namespace: sdm-istio-system spec: ... istiod: deployment: env: - name: ISTIO_MULTIROOT_MESH value: "true" meshConfig: defaultConfig: proxyMetadata: PROXY_CONFIG_XDS_AGENT: "true" caCertificates: - pem: | # CA certificate of the upstream Istio Control Plane running in PEM format ...
Note: If there are more CAs to trust just add a separate
- pem:item for each of them.
On the Upstream Istio Control Plane side, assuming that the
istiooperators.install.istio.io custom resource was used to deploy the Istio Control Plane, the following configuration is needed:
apiVersion: install.istio.io/v1alpha1 kind: IstioOperator ... spec: ... components: ... pilot: k8s: env: - name: ISTIO_MULTIROOT_MESH value: "true" ... meshConfig: caCertificates: - pem: | <ca-cert of the Banzai Cloud Istio Control Plane in PEM format - for example, the external-ca-cert in the sdm-istio-system namespace> defaultConfig: proxyMetadata: PROXY_CONFIG_XDS_AGENT: "true" # enable config distribution through XDS
Note: If you configure the upstream Istio to use the CSR operator - which comes with Streaming Data Manager - as an external CA, setting up the trust between the two meshes may become even simpler.