Operator Mode
The following example demonstrates how to enable SDM resource synchronization between two Kubernetes clusters in Operator mode.
Install Cluster Registry
-
Installing Cluster Registry in operator mode follows the same principle as installing other components: via modifying and reconciling the Streaming Data Manager ApplicationManifest custom resource. For example, the following configuration in the ApplicationManifest CR would install Cluster Registry controller under the “cluster-registry” namespace
apiVersion: supertubes.banzaicloud.io/v1beta1 kind: ApplicationManifest metadata: name: applicationmanifest spec: ... clusterregistry: enabled: true namespace: cluster-registry ...
-
After the Cluster Registry controller is up and running, deploy a local Cluster CR to represent your Kubernetes cluster. Note the following points:
- The
spec.clusterID
field needs to be the UID of namespacekube-system
to ensure the uniqueness of the Cluster CR. - The
spec.authInfo.secretRef
holds information that describes how the peer clusters can get credentials to access the cluster. Therefore, the name of thespec.authInfo.secretRef
is recommended to be the name of the Cluster CR to avoid confusion. - The server address under
spec.kubernetesApiEndpoints
must be accessible from the peer clusters.
kubectl apply -f -<<EOF apiVersion: clusterregistry.k8s.cisco.com/v1alpha1 kind: Cluster metadata: name: cluster1 spec: clusterID: d51418e7-547e-468b-adbc-9c2968389184 authInfo: secretRef: name: cluster1 namespace: cluster-registry kubernetesApiEndpoints: - serverAddress: https://3.13.240.171:6443 EOF
- The
-
After the installation is finished, you can attach Kubernetes clusters and detach Kubernetes clusters.
Attach Kubernetes clusters
There are several steps for attaching one Kubernetes cluster to another and have them fully synchronized.
-
First of all, we need to deploy a peer
Cluster
CR in each cluster and the associated secret to each Kubernetes cluster to make the Cluster Registry controller aware the existence of the other cluster:-
On the first Kubernetes cluster
-
Copy the local Cluster CR contents from the second Kubernetes cluster and deploy it to the first Kubernetes cluster:
kubectl apply -f -<<EOF apiVersion: clusterregistry.k8s.cisco.com/v1alpha1 kind: Cluster metadata: name: cluster2 spec: # clusterID is the UID of kube-system namespace in the second Kubernetes cluster clusterID: ac21678f-9ebf-4ee1-83d4-3cca0931af19 authInfo: secretRef: name: cluster2 namespace: cluster-registry kubernetesApiEndpoints: - serverAddress: https://18.190.53.82:6443 EOF
-
Copy the secret that is specified in the peer
Cluster
from the second Kuberentes cluster and deploy it to the first Kubernetes cluster:kubectl apply -f -<<EOF apiVersion: v1 data: kubeconfig: YXBpVmVyc2lvbjogdjEKY2x1c3RlcnM6Ci0gY2x1c3RlcjoKICAgIGNlcnRpZmljYXRlLWF1dGhvcml0eS1kYXRhOiBMUzB0TFMxQ1JVZEpUaUJEUlZKVVNVWkpRMEZVUlMwdExTMHRDazFKU1VSUVZFTkRRV2xYWjBGM1NVSkJaMGxWVDNWaFZtTTRNVFU1YlZKMlJrRkRRbHB4TW5sd1JWTktNV2h6ZDBSUldVcExiMXBKYUhaalRrRlJSVXdLUWxGQmQwZHFSVmxOUWxsSFFURlZSVUY0VFZCWk1uZ3hZek5TYkdOcE1ETk9SRlYzVEZkT2FFMUNORmhFVkVsNVRVUk5lVTFVU1hoTlZHdDNUVEZ2V0FwRVZFa3pUVVJOZVUxRVNYbE5WR3Q2VFRGdmQwZEVSVmROUWxGSFFURlZSVUY0VFU1aE0xWnBXbGhLZFZwWVVteGplVEZxV1ZSRFEwRlRTWGRFVVZsS0NrdHZXa2xvZG1OT1FWRkZRa0pSUVVSblowVlFRVVJEUTBGUmIwTm5aMFZDUVV0UlNuVjJPR3BMWnpoQk5XdDZlVWR0TVZOSmNDdDNlRVJGUzFWc09FOEtTVXBSSzNCdEwwOHZZMDF6UWpSRU1DOW5WVEZLVGxoR01tUk1MMU5VTm1wbFVFUnJXakZsTVZSUVZHMDNhVkppZFhacGRWQk9TV1J6U2tZMVpHRk9ZUXBOY0V4TVkydEpUR1ZCVkhweFNsVkZabXR0ZFVVdmFsaFdXa2hEYW1KMmRVdE1TbFZxZVRSR1JVUlhNalZsWlVReVJsUnRXVlV3TldkbVp6bFpXamRPQ2tZeGVFUjBlalJ5TDJWbWN6bGpOM1J0UzJsUmF6WlVlbnBtUWtOSWRUQkRjR2xXWmxSMlpFdzRURFZYWTFsdFkxaGlVbEZhWm1SQ1RsVjRjblZSTHpRS1R6ZHNhVmt2ZEZsRVRDODRaekJ1YlRGblNUWlhRa0ZSWWxWbU1sRTRkMEkxY3psNmVtRkVNWEpFYXk5b2ExRlRiR2d3TmxSSU5uRkNka1JuWmpOWU5ncGhaV1kzUTFKNk9XdEthRlpUU1c5aFEzTmtiMHRxWms4M2JIZFlSRm95Vm1wSFRWbEhNME51V1dWRmJteHJhVkIzVXpsUlRUZFZRMEYzUlVGQllVNDVDazFJYzNkRVoxbEVWbEl3VUVGUlNDOUNRVkZFUVdkRlIwMUJPRWRCTVZWa1JYZEZRaTkzVVVaTlFVMUNRV1k0ZDBoUldVUldVakJQUWtKWlJVWlFlWFVLUTBWSVNtRktVMlZqY213eFJuQjRVMkU0TlU1dGNIaHlUVUk0UjBFeFZXUkpkMUZaVFVKaFFVWkJUakFyWlZSdksxUk9OMFoxUkcxa2FrMUpTREp2TXdwNGVFMUtUVUpuUjBFeFZXUkZVVkZTVFVFclEwUlhkREZaYlZaNVltMVdNRnBZVFhSWk1rVjNSRkZaU2t0dldrbG9kbU5PUVZRlRFSlJRVVJuWjBWQ0NrRk1XbWxQYldSbk1UZG5iVUUzZWtWWU1sUkNURmc0TkZNeVJUQTNUQzlLUjJsNFpubEpiazFoVG5wb1NUVm9MeTlWYTBsV1VVeHdablp1T1ZRclZGVUtkMWQwWTFWRlUyTkZia0ZVYm14WFRVODVNMVZsTDNaNk5YRjRiVWMwVFhSMVNUWlpiV1ZTVDJGSWF6aGFkM016VUUxemRYVndkRVEwTWs4elFraFhTZ3BVU1ROdlMwbFpVbUpqWjNwa2NEbEVjSFJzYjBOT2J6TlVlRkJWWWs5R1V6aFNSRTh4S3pSWVNsVkxZV1J1U21wek9FbEtWMEpCTlU5b2VUTXdjWGw1Q2t4QmNXWTNURU52T1dzeWQyZHhSMUV3Uml0amFWbzRVaXRJT1VSaVJYVXlTa2xZWWxkcFlWVk5RMlZaWTBjdmJIZDBhRFo0VjJGaVIzWkVOMjlzVkRBS1dXaDFTVWt5UVZnelJHRnFkVlZUTkVSTU9WaHBiSEZSVlZWSFRVYzFiRzlPWVRCalZGWm1UemhtUzBFdmRXVnBXRlJYVEhScVQyNUdhMko1WjFsNU5RcFBWV2xDY0ZNMk9YSnNaM1Z0UTNWV1EyVlJXa1Z6WnowS0xTMHRMUzFGVGtRZ1EwVlNWRWxHU1VOQlZFVXRMUzB0TFFvdExTMHRMVUpGUjBsT0lFTkZVbFJKUmtsRFFWUkZMUzB0TFMwS1RVbEpSRkZVUTBOQmFXMW5RWGRKUWtGblNWVmpXRXc0U25aa2FXMDVkRXBKZFcxTVNscExkV0pYVG1STWFXdDNSRkZaU2t0dldrbG9kbU5PUVZGRlRBcENVVUYzUjJwRldVMUNXVWRCTVZWRlFYaE5VRmt5ZURGak0xSnNZMmt3TTA1RVZYZE1WMDVvVFVJMFdFUlVTWGxOUkUxNVRWUkplRTFVYTNkTk1XOVlDa1JVU1ROTlJFMTVUVVJKZVUxVWEzcE5NVzkzUjJwRldVMUNXVWRCTVZWRlFYaE5VRmt5ZURGak0xSnNZMmt3TTA1RVZYZE1WMDVvVFVsSlFrbHFRVTRLUW1kcmNXaHJhVWM1ZHpCQ1FWRkZSa0ZCVDBOQlVUaEJUVWxKUWtOblMwTkJVVVZCZVZkTlVURlRja1ZpVjJGS2VFUldSakpuTVV0RmFHcDFaMmh6VWdwdFJYTm5NWE5tT0VGVWQwZHZaRVpQVVRsUU0zZDVWMlkzY1c1NmVreFpUUzgyVjNOS2JHcE1jM0JLVEVkaVMyTkpPRVV2V0hab1pVNVRUbmRsZFdab0NraGlUM1ZsVERkNFRHSmxhMlZwT0hvd1pqaHdNbkZrUXpKQlowd3pTR3RaYkZkNlUyWmxaMUZMYmxsWlFVSlFNME0zV1hZMFYyVmtaRXRxYlZkWlRVWUtjVmxXU2xZd1MwOUpTbEo0S3k5emJFWlhaVE4yZFZSclEwZHlTM1ZuYzB4dlNWSjVPRnBxYWpkVVRHRm1lR1ZtVW5WbFNXTlJhWHBJVW5KVFMyeFJMd3BsVlRKeFdtUjFja2RoTjNwd2JVZHdMMEpRYzFWVFpsSnZNVE5JUm1aTGEwVllPV2h6SzNGeWRuQnhaWGRqZGtweFMwVkdRbEpaTkZoME1sZFVlbGRFQ21KaWVHTmxUVkkzUWs4MWFIWkNXWGRUU0dGVU5IaGpUM0JJVkd4V1FUZHNUMmhtUWxCbFJFcFlRMWQxYlZkWU16TTJUMmxzUkdoeVFuZEpSRUZSUVVJS2J6TTRkMlpVUVU5Q1owNVdTRkU0UWtGbU9FVkNRVTFEUVZGWmQwUjNXVVJXVWpCVVFWRklMMEpCVlhkQmQwVkNMM3BCWkVKblRsWklVVFJGUm1kUlZRcEJNMVExTlU5cU5VMHpjMWMwVDFveVRYZG5abUZxWmtoRmQydDNTSGRaUkZaU01HcENRbWQzUm05QlZVRXpWRFUxVDJvMVRUTnpWelJQV2pKTmQyZG1DbUZxWmtoRmQydDNSMmRaUkZaU01GSkNRazEzUlZsSlVGa3llREZqTTFKc1kya3dNMDVFVlhkTVYwNW9UVUV3UjBOVGNVZFRTV0l6UkZGRlFrTjNWVUVLUVRSSlFrRlJRbWM1ZVdkV1VpdEVWRlV3WTNjd1UxazNSSGt5UmpacmRXTm1VVXBuWkZaU2R6QndaREUxWVVwbGNtODBWVEZ3VVdwRFpuaE9URU5wTkFwNVdqWk9hVU53VEdGSlJXMXJZMDVhVVhKTllWbzJZMHhEU2pGTFRVZElOamwyUkdSelNsWlFTR3MzV21rMFFXMXdjazUzWnpOVWRVWnFRVWhCU1NzeENrdFlMMG8xWTNSNmVtcERXWGxzYUU0MVpubFllRmxpTlhRd1UwUlZiMHhRZGtwVGQxUmxUVUZxYVZSalJVWTJUMlV6YkZCd2NVSklaV2xrUlU5MFNTOEtRbk5DVVhGcGRWSkpWVXd2YldOelVGZzNOWEZTTTAxNVMzbG9lSHBGWjNwR05EUXZkRkZHVUN0WVJFeEZRMGh5YWxsbllTOU9VVlZYTmpKQ01ITm9id3A1VVdoTVZYRTNTaXRxTjFSRGVuZDBRV0l5VERoRVQxTXlTVk13T0dKa1RGVktkRXBJVmxnNWFUaEpiRXMxVG5KQ1RFeEhURUZrUlVkVU1WRjVjVm80Q2pka1JVRTNiMlZQV0ROb05GSjJNak13Y3l0bVdEZE1jelZ6VEc0S0xTMHRMUzFGVGtRZ1EwVlNWRWxHU1VOQlZFVXRMUzB0TFE9PQogICAgc2VydmVyOiBodHRwczovLzE4LjExNi4xNi42NTo2NDQzCiAgbmFtZTogc2RtLWNsdS1yZWctYQpjb250ZXh0czoKLSBjb250ZXh0OgogICAgY2x1c3Rlcjogc2RtLWNsdS1yZWctYQogICAgdXNlcjogY2x1c3Rlci1yZWdpc3RyeS1jb250cm9sbGVyLXJlYWRlcgogIG5hbWU6IHNkbS1jbHUtcmVnLWEKY3VycmVudC1jb250ZXh0OiBzZG0tY2x1LXJlZy1hCmtpbmQ6IENvbmZpZwpwcmVmZXJlbmNlczoge30KdXNlcnM6Ci0gbmFtZTogY2x1c3Rlci1yZWdpc3RyeS1jb250cm9sbGVyLXJlYWRlcgogIHVzZXI6CiAgICB0b2tlbjogZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNkltOXJjR2RvU0VoWmJIVXdhMlpuTmxBemREVjVNMnM0UVVwQlNVRlliMjFMUkZOUU1WUjBiRFpzZVdjaWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUpqYkhWemRHVnlMWEpsWjJsemRISjVJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJbU5zZFhOMFpYSXRjbVZuYVhOMGNua3RZMjl1ZEhKdmJHeGxjaTF5WldGa1pYSXRkRzlyWlc0dGQyMDRabllpTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxjblpwWTJVdFlXTmpiM1Z1ZEM1dVlXMWxJam9pWTJ4MWMzUmxjaTF5WldkcGMzUnllUzFqYjI1MGNtOXNiR1Z5TFhKbFlXUmxjaUlzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVnlkbWxqWlMxaFkyTnZkVzUwTG5WcFpDSTZJalpsTURSa05qVm1MVGd3TjJJdE5Ea3hOUzFoT1RnMExUaGpNVEF6T0RRMk0yWTBNeUlzSW5OMVlpSTZJbk41YzNSbGJUcHpaWEoyYVdObFlXTmpiM1Z1ZERwamJIVnpkR1Z5TFhKbFoybHpkSEo1T21Oc2RYTjBaWEl0Y21WbmFYTjBjbmt0WTI5dWRISnZiR3hsY2kxeVpXRmtaWElpZlEub09udzBQU29RTEUxNE1SUGhwNzMxM01udGlGUEdvUVBhd0tUbThNV2hVbmh3Y0xOdkRSYVZzdlhhYXlIb1k4Q1dLY3c2QW55TXJJZmhrUVpSQ3A3eTlhcnlwT1FCTnJZbk0tS25vTVhCV21RMzEzQjRuYzFZMXlFR0Jhdko2c1d4YmdVa3ZxQ1FRNkM0VjhpaVBoVGtTUHNfMmcxSkx2UlNhZmtpRi1LZlhnZzVzMF9sOXJsQTdpaS0xMUh4MEdJOHpQNFFMaGRIeTA3UXBsbjE3enFWbXQ0Nkozejc4RGhtM2UyWWdOOHZ2OWZsTGVTN2xLV215R2Nwak9LNUhGRVNtTkVFOGs0YzlfazRoYjF4bzJXQmhoQWxyMjJIeWhMeXF0clFhamhJRnBjallCVDl3N1ZFc0hRTTNKWFBwbmpaWllzaUR0a2pHUWxuLUNhVzZLZ0lBCg== kind: Secret metadata: name: cluster2 namespace: cluster-registry type: k8s.cisco.com/cluster-registry-secret EOF
Note: the name and namespace of the secret need to match the the contents under
spec.authInfo.secretRef
in the peerCluster
CR created in the previous step
-
-
On the second Kubernetes cluster:
Repeat steps on the first Kubernetes cluster to copy the contents of the first Kubernetes cluster’s local
Cluster
CR and the associated secret.
-
-
At this stage, the Cluster Registry controller on both clusters are aware of the existence of the other Kuberentes cluster and tries to perform synchronization on certain resources. The following example shows the steps to have both Kubernetes clusters to synchronize the
KafkaCluster
resource-
Create necessary RBAC resources (
ClusterRole
,ServiceAccount
andClusterRoleBinding
) so the Cluster Registry controller have the permission to perform read operations from the peer Kubernetes cluster, and write operations to the local Kubernetes clusterkubectl apply -f -<<EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: cluster-registry.k8s.cisco.com/controller-aggregated: "true" name: cluster-registry-sdm rules: - apiGroups: - kafka.banzaicloud.io resources: - kafkaclusters - kafkaclusters/status verbs: - '*' EOF
kubectl apply -f -<<EOF apiVersion: v1 kind: ServiceAccount metadata: name: cluster-registry-sdm namespace: supertubes-system EOF
kubectl apply -f -<<EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: cluster-registry-sdm roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-registry-sdm subjects: - kind: ServiceAccount name: cluster-registry-sdm namespace: supertubes-system EOF
kubectl apply -f -<<EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: cluster-registry.k8s.cisco.com/reader-aggregated: "true" name: cluster-registry-sdm-reader rules: - apiGroups: - kafka.banzaicloud.io resources: - kafkaclusters - kafkaclusters/status verbs: - get - list - watch EOF
kubectl apply -f -<<EOF apiVersion: v1 kind: ServiceAccount metadata: name: cluster-registry-sdm-reader namespace: supertubes-system EOF
kubectl apply -f -<<EOF apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: cluster-registry-sdm-reader roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-registry-sdm-reader subjects: - kind: ServiceAccount name: cluster-registry-sdm-reader namespace: supertubes-system EOF
-
Deploy necessary Cluster Registry CRs (
ClusterFeatuer
andResourceSyncRule
):kubectl apply -f -<<EOF apiVersion: clusterregistry.k8s.cisco.com/v1alpha1 kind: ClusterFeature metadata: labels: cluster-registry-controller.k8s.cisco.com/core-sync-resource: "true" name: sdm-core-resources spec: featureName: sdm-core-resources-source EOF
kubectl apply -f -<<EOF apiVersion: clusterregistry.k8s.cisco.com/v1alpha1 kind: ResourceSyncRule metadata: annotations: cluster-registry.k8s.cisco.com/resource-sync-disabled: "true" labels: cluster-registry-controller.k8s.cisco.com/core-sync-resource: "true" name: sdm-core-resources-kafka-clusters-sink spec: clusterFeatureMatch: - featureName: sdm-core-resources-source groupVersionKind: group: kafka.banzaicloud.io kind: KafkaCluster version: v1beta1 rules: - mutations: overrides: - path: /metadata/name type: replace value: '{{ printf "%s-%s-%s" .Object.GetName .Cluster.GetName (trunc 4 (sha256sum .Cluster.GetName))}}' syncStatus: true EOF
Note:
- The above steps to create the RBAC and Cluster Registry resources need to be run in both Kubernetes clusters.
- If you would like to synchronize other resources, update the
rules
section in theClusterRole
CR, and create aResourceSyncRule
CR withspec.groupVersionKind
that matches the resource you’d like to synchronize. - The
ClusterFeature
CR can be shared between multipleResourceSyncRule
resources.
-
Detach Kubernetes clusters
To detach a specific Kubernetes cluster from the cluster group, delete the RBAC resouces (ClusterRole
, ServiceAccount
and ClusterRoleBinding
), the Cluster Registry CRs (ClusterFeatuer
and ResourceSyncRule
) that were created while attaching, and the synchronized resources (in this example, KafkaCluster
) from both Kuberentes clusters.
CAUTION:
The Cluster Registry CRs (ClusterFeatuer
and ResourceSyncRule
) must be deleted before you can delete the synchronized resources, otherwise the Cluster Registry controller will keep synchonizing the resources you specified in the ResourceSyncRule
If there is no other resource being synchronized between the detaching cluster (let’s say cluster2
) and other clusters in the cluster group (let’s say only cluster1
is remained in the cluster group), you may also delete the peer Cluster CR that represents cluster2
and the associated secret from clsuter1
, and delete the peer Cluster CR that represents cluster1
and the associated secrets from cluster2
Uninstall Cluster Registry
To uninstall Cluster Registry disable the corresponding component in ApplicationManifest and the operator will uninstall the Cluster Registry controller in the next reconcilation:
apiVersion: supertubes.banzaicloud.io/v1beta1
kind: ApplicationManifest
metadata:
name: applicationmanifest
spec:
...
clusterregistry:
enabled: false
namespace: cluster-registry
...
Note: If there is no other component using Cluster Registry, you may want to clean up the previously created local
Cluster
CR.