Operator Mode

The following example demonstrates how to enable SDM resource synchronization between two Kubernetes clusters in Operator mode.

Install Cluster Registry

  1. Installing Cluster Registry in operator mode follows the same principle as installing other components: via modifying and reconciling the Streaming Data Manager ApplicationManifest custom resource. For example, the following configuration in the ApplicationManifest CR would install Cluster Registry controller under the “cluster-registry” namespace

    apiVersion: supertubes.banzaicloud.io/v1beta1
kind: ApplicationManifest
metadata:
  name: applicationmanifest
spec:
...
  clusterregistry:
    enabled: true
    namespace: cluster-registry
...

  2. After the Cluster Registry controller is up and running, deploy a local Cluster CR to represent your Kubernetes cluster. Note the following points:

    • The spec.clusterID field needs to be the UID of namespace kube-system to ensure the uniqueness of the Cluster CR.
    • The spec.authInfo.secretRef holds information that describes how the peer clusters can get credentials to access the cluster. Therefore, the name of the spec.authInfo.secretRef is recommended to be the name of the Cluster CR to avoid confusion.
    • The server address under spec.kubernetesApiEndpoints must be accessible from the peer clusters.
    kubectl apply -f -<<EOF
apiVersion: clusterregistry.k8s.cisco.com/v1alpha1
kind: Cluster
metadata:
  name: cluster1
spec:
  clusterID: d51418e7-547e-468b-adbc-9c2968389184
  authInfo:
    secretRef:
      name: cluster1
      namespace: cluster-registry
  kubernetesApiEndpoints:
  - serverAddress: https://3.13.240.171:6443
EOF

  3. After the installation is finished, you can attach Kubernetes clusters and detach Kubernetes clusters.

Attach Kubernetes clusters

There are several steps for attaching one Kubernetes cluster to another and have them fully synchronized.

  1. First of all, we need to deploy a peer Cluster CR in each cluster and the associated secret to each Kubernetes cluster to make the Cluster Registry controller aware the existence of the other cluster:

    1. On the first Kubernetes cluster

      • Copy the local Cluster CR contents from the second Kubernetes cluster and deploy it to the first Kubernetes cluster:

        kubectl apply -f -<<EOF
apiVersion: clusterregistry.k8s.cisco.com/v1alpha1
kind: Cluster
metadata:
  name: cluster2
spec:
  # clusterID is the UID of kube-system namespace in the second Kubernetes cluster
  clusterID: ac21678f-9ebf-4ee1-83d4-3cca0931af19
  authInfo:
    secretRef:
      name: cluster2
      namespace: cluster-registry
  kubernetesApiEndpoints:
  - serverAddress: https://18.190.53.82:6443
EOF

      • Copy the secret that is specified in the peer Cluster from the second Kuberentes cluster and deploy it to the first Kubernetes cluster:

        kubectl apply -f -<<EOF
apiVersion: v1
data:
  kubeconfig: YXBpVmVyc2lvbjogdjEKY2x1c3RlcnM6Ci0gY2x1c3RlcjoKICAgIGNlcnRpZmljYXRlLWF1dGhvcml0eS1kYXRhOiBMUzB0TFMxQ1JVZEpUaUJEUlZKVVNVWkpRMEZVUlMwdExTMHRDazFKU1VSUVZFTkRRV2xYWjBGM1NVSkJaMGxWVDNWaFZtTTRNVFU1YlZKMlJrRkRRbHB4TW5sd1JWTktNV2h6ZDBSUldVcExiMXBKYUhaalRrRlJSVXdLUWxGQmQwZHFSVmxOUWxsSFFURlZSVUY0VFZCWk1uZ3hZek5TYkdOcE1ETk9SRlYzVEZkT2FFMUNORmhFVkVsNVRVUk5lVTFVU1hoTlZHdDNUVEZ2V0FwRVZFa3pUVVJOZVUxRVNYbE5WR3Q2VFRGdmQwZEVSVmROUWxGSFFURlZSVUY0VFU1aE0xWnBXbGhLZFZwWVVteGplVEZxV1ZSRFEwRlRTWGRFVVZsS0NrdHZXa2xvZG1OT1FWRkZRa0pSUVVSblowVlFRVVJEUTBGUmIwTm5aMFZDUVV0UlNuVjJPR3BMWnpoQk5XdDZlVWR0TVZOSmNDdDNlRVJGUzFWc09FOEtTVXBSSzNCdEwwOHZZMDF6UWpSRU1DOW5WVEZLVGxoR01tUk1MMU5VTm1wbFVFUnJXakZsTVZSUVZHMDNhVkppZFhacGRWQk9TV1J6U2tZMVpHRk9ZUXBOY0V4TVkydEpUR1ZCVkhweFNsVkZabXR0ZFVVdmFsaFdXa2hEYW1KMmRVdE1TbFZxZVRSR1JVUlhNalZsWlVReVJsUnRXVlV3TldkbVp6bFpXamRPQ2tZeGVFUjBlalJ5TDJWbWN6bGpOM1J0UzJsUmF6WlVlbnBtUWtOSWRUQkRjR2xXWmxSMlpFdzRURFZYWTFsdFkxaGlVbEZhWm1SQ1RsVjRjblZSTHpRS1R6ZHNhVmt2ZEZsRVRDODRaekJ1YlRGblNUWlhRa0ZSWWxWbU1sRTRkMEkxY3psNmVtRkVNWEpFYXk5b2ExRlRiR2d3TmxSSU5uRkNka1JuWmpOWU5ncGhaV1kzUTFKNk9XdEthRlpUU1c5aFEzTmtiMHRxWms4M2JIZFlSRm95Vm1wSFRWbEhNME51V1dWRmJteHJhVkIzVXpsUlRUZFZRMEYzUlVGQllVNDVDazFJYzNkRVoxbEVWbEl3VUVGUlNDOUNRVkZFUVdkRlIwMUJPRWRCTVZWa1JYZEZRaTkzVVVaTlFVMUNRV1k0ZDBoUldVUldVakJQUWtKWlJVWlFlWFVLUTBWSVNtRktVMlZqY213eFJuQjRVMkU0TlU1dGNIaHlUVUk0UjBFeFZXUkpkMUZaVFVKaFFVWkJUakFyWlZSdksxUk9OMFoxUkcxa2FrMUpTREp2TXdwNGVFMUtUVUpuUjBFeFZXUkZVVkZTVFVFclEwUlhkREZaYlZaNVltMVdNRnBZVFhSWk1rVjNSRkZaU2t0dldrbG9kbU5PUVZRlRFSlJRVVJuWjBWQ0NrRk1XbWxQYldSbk1UZG5iVUUzZWtWWU1sUkNURmc0TkZNeVJUQTNUQzlLUjJsNFpubEpiazFoVG5wb1NUVm9MeTlWYTBsV1VVeHdablp1T1ZRclZGVUtkMWQwWTFWRlUyTkZia0ZVYm14WFRVODVNMVZsTDNaNk5YRjRiVWMwVFhSMVNUWlpiV1ZTVDJGSWF6aGFkM016VUUxemRYVndkRVEwTWs4elFraFhTZ3BVU1ROdlMwbFpVbUpqWjNwa2NEbEVjSFJzYjBOT2J6TlVlRkJWWWs5R1V6aFNSRTh4S3pSWVNsVkxZV1J1U21wek9FbEtWMEpCTlU5b2VUTXdjWGw1Q2t4QmNXWTNURU52T1dzeWQyZHhSMUV3Uml0amFWbzRVaXRJT1VSaVJYVXlTa2xZWWxkcFlWVk5RMlZaWTBjdmJIZDBhRFo0VjJGaVIzWkVOMjlzVkRBS1dXaDFTVWt5UVZnelJHRnFkVlZUTkVSTU9WaHBiSEZSVlZWSFRVYzFiRzlPWVRCalZGWm1UemhtUzBFdmRXVnBXRlJYVEhScVQyNUdhMko1WjFsNU5RcFBWV2xDY0ZNMk9YSnNaM1Z0UTNWV1EyVlJXa1Z6WnowS0xTMHRMUzFGVGtRZ1EwVlNWRWxHU1VOQlZFVXRMUzB0TFFvdExTMHRMVUpGUjBsT0lFTkZVbFJKUmtsRFFWUkZMUzB0TFMwS1RVbEpSRkZVUTBOQmFXMW5RWGRKUWtGblNWVmpXRXc0U25aa2FXMDVkRXBKZFcxTVNscExkV0pYVG1STWFXdDNSRkZaU2t0dldrbG9kbU5PUVZGRlRBcENVVUYzUjJwRldVMUNXVWRCTVZWRlFYaE5VRmt5ZURGak0xSnNZMmt3TTA1RVZYZE1WMDVvVFVJMFdFUlVTWGxOUkUxNVRWUkplRTFVYTNkTk1XOVlDa1JVU1ROTlJFMTVUVVJKZVUxVWEzcE5NVzkzUjJwRldVMUNXVWRCTVZWRlFYaE5VRmt5ZURGak0xSnNZMmt3TTA1RVZYZE1WMDVvVFVsSlFrbHFRVTRLUW1kcmNXaHJhVWM1ZHpCQ1FWRkZSa0ZCVDBOQlVUaEJUVWxKUWtOblMwTkJVVVZCZVZkTlVURlRja1ZpVjJGS2VFUldSakpuTVV0RmFHcDFaMmh6VWdwdFJYTm5NWE5tT0VGVWQwZHZaRVpQVVRsUU0zZDVWMlkzY1c1NmVreFpUUzgyVjNOS2JHcE1jM0JLVEVkaVMyTkpPRVV2V0hab1pVNVRUbmRsZFdab0NraGlUM1ZsVERkNFRHSmxhMlZwT0hvd1pqaHdNbkZrUXpKQlowd3pTR3RaYkZkNlUyWmxaMUZMYmxsWlFVSlFNME0zV1hZMFYyVmtaRXRxYlZkWlRVWUtjVmxXU2xZd1MwOUpTbEo0S3k5emJFWlhaVE4yZFZSclEwZHlTM1ZuYzB4dlNWSjVPRnBxYWpkVVRHRm1lR1ZtVW5WbFNXTlJhWHBJVW5KVFMyeFJMd3BsVlRKeFdtUjFja2RoTjNwd2JVZHdMMEpRYzFWVFpsSnZNVE5JUm1aTGEwVllPV2h6SzNGeWRuQnhaWGRqZGtweFMwVkdRbEpaTkZoME1sZFVlbGRFQ21KaWVHTmxUVkkzUWs4MWFIWkNXWGRUU0dGVU5IaGpUM0JJVkd4V1FUZHNUMmhtUWxCbFJFcFlRMWQxYlZkWU16TTJUMmxzUkdoeVFuZEpSRUZSUVVJS2J6TTRkMlpVUVU5Q1owNVdTRkU0UWtGbU9FVkNRVTFEUVZGWmQwUjNXVVJXVWpCVVFWRklMMEpCVlhkQmQwVkNMM3BCWkVKblRsWklVVFJGUm1kUlZRcEJNMVExTlU5cU5VMHpjMWMwVDFveVRYZG5abUZxWmtoRmQydDNTSGRaUkZaU01HcENRbWQzUm05QlZVRXpWRFUxVDJvMVRUTnpWelJQV2pKTmQyZG1DbUZxWmtoRmQydDNSMmRaUkZaU01GSkNRazEzUlZsSlVGa3llREZqTTFKc1kya3dNMDVFVlhkTVYwNW9UVUV3UjBOVGNVZFRTV0l6UkZGRlFrTjNWVUVLUVRSSlFrRlJRbWM1ZVdkV1VpdEVWRlV3WTNjd1UxazNSSGt5UmpacmRXTm1VVXBuWkZaU2R6QndaREUxWVVwbGNtODBWVEZ3VVdwRFpuaE9URU5wTkFwNVdqWk9hVU53VEdGSlJXMXJZMDVhVVhKTllWbzJZMHhEU2pGTFRVZElOamwyUkdSelNsWlFTR3MzV21rMFFXMXdjazUzWnpOVWRVWnFRVWhCU1NzeENrdFlMMG8xWTNSNmVtcERXWGxzYUU0MVpubFllRmxpTlhRd1UwUlZiMHhRZGtwVGQxUmxUVUZxYVZSalJVWTJUMlV6YkZCd2NVSklaV2xrUlU5MFNTOEtRbk5DVVhGcGRWSkpWVXd2YldOelVGZzNOWEZTTTAxNVMzbG9lSHBGWjNwR05EUXZkRkZHVUN0WVJFeEZRMGh5YWxsbllTOU9VVlZYTmpKQ01ITm9id3A1VVdoTVZYRTNTaXRxTjFSRGVuZDBRV0l5VERoRVQxTXlTVk13T0dKa1RGVktkRXBJVmxnNWFUaEpiRXMxVG5KQ1RFeEhURUZrUlVkVU1WRjVjVm80Q2pka1JVRTNiMlZQV0ROb05GSjJNak13Y3l0bVdEZE1jelZ6VEc0S0xTMHRMUzFGVGtRZ1EwVlNWRWxHU1VOQlZFVXRMUzB0TFE9PQogICAgc2VydmVyOiBodHRwczovLzE4LjExNi4xNi42NTo2NDQzCiAgbmFtZTogc2RtLWNsdS1yZWctYQpjb250ZXh0czoKLSBjb250ZXh0OgogICAgY2x1c3Rlcjogc2RtLWNsdS1yZWctYQogICAgdXNlcjogY2x1c3Rlci1yZWdpc3RyeS1jb250cm9sbGVyLXJlYWRlcgogIG5hbWU6IHNkbS1jbHUtcmVnLWEKY3VycmVudC1jb250ZXh0OiBzZG0tY2x1LXJlZy1hCmtpbmQ6IENvbmZpZwpwcmVmZXJlbmNlczoge30KdXNlcnM6Ci0gbmFtZTogY2x1c3Rlci1yZWdpc3RyeS1jb250cm9sbGVyLXJlYWRlcgogIHVzZXI6CiAgICB0b2tlbjogZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNkltOXJjR2RvU0VoWmJIVXdhMlpuTmxBemREVjVNMnM0UVVwQlNVRlliMjFMUkZOUU1WUjBiRFpzZVdjaWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUpqYkhWemRHVnlMWEpsWjJsemRISjVJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJbU5zZFhOMFpYSXRjbVZuYVhOMGNua3RZMjl1ZEhKdmJHeGxjaTF5WldGa1pYSXRkRzlyWlc0dGQyMDRabllpTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxjblpwWTJVdFlXTmpiM1Z1ZEM1dVlXMWxJam9pWTJ4MWMzUmxjaTF5WldkcGMzUnllUzFqYjI1MGNtOXNiR1Z5TFhKbFlXUmxjaUlzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVnlkbWxqWlMxaFkyTnZkVzUwTG5WcFpDSTZJalpsTURSa05qVm1MVGd3TjJJdE5Ea3hOUzFoT1RnMExUaGpNVEF6T0RRMk0yWTBNeUlzSW5OMVlpSTZJbk41YzNSbGJUcHpaWEoyYVdObFlXTmpiM1Z1ZERwamJIVnpkR1Z5TFhKbFoybHpkSEo1T21Oc2RYTjBaWEl0Y21WbmFYTjBjbmt0WTI5dWRISnZiR3hsY2kxeVpXRmtaWElpZlEub09udzBQU29RTEUxNE1SUGhwNzMxM01udGlGUEdvUVBhd0tUbThNV2hVbmh3Y0xOdkRSYVZzdlhhYXlIb1k4Q1dLY3c2QW55TXJJZmhrUVpSQ3A3eTlhcnlwT1FCTnJZbk0tS25vTVhCV21RMzEzQjRuYzFZMXlFR0Jhdko2c1d4YmdVa3ZxQ1FRNkM0VjhpaVBoVGtTUHNfMmcxSkx2UlNhZmtpRi1LZlhnZzVzMF9sOXJsQTdpaS0xMUh4MEdJOHpQNFFMaGRIeTA3UXBsbjE3enFWbXQ0Nkozejc4RGhtM2UyWWdOOHZ2OWZsTGVTN2xLV215R2Nwak9LNUhGRVNtTkVFOGs0YzlfazRoYjF4bzJXQmhoQWxyMjJIeWhMeXF0clFhamhJRnBjallCVDl3N1ZFc0hRTTNKWFBwbmpaWllzaUR0a2pHUWxuLUNhVzZLZ0lBCg==
kind: Secret
metadata:
  name: cluster2
  namespace: cluster-registry
type: k8s.cisco.com/cluster-registry-secret
EOF

        Note: the name and namespace of the secret need to match the the contents under spec.authInfo.secretRef in the peer Cluster CR created in the previous step

    2. On the second Kubernetes cluster:

      Repeat steps on the first Kubernetes cluster to copy the contents of the first Kubernetes cluster’s local Cluster CR and the associated secret.

  2. At this stage, the Cluster Registry controller on both clusters are aware of the existence of the other Kuberentes cluster and tries to perform synchronization on certain resources. The following example shows the steps to have both Kubernetes clusters to synchronize the KafkaCluster resource

    • Create necessary RBAC resources (ClusterRole, ServiceAccount and ClusterRoleBinding) so the Cluster Registry controller have the permission to perform read operations from the peer Kubernetes cluster, and write operations to the local Kubernetes cluster

      kubectl apply -f -<<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    cluster-registry.k8s.cisco.com/controller-aggregated: "true"
  name: cluster-registry-sdm
rules:
- apiGroups:
  - kafka.banzaicloud.io
  resources:
  - kafkaclusters
  - kafkaclusters/status 
  verbs:
  - '*'
EOF

      kubectl apply -f -<<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cluster-registry-sdm
  namespace: supertubes-system
EOF

      kubectl apply -f -<<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cluster-registry-sdm
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-registry-sdm
subjects:
- kind: ServiceAccount
  name: cluster-registry-sdm
  namespace: supertubes-system
EOF

      kubectl apply -f -<<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    cluster-registry.k8s.cisco.com/reader-aggregated: "true"
  name: cluster-registry-sdm-reader
rules:
- apiGroups:
  - kafka.banzaicloud.io
  resources:
  - kafkaclusters
  - kafkaclusters/status
  verbs:
  - get
  - list
  - watch
EOF

      kubectl apply -f -<<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cluster-registry-sdm-reader
  namespace: supertubes-system
EOF

      kubectl apply -f -<<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: cluster-registry-sdm-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-registry-sdm-reader
subjects:
- kind: ServiceAccount
  name: cluster-registry-sdm-reader
  namespace: supertubes-system
EOF

    • Deploy necessary Cluster Registry CRs (ClusterFeatuer and ResourceSyncRule):

      kubectl apply -f -<<EOF
apiVersion: clusterregistry.k8s.cisco.com/v1alpha1
kind: ClusterFeature
metadata:
  labels:
    cluster-registry-controller.k8s.cisco.com/core-sync-resource: "true"
  name: sdm-core-resources
spec:
  featureName: sdm-core-resources-source
EOF

      kubectl apply -f -<<EOF
apiVersion: clusterregistry.k8s.cisco.com/v1alpha1
kind: ResourceSyncRule
metadata:
  annotations:
    cluster-registry.k8s.cisco.com/resource-sync-disabled: "true"
  labels:
    cluster-registry-controller.k8s.cisco.com/core-sync-resource: "true"
  name: sdm-core-resources-kafka-clusters-sink
spec:
  clusterFeatureMatch:
  - featureName: sdm-core-resources-source
  groupVersionKind:
    group: kafka.banzaicloud.io
    kind: KafkaCluster
    version: v1beta1
  rules:
  - mutations:
      overrides:
      - path: /metadata/name
        type: replace
        value: '{{ printf "%s-%s-%s" .Object.GetName .Cluster.GetName (trunc 4 (sha256sum
          .Cluster.GetName))}}'
      syncStatus: true
EOF

    Note:

    • The above steps to create the RBAC and Cluster Registry resources need to be run in both Kubernetes clusters.
    • If you would like to synchronize other resources, update the rules section in the ClusterRole CR, and create a ResourceSyncRule CR with spec.groupVersionKind that matches the resource you’d like to synchronize.
    • The ClusterFeature CR can be shared between multiple ResourceSyncRule resources.

Detach Kubernetes clusters

To detach a specific Kubernetes cluster from the cluster group, delete the RBAC resouces (ClusterRole, ServiceAccount and ClusterRoleBinding), the Cluster Registry CRs (ClusterFeatuer and ResourceSyncRule) that were created while attaching, and the synchronized resources (in this example, KafkaCluster) from both Kuberentes clusters.

CAUTION:

The Cluster Registry CRs (ClusterFeatuer and ResourceSyncRule) must be deleted before you can delete the synchronized resources, otherwise the Cluster Registry controller will keep synchonizing the resources you specified in the ResourceSyncRule

If there is no other resource being synchronized between the detaching cluster (let’s say cluster2) and other clusters in the cluster group (let’s say only cluster1 is remained in the cluster group), you may also delete the peer Cluster CR that represents cluster2 and the associated secret from clsuter1, and delete the peer Cluster CR that represents cluster1 and the associated secrets from cluster2

Uninstall Cluster Registry

To uninstall Cluster Registry disable the corresponding component in ApplicationManifest and the operator will uninstall the Cluster Registry controller in the next reconcilation:

apiVersion: supertubes.banzaicloud.io/v1beta1
kind: ApplicationManifest
metadata:
  name: applicationmanifest
spec:
...
  clusterregistry:
    enabled: false
    namespace: cluster-registry
...

Note: If there is no other component using Cluster Registry, you may want to clean up the previously created local Cluster CR.