Operator Mode

The following example demonstrates how to enable SDM resource synchronization between two Kubernetes clusters in Operator mode.

Install Cluster Registry

  1. Installing Cluster Registry in operator mode follows the same principle as installing other components: via modifying and reconciling the Streaming Data Manager ApplicationManifest custom resource. For example, the following configuration in the ApplicationManifest CR would install Cluster Registry controller under the “cluster-registry” namespace

    apiVersion: supertubes.banzaicloud.io/v1beta1
    kind: ApplicationManifest
    metadata:
      name: applicationmanifest
    spec:
    ...
      clusterregistry:
        enabled: true
        namespace: cluster-registry
    ...
    
  2. After the Cluster Registry controller is up and running, deploy a local Cluster CR to represent your Kubernetes cluster. Note the following points:

    • The spec.clusterID field needs to be the UID of namespace kube-system to ensure the uniqueness of the Cluster CR.
    • The spec.authInfo.secretRef holds information that describes how the peer clusters can get credentials to access the cluster. Therefore, the name of the spec.authInfo.secretRef is recommended to be the name of the Cluster CR to avoid confusion.
    • The server address under spec.kubernetesApiEndpoints must be accessible from the peer clusters.
    kubectl apply -f -<<EOF
    apiVersion: clusterregistry.k8s.cisco.com/v1alpha1
    kind: Cluster
    metadata:
      name: cluster1
    spec:
      clusterID: d51418e7-547e-468b-adbc-9c2968389184
      authInfo:
        secretRef:
          name: cluster1
          namespace: cluster-registry
      kubernetesApiEndpoints:
      - serverAddress: https://3.13.240.171:6443
    EOF
    
  3. After the installation is finished, you can attach Kubernetes clusters and detach Kubernetes clusters.

Attach Kubernetes clusters

There are several steps for attaching one Kubernetes cluster to another and have them fully synchronized.

  1. First of all, we need to deploy a peer Cluster CR in each cluster and the associated secret to each Kubernetes cluster to make the Cluster Registry controller aware the existence of the other cluster:

    1. On the first Kubernetes cluster

      • Copy the local Cluster CR contents from the second Kubernetes cluster and deploy it to the first Kubernetes cluster:

        kubectl apply -f -<<EOF
        apiVersion: clusterregistry.k8s.cisco.com/v1alpha1
        kind: Cluster
        metadata:
          name: cluster2
        spec:
          # clusterID is the UID of kube-system namespace in the second Kubernetes cluster
          clusterID: ac21678f-9ebf-4ee1-83d4-3cca0931af19
          authInfo:
            secretRef:
              name: cluster2
              namespace: cluster-registry
          kubernetesApiEndpoints:
          - serverAddress: https://18.190.53.82:6443
        EOF
        
      • Copy the secret that is specified in the peer Cluster from the second Kuberentes cluster and deploy it to the first Kubernetes cluster:

        kubectl apply -f -<<EOF
        apiVersion: v1
        data:
          kubeconfig: YXBpVmVyc2lvbjogdjEKY2x1c3RlcnM6Ci0gY2x1c3RlcjoKICAgIGNlcnRpZmljYXRlLWF1dGhvcml0eS1kYXRhOiBMUzB0TFMxQ1JVZEpUaUJEUlZKVVNVWkpRMEZVUlMwdExTMHRDazFKU1VSUVZFTkRRV2xYWjBGM1NVSkJaMGxWVDNWaFZtTTRNVFU1YlZKMlJrRkRRbHB4TW5sd1JWTktNV2h6ZDBSUldVcExiMXBKYUhaalRrRlJSVXdLUWxGQmQwZHFSVmxOUWxsSFFURlZSVUY0VFZCWk1uZ3hZek5TYkdOcE1ETk9SRlYzVEZkT2FFMUNORmhFVkVsNVRVUk5lVTFVU1hoTlZHdDNUVEZ2V0FwRVZFa3pUVVJOZVUxRVNYbE5WR3Q2VFRGdmQwZEVSVmROUWxGSFFURlZSVUY0VFU1aE0xWnBXbGhLZFZwWVVteGplVEZxV1ZSRFEwRlRTWGRFVVZsS0NrdHZXa2xvZG1OT1FWRkZRa0pSUVVSblowVlFRVVJEUTBGUmIwTm5aMFZDUVV0UlNuVjJPR3BMWnpoQk5XdDZlVWR0TVZOSmNDdDNlRVJGUzFWc09FOEtTVXBSSzNCdEwwOHZZMDF6UWpSRU1DOW5WVEZLVGxoR01tUk1MMU5VTm1wbFVFUnJXakZsTVZSUVZHMDNhVkppZFhacGRWQk9TV1J6U2tZMVpHRk9ZUXBOY0V4TVkydEpUR1ZCVkhweFNsVkZabXR0ZFVVdmFsaFdXa2hEYW1KMmRVdE1TbFZxZVRSR1JVUlhNalZsWlVReVJsUnRXVlV3TldkbVp6bFpXamRPQ2tZeGVFUjBlalJ5TDJWbWN6bGpOM1J0UzJsUmF6WlVlbnBtUWtOSWRUQkRjR2xXWmxSMlpFdzRURFZYWTFsdFkxaGlVbEZhWm1SQ1RsVjRjblZSTHpRS1R6ZHNhVmt2ZEZsRVRDODRaekJ1YlRGblNUWlhRa0ZSWWxWbU1sRTRkMEkxY3psNmVtRkVNWEpFYXk5b2ExRlRiR2d3TmxSSU5uRkNka1JuWmpOWU5ncGhaV1kzUTFKNk9XdEthRlpUU1c5aFEzTmtiMHRxWms4M2JIZFlSRm95Vm1wSFRWbEhNME51V1dWRmJteHJhVkIzVXpsUlRUZFZRMEYzUlVGQllVNDVDazFJYzNkRVoxbEVWbEl3VUVGUlNDOUNRVkZFUVdkRlIwMUJPRWRCTVZWa1JYZEZRaTkzVVVaTlFVMUNRV1k0ZDBoUldVUldVakJQUWtKWlJVWlFlWFVLUTBWSVNtRktVMlZqY213eFJuQjRVMkU0TlU1dGNIaHlUVUk0UjBFeFZXUkpkMUZaVFVKaFFVWkJUakFyWlZSdksxUk9OMFoxUkcxa2FrMUpTREp2TXdwNGVFMUtUVUpuUjBFeFZXUkZVVkZTVFVFclEwUlhkREZaYlZaNVltMVdNRnBZVFhSWk1rVjNSRkZaU2t0dldrbG9kbU5PUVZRlRFSlJRVVJuWjBWQ0NrRk1XbWxQYldSbk1UZG5iVUUzZWtWWU1sUkNURmc0TkZNeVJUQTNUQzlLUjJsNFpubEpiazFoVG5wb1NUVm9MeTlWYTBsV1VVeHdablp1T1ZRclZGVUtkMWQwWTFWRlUyTkZia0ZVYm14WFRVODVNMVZsTDNaNk5YRjRiVWMwVFhSMVNUWlpiV1ZTVDJGSWF6aGFkM016VUUxemRYVndkRVEwTWs4elFraFhTZ3BVU1ROdlMwbFpVbUpqWjNwa2NEbEVjSFJzYjBOT2J6TlVlRkJWWWs5R1V6aFNSRTh4S3pSWVNsVkxZV1J1U21wek9FbEtWMEpCTlU5b2VUTXdjWGw1Q2t4QmNXWTNURU52T1dzeWQyZHhSMUV3Uml0amFWbzRVaXRJT1VSaVJYVXlTa2xZWWxkcFlWVk5RMlZaWTBjdmJIZDBhRFo0VjJGaVIzWkVOMjlzVkRBS1dXaDFTVWt5UVZnelJHRnFkVlZUTkVSTU9WaHBiSEZSVlZWSFRVYzFiRzlPWVRCalZGWm1UemhtUzBFdmRXVnBXRlJYVEhScVQyNUdhMko1WjFsNU5RcFBWV2xDY0ZNMk9YSnNaM1Z0UTNWV1EyVlJXa1Z6WnowS0xTMHRMUzFGVGtRZ1EwVlNWRWxHU1VOQlZFVXRMUzB0TFFvdExTMHRMVUpGUjBsT0lFTkZVbFJKUmtsRFFWUkZMUzB0TFMwS1RVbEpSRkZVUTBOQmFXMW5RWGRKUWtGblNWVmpXRXc0U25aa2FXMDVkRXBKZFcxTVNscExkV0pYVG1STWFXdDNSRkZaU2t0dldrbG9kbU5PUVZGRlRBcENVVUYzUjJwRldVMUNXVWRCTVZWRlFYaE5VRmt5ZURGak0xSnNZMmt3TTA1RVZYZE1WMDVvVFVJMFdFUlVTWGxOUkUxNVRWUkplRTFVYTNkTk1XOVlDa1JVU1ROTlJFMTVUVVJKZVUxVWEzcE5NVzkzUjJwRldVMUNXVWRCTVZWRlFYaE5VRmt5ZURGak0xSnNZMmt3TTA1RVZYZE1WMDVvVFVsSlFrbHFRVTRLUW1kcmNXaHJhVWM1ZHpCQ1FWRkZSa0ZCVDBOQlVUaEJUVWxKUWtOblMwTkJVVVZCZVZkTlVURlRja1ZpVjJGS2VFUldSakpuTVV0RmFHcDFaMmh6VWdwdFJYTm5NWE5tT0VGVWQwZHZaRVpQVVRsUU0zZDVWMlkzY1c1NmVreFpUUzgyVjNOS2JHcE1jM0JLVEVkaVMyTkpPRVV2V0hab1pVNVRUbmRsZFdab0NraGlUM1ZsVERkNFRHSmxhMlZwT0hvd1pqaHdNbkZrUXpKQlowd3pTR3RaYkZkNlUyWmxaMUZMYmxsWlFVSlFNME0zV1hZMFYyVmtaRXRxYlZkWlRVWUtjVmxXU2xZd1MwOUpTbEo0S3k5emJFWlhaVE4yZFZSclEwZHlTM1ZuYzB4dlNWSjVPRnBxYWpkVVRHRm1lR1ZtVW5WbFNXTlJhWHBJVW5KVFMyeFJMd3BsVlRKeFdtUjFja2RoTjNwd2JVZHdMMEpRYzFWVFpsSnZNVE5JUm1aTGEwVllPV2h6SzNGeWRuQnhaWGRqZGtweFMwVkdRbEpaTkZoME1sZFVlbGRFQ21KaWVHTmxUVkkzUWs4MWFIWkNXWGRUU0dGVU5IaGpUM0JJVkd4V1FUZHNUMmhtUWxCbFJFcFlRMWQxYlZkWU16TTJUMmxzUkdoeVFuZEpSRUZSUVVJS2J6TTRkMlpVUVU5Q1owNVdTRkU0UWtGbU9FVkNRVTFEUVZGWmQwUjNXVVJXVWpCVVFWRklMMEpCVlhkQmQwVkNMM3BCWkVKblRsWklVVFJGUm1kUlZRcEJNMVExTlU5cU5VMHpjMWMwVDFveVRYZG5abUZxWmtoRmQydDNTSGRaUkZaU01HcENRbWQzUm05QlZVRXpWRFUxVDJvMVRUTnpWelJQV2pKTmQyZG1DbUZxWmtoRmQydDNSMmRaUkZaU01GSkNRazEzUlZsSlVGa3llREZqTTFKc1kya3dNMDVFVlhkTVYwNW9UVUV3UjBOVGNVZFRTV0l6UkZGRlFrTjNWVUVLUVRSSlFrRlJRbWM1ZVdkV1VpdEVWRlV3WTNjd1UxazNSSGt5UmpacmRXTm1VVXBuWkZaU2R6QndaREUxWVVwbGNtODBWVEZ3VVdwRFpuaE9URU5wTkFwNVdqWk9hVU53VEdGSlJXMXJZMDVhVVhKTllWbzJZMHhEU2pGTFRVZElOamwyUkdSelNsWlFTR3MzV21rMFFXMXdjazUzWnpOVWRVWnFRVWhCU1NzeENrdFlMMG8xWTNSNmVtcERXWGxzYUU0MVpubFllRmxpTlhRd1UwUlZiMHhRZGtwVGQxUmxUVUZxYVZSalJVWTJUMlV6YkZCd2NVSklaV2xrUlU5MFNTOEtRbk5DVVhGcGRWSkpWVXd2YldOelVGZzNOWEZTTTAxNVMzbG9lSHBGWjNwR05EUXZkRkZHVUN0WVJFeEZRMGh5YWxsbllTOU9VVlZYTmpKQ01ITm9id3A1VVdoTVZYRTNTaXRxTjFSRGVuZDBRV0l5VERoRVQxTXlTVk13T0dKa1RGVktkRXBJVmxnNWFUaEpiRXMxVG5KQ1RFeEhURUZrUlVkVU1WRjVjVm80Q2pka1JVRTNiMlZQV0ROb05GSjJNak13Y3l0bVdEZE1jelZ6VEc0S0xTMHRMUzFGVGtRZ1EwVlNWRWxHU1VOQlZFVXRMUzB0TFE9PQogICAgc2VydmVyOiBodHRwczovLzE4LjExNi4xNi42NTo2NDQzCiAgbmFtZTogc2RtLWNsdS1yZWctYQpjb250ZXh0czoKLSBjb250ZXh0OgogICAgY2x1c3Rlcjogc2RtLWNsdS1yZWctYQogICAgdXNlcjogY2x1c3Rlci1yZWdpc3RyeS1jb250cm9sbGVyLXJlYWRlcgogIG5hbWU6IHNkbS1jbHUtcmVnLWEKY3VycmVudC1jb250ZXh0OiBzZG0tY2x1LXJlZy1hCmtpbmQ6IENvbmZpZwpwcmVmZXJlbmNlczoge30KdXNlcnM6Ci0gbmFtZTogY2x1c3Rlci1yZWdpc3RyeS1jb250cm9sbGVyLXJlYWRlcgogIHVzZXI6CiAgICB0b2tlbjogZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNkltOXJjR2RvU0VoWmJIVXdhMlpuTmxBemREVjVNMnM0UVVwQlNVRlliMjFMUkZOUU1WUjBiRFpzZVdjaWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUpqYkhWemRHVnlMWEpsWjJsemRISjVJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJbU5zZFhOMFpYSXRjbVZuYVhOMGNua3RZMjl1ZEhKdmJHeGxjaTF5WldGa1pYSXRkRzlyWlc0dGQyMDRabllpTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxjblpwWTJVdFlXTmpiM1Z1ZEM1dVlXMWxJam9pWTJ4MWMzUmxjaTF5WldkcGMzUnllUzFqYjI1MGNtOXNiR1Z5TFhKbFlXUmxjaUlzSW10MVltVnlibVYwWlhNdWFXOHZjMlZ5ZG1salpXRmpZMjkxYm5RdmMyVnlkbWxqWlMxaFkyTnZkVzUwTG5WcFpDSTZJalpsTURSa05qVm1MVGd3TjJJdE5Ea3hOUzFoT1RnMExUaGpNVEF6T0RRMk0yWTBNeUlzSW5OMVlpSTZJbk41YzNSbGJUcHpaWEoyYVdObFlXTmpiM1Z1ZERwamJIVnpkR1Z5TFhKbFoybHpkSEo1T21Oc2RYTjBaWEl0Y21WbmFYTjBjbmt0WTI5dWRISnZiR3hsY2kxeVpXRmtaWElpZlEub09udzBQU29RTEUxNE1SUGhwNzMxM01udGlGUEdvUVBhd0tUbThNV2hVbmh3Y0xOdkRSYVZzdlhhYXlIb1k4Q1dLY3c2QW55TXJJZmhrUVpSQ3A3eTlhcnlwT1FCTnJZbk0tS25vTVhCV21RMzEzQjRuYzFZMXlFR0Jhdko2c1d4YmdVa3ZxQ1FRNkM0VjhpaVBoVGtTUHNfMmcxSkx2UlNhZmtpRi1LZlhnZzVzMF9sOXJsQTdpaS0xMUh4MEdJOHpQNFFMaGRIeTA3UXBsbjE3enFWbXQ0Nkozejc4RGhtM2UyWWdOOHZ2OWZsTGVTN2xLV215R2Nwak9LNUhGRVNtTkVFOGs0YzlfazRoYjF4bzJXQmhoQWxyMjJIeWhMeXF0clFhamhJRnBjallCVDl3N1ZFc0hRTTNKWFBwbmpaWllzaUR0a2pHUWxuLUNhVzZLZ0lBCg==
        kind: Secret
        metadata:
          name: cluster2
          namespace: cluster-registry
        type: k8s.cisco.com/cluster-registry-secret
        EOF
        

        Note: the name and namespace of the secret need to match the the contents under spec.authInfo.secretRef in the peer Cluster CR created in the previous step

    2. On the second Kubernetes cluster:

      Repeat steps on the first Kubernetes cluster to copy the contents of the first Kubernetes cluster’s local Cluster CR and the associated secret.

  2. At this stage, the Cluster Registry controller on both clusters are aware of the existence of the other Kuberentes cluster and tries to perform synchronization on certain resources. The following example shows the steps to have both Kubernetes clusters to synchronize the KafkaCluster resource

    • Create necessary RBAC resources (ClusterRole, ServiceAccount and ClusterRoleBinding) so the Cluster Registry controller have the permission to perform read operations from the peer Kubernetes cluster, and write operations to the local Kubernetes cluster

      kubectl apply -f -<<EOF
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRole
      metadata:
        labels:
          cluster-registry.k8s.cisco.com/controller-aggregated: "true"
        name: cluster-registry-sdm
      rules:
      - apiGroups:
        - kafka.banzaicloud.io
        resources:
        - kafkaclusters
        - kafkaclusters/status 
        verbs:
        - '*'
      EOF
      
      kubectl apply -f -<<EOF
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: cluster-registry-sdm
        namespace: supertubes-system
      EOF
      
      kubectl apply -f -<<EOF
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRoleBinding
      metadata:
        name: cluster-registry-sdm
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: cluster-registry-sdm
      subjects:
      - kind: ServiceAccount
        name: cluster-registry-sdm
        namespace: supertubes-system
      EOF
      
      kubectl apply -f -<<EOF
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRole
      metadata:
        labels:
          cluster-registry.k8s.cisco.com/reader-aggregated: "true"
        name: cluster-registry-sdm-reader
      rules:
      - apiGroups:
        - kafka.banzaicloud.io
        resources:
        - kafkaclusters
        - kafkaclusters/status
        verbs:
        - get
        - list
        - watch
      EOF
      
      kubectl apply -f -<<EOF
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: cluster-registry-sdm-reader
        namespace: supertubes-system
      EOF
      
      kubectl apply -f -<<EOF
      apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRoleBinding
      metadata:
        name: cluster-registry-sdm-reader
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: ClusterRole
        name: cluster-registry-sdm-reader
      subjects:
      - kind: ServiceAccount
        name: cluster-registry-sdm-reader
        namespace: supertubes-system
      EOF
      
    • Deploy necessary Cluster Registry CRs (ClusterFeatuer and ResourceSyncRule):

      kubectl apply -f -<<EOF
      apiVersion: clusterregistry.k8s.cisco.com/v1alpha1
      kind: ClusterFeature
      metadata:
        labels:
          cluster-registry-controller.k8s.cisco.com/core-sync-resource: "true"
        name: sdm-core-resources
      spec:
        featureName: sdm-core-resources-source
      EOF
      
      kubectl apply -f -<<EOF
      apiVersion: clusterregistry.k8s.cisco.com/v1alpha1
      kind: ResourceSyncRule
      metadata:
        annotations:
          cluster-registry.k8s.cisco.com/resource-sync-disabled: "true"
        labels:
          cluster-registry-controller.k8s.cisco.com/core-sync-resource: "true"
        name: sdm-core-resources-kafka-clusters-sink
      spec:
        clusterFeatureMatch:
        - featureName: sdm-core-resources-source
        groupVersionKind:
          group: kafka.banzaicloud.io
          kind: KafkaCluster
          version: v1beta1
        rules:
        - mutations:
            overrides:
            - path: /metadata/name
              type: replace
              value: '{{ printf "%s-%s-%s" .Object.GetName .Cluster.GetName (trunc 4 (sha256sum
                .Cluster.GetName))}}'
            syncStatus: true
      EOF
      

    Note:

    • The above steps to create the RBAC and Cluster Registry resources need to be run in both Kubernetes clusters.
    • If you would like to synchronize other resources, update the rules section in the ClusterRole CR, and create a ResourceSyncRule CR with spec.groupVersionKind that matches the resource you’d like to synchronize.
    • The ClusterFeature CR can be shared between multiple ResourceSyncRule resources.

Detach Kubernetes clusters

To detach a specific Kubernetes cluster from the cluster group, delete the RBAC resouces (ClusterRole, ServiceAccount and ClusterRoleBinding), the Cluster Registry CRs (ClusterFeatuer and ResourceSyncRule) that were created while attaching, and the synchronized resources (in this example, KafkaCluster) from both Kuberentes clusters.

CAUTION:

The Cluster Registry CRs (ClusterFeatuer and ResourceSyncRule) must be deleted before you can delete the synchronized resources, otherwise the Cluster Registry controller will keep synchonizing the resources you specified in the ResourceSyncRule

If there is no other resource being synchronized between the detaching cluster (let’s say cluster2) and other clusters in the cluster group (let’s say only cluster1 is remained in the cluster group), you may also delete the peer Cluster CR that represents cluster2 and the associated secret from clsuter1, and delete the peer Cluster CR that represents cluster1 and the associated secrets from cluster2

Uninstall Cluster Registry

To uninstall Cluster Registry disable the corresponding component in ApplicationManifest and the operator will uninstall the Cluster Registry controller in the next reconcilation:

apiVersion: supertubes.banzaicloud.io/v1beta1
kind: ApplicationManifest
metadata:
  name: applicationmanifest
spec:
...
  clusterregistry:
    enabled: false
    namespace: cluster-registry
...

Note: If there is no other component using Cluster Registry, you may want to clean up the previously created local Cluster CR.