Mandatory KafkaCluster CR elements in Streaming Data Manager

When you create your own KafkaCluster custom resource, the following sources can help you:

Mandatory KafkaCluster CR elements in Streaming Data Manager

When creating your own Kafka cluster using Streaming Data Manager, some elements of the KafkaCluster CR configurations are mandatory. You need to set them properly to create a healthy Kafka cluster that works well in a Calisti environment.

  1. Istio ingress configurations
  2. Monitoring configuration
  3. Disable headless service
  4. Istio broker pod annotation
  5. Cruise Control configurations
  6. Configurations that are set by Streaming Data Manager automatically

Istio ingress configurations

Streaming Data Manager relies on Istio, so the KafkaCluster CR must include some Istio specific settings. For details, see Kafka ACLs over Istio overview.

spec: 
  ...
  ingressController: "istioingress"
  istioIngressConfig:
    gatewayConfig:
      # PASSTHROUGH or ISTIO_MUTUAL
      mode: PASSTHROUGH
  istioControlPlane:
    # name of the IstioControlPlane custom resource
    name: sdm-icp-v115x
    # namespace of the IstioControlPlane custom resource
    namespace: istio-system
  ...

istioControlPlane is a reference to the IstioControlPlane resource for Istio proxy configuration. It must be specified if Istio ingress is used.

The Istio control plane is an important part of the Istio system: it configures the Istio proxy. Streaming Data Manager creates Istio resources for the Kafka cluster. To bind these resources to the Istio control plane, you must specify the name and namespace of the istioControlPlane in the KafkaCluster CR.

Monitoring configuration

Streaming Data Manager uses Prometheus to collect metrics for dashboards and alert-based scaling. The metrics (generated by JMX exporter), need to be in proper format for Streaming Data Manager. The proper format and the necessary metrics are specified in the kafkaJMXExporterConfig configuration (see the following sample).

JMX exporter connects to Java’s native metric collection system, Java Management Extensions (JMX), and converts the metrics into a format that Prometheus can understand.

Include the following configuration in every KafkaCluster CR that you use in a Streaming Data Manager environment.



Disable headless service


The Istio ecosystem does not support headless services in some use cases that are needed for Streaming Data Manager. Therefore, the headless service must be disabled in the KafkaCluster CR.


spec:
  ...
  headlessServiceEnabled: false
  ...



Istio Prometheus broker pod annotation


You need to set a specific annotation for every broker pod to generate Istio proxy metrics separately from other metrics for Prometheus. Unexpected errors can occur if this annotation is missing.


You can either specify the annotation for every broker independently, or use brokerConfigGroups.


spec:
  ...
  brokerConfigGroups:
    default:
      brokerAnnotations:
        prometheus.istio.io/merge-metrics: "false"
  ...



Cruise control configurations


You have to specify the Cruise Control configuration. In the config section you can change the property values to your preferences.




Configurations set automatically by Streaming Data Manager


Streaming Data Manager automatically sets some mandatory configurations using a webhook. These configurations depend on the deployed KafkaCluster CR settings. When deploying a KafkaCluster CR, Streaming Data Manager automatically configures these settings into your KafkaCluster CR.


    

  • 

    Calisti image pull secrets for broker pods:
    

          imagePullSecrets:
      - name: registry-creds
      - name: smm-pull-secret

    

    • 

  • 

    Init containers for broker pods:
    

          initContainers:
      - args:
        - -c
        - cp -r ${JAR_PATH}/* /opt/cisco/kafka/lib/authn-javaagent-libs
        command:
        - /bin/sh
        image: 033498657557.dkr.ecr.us-east-2.amazonaws.com/banzaicloud/kafka-authn-agent:3.1.0-1.0
        name: authn-javaagent-libs-loader
        resources:
          limits:
            cpu: 100m
            memory: 128Mi
        volumeMounts:
        - mountPath: /opt/cisco/kafka/lib/authn-javaagent-libs
          name: authn-javaagent-libs
      serviceAccountName: kafka-cluster
      storageConfigs:
      - mountPath: /kafka-logs
        pvcSpec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi
      terminationGracePeriodSeconds: 120
      volumeMounts:
      - mountPath: /opt/cisco/kafka/lib/authn-javaagent-libs
        name: authn-javaagent-libs
      volumes:
      - name: authn-javaagent-libs

    

    • 

  • 

    Broker pod annotations:
    

          brokerAnnotations:
        sidecar.istio.io/userVolumeMount: '[{"name":"exitfile","readOnly":true,"mountPath":"/var/run/wait"}]'

    

    • 

  • 

    Broker pod environment variables:
    

          envs:
      - name: CLASSPATH+
        value: :/opt/cisco/kafka/lib/authn-javaagent-libs/*
      - name: KAFKA_OPTS+
        value: ' -javaagent:/opt/cisco/kafka/lib/authn-javaagent-libs/agent.jar'
      - name: KAFKA_OPTS+
        value: ' --add-exports java.base/jdk.internal.org.objectweb.asm=ALL-UNNAMED'

    

    • 

  • 

    Cruise Control modules init container:
    

        initContainers:
    - args:
      - -c
      - cp -r ${CRUISE_CONTROL_LIBS}/* /var/lib/cruise-control-ext-libs
      command:
      - /bin/sh
      image: 033498657557.dkr.ecr.us-east-2.amazonaws.com/banzaicloud/cruisecontrol-modules:2.5.101-1.0
      imagePullPolicy: IfNotPresent
      name: external-libs-loader
      resources:
        limits:
          cpu: 100m
          memory: 128Mi
      volumeMounts:
      - mountPath: /var/lib/cruise-control-ext-libs
        name: external-libs
    serviceAccountName: kafka-cluster
    volumeMounts:
    - mountPath: /var/lib/cruise-control-ext-libs
      name: external-libs
    volumes:
    - name: external-libs

    

    • 

  • 

    Super users for the Kafka cluster. When ACL is enabled, additional super users are needed for the Kafka cluster to provide full access for the necessary Streaming Data Manager components.
    

    readOnlyConfig: |-
    ...
    authorizer.class.name=kafka.security.authorizer.AclAuthorizer
    ...
    super.users=User:CN=kafka-default;User:CN=kafka-kafka-operator;User:CN=supertubes-system-supertubes;User:CN=supertubes-system-supertubes-ui


    

    • 

  • 

    Virtual service annotation:
    

    istioIngressConfig:
  ...
    virtualServiceAnnotations:
      virtualservice.istio.banzaicloud.io/direct-connect-destinations: '{"kafka-0.kafka.svc.cluster.local":[9094],"kafka-1.kafka.svc.cluster.local":[9094],"kafka-all-broker.kafka.svc.cluster.local":[9094]}'

    

    •