KafkaCluster CR examples

The following KafkaCluster custom resource (CR) examples show you some basic use cases. You can use these examples as a base to create your own Kafka cluster when using Streaming Data Manager.

Kafka cluster for Kafka clients in the same Istio mesh and Kubernetes cluster

In this scenario, brokers accept connections on listeners that are used for internal communication only. Only Kafka clients that are in the same Kubernetes cluster and Istio mesh as the Kafka cluster can connect to it.
This is the same KafkaCluster configuration that is generated by the CLI.

• Kafka cluster with internal listeners

The two listeners can be accessed on the 29092 and 29093 ports.

listenersConfig:
    internalListeners:
      - type: "plaintext"
        name: "internal"
        containerPort: 29092
        # this listener is used for inner broker communication
        usedForInnerBrokerCommunication: true
      - type: "plaintext"
        name: "controller"
        containerPort: 29093
        usedForInnerBrokerCommunication: false
        # this listener is used for controller communication
        usedForControllerCommunication: true

The Kafka cluster has two brokers and they are using the default broker configuration group:

  brokers:
    - id: 0
      brokerConfigGroup: "default"
    - id: 1
      brokerConfigGroup: "default"

The default broker configuration group configures the storage, a service account, and the broker pod annotations:

brokerConfigGroups:
    default:
      serviceAccountName: default
      brokerAnnotations:
        prometheus.istio.io/merge-metrics: "false"
      storageConfigs:
        - mountPath: "/kafka-logs"
          pvcSpec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 10Gi

Kafka cluster with ACL Authorizer

This configuration is the same as the first example, but ACL Authorizer is enabled. This means that even though the Kafka clients are in the same Istio mesh and Kubernetes cluster, they must authenticate themselves. Authentication happens based on the Kafka client’s service account and namespace. Use the KafkaACL custom resource to set permissions for Kafka resources. For details, see Kafka ACLs over Istio overview.

• Internal listeners with ACL Authorizer

Kafka cluster with external listener and without ACL Authorizer

This KafkaCluster CR has an external listener, so Kafka clients from outside the Kubernetes cluster can reach this listener in plaintext mode. You can get the listener’s external address for the Kafka clients from the KafkaCluster CR’s status.

• Kafka cluster with external listener

Kafka cluster with external listener and with ACL Authorizer

This KafkaCluster CR has an external listener that accepts SSL connections. The connection between your external Kafka client and the Istio proxy is encrypted. The Istio proxy in the broker pod terminates the SSL, so the communication within the broker pod happens in plaintext. This is why the listener is in plaintext mode in the configuration but the real communication between the Kafka clients and the Kafka cluster is encrypted. Kafka clients that are outside the Kubernetes cluster can reach the Kafka cluster on this listener only with proper SSL certificates.

ACL Authorizer is enabled. For details, see Kafka ACLs over Istio overview.

• Kafka cluster with external listener and ACL Authorizer

Kafka cluster for performance test

This example can be used for performance tests. It contains Kafka configuration tweaks and JAVA VM tweaks that can improve the performance.

• Kafka cluster for performance tests

KafkaCluster CR

For more details, see this sample KafkaCluster CR with comments on what each field signifies.