Managing custom Kafka distribution with Streaming Data Manager

Streaming Data Manager can deploy and manage Apache Kafka clusters that use custom Kafka distributions. The distribution must meet the following requirements:

• The operator supports the following Kafka versions: 2.5.0,2.5.1,2.6.0,2.6.2,2.7.0,2.7.1,2.8.0,2.8.1, and 3.1.0.
• The Kafka docker image is built on Java 11.

The following meta information about the Kafka distribution must be provided through broker annotations in the KafkaCluster custom resource:

You can either specify these annotations for the entire broker configuration group, or also separately for each broker. If you specify an annotation both on the broker configuration group and the broker level, the broker-level annotation is used.

Provide Kafka meta information through broker config group

apiVersion: kafka.banzaicloud.io/v1beta1
kind: KafkaCluster
metadata:
  name: kafka
spec:
...
  brokerConfigGroups:
    default:
    ...
      brokerAnnotations:
        jvm.kafka.banzaicloud.io/version: "11.0.10"
        kafka.banzaicloud.io/version: "2.6.1"

Provide Kafka meta information through broker level config

apiVersion: kafka.banzaicloud.io/v1beta1
kind: KafkaCluster
metadata:
  name: kafka
spec:
...
  brokerConfigGroups:
    default:
    ...
  brokers:
    - id: 0
      brokerConfigGroup: "default"
      brokerAnnotations:
       jvm.kafka.banzaicloud.io/version: "11.0.10"
       kafka.banzaicloud.io/version: "2.6.1"