Managing custom Kafka distribution with Streaming Data Manager
Streaming Data Manager can deploy and manage Apache Kafka clusters that use custom Kafka distributions. The distribution must meet the following requirements:
- The operator supports the following Kafka versions: 2.5.0,2.5.1,2.6.0,2.6.2,2.7.0,2.7.1,2.8.0,2.8.1, and 3.1.0.
- The Kafka docker image is built on Java 11.
The following meta information about the Kafka distribution must be provided through broker annotations in the KafkaCluster custom resource:
annotation | Description | |
---|---|---|
jvm.kafka.banzaicloud.io/version | Java version in major.minor.path format | required |
kafka.banzaicloud.io/version | Kafka version | required |
kafka.banzaicloud.io/build-info | Build info | optional |
kafka.banzaicloud.io/image-provider | Name of the docker image provider | optional |
You can either specify these annotations for the entire broker configuration group, or also separately for each broker. If you specify an annotation both on the broker configuration group and the broker level, the broker-level annotation is used.
Provide Kafka meta information through broker config group
apiVersion: kafka.banzaicloud.io/v1beta1
kind: KafkaCluster
metadata:
name: kafka
spec:
...
brokerConfigGroups:
default:
...
brokerAnnotations:
jvm.kafka.banzaicloud.io/version: "11.0.10"
kafka.banzaicloud.io/version: "2.6.1"
Provide Kafka meta information through broker level config
apiVersion: kafka.banzaicloud.io/v1beta1
kind: KafkaCluster
metadata:
name: kafka
spec:
...
brokerConfigGroups:
default:
...
brokers:
- id: 0
brokerConfigGroup: "default"
brokerAnnotations:
jvm.kafka.banzaicloud.io/version: "11.0.10"
kafka.banzaicloud.io/version: "2.6.1"