Manage Kafka Connect clusters
Streaming Data Manager automates the deployment of Kafka Connect clusters and the creation of connectors through KafkaConnect custom resource instances and KafkaConnector custom resource instances, respectively.
You can manage these resources both imperatively and declaratively.
Imperative Kafka Connect cluster management
The Streaming Data Manager CLI provides commands that deploy Kafka Connect clusters, and connectors to these clusters . The easiest way to quickly set up a Kafka Connect cluster is to run the:
smm sdm cluster kafka-connect create
command which will set up a Kafka Connect cluster with default settings, linked to the provisioned demo Apache Kafka cluster.
To create an additional Kafka Connect cluster, or update or delete an existing one, use the following command:
smm sdm cluster kafka-connect [command]
These create and update sub-commands require KafkaConnect custom resource specifications as descriptors. For details on these commands, see smm sdm cluster kafka-connect.
Declarative Kafka Connect cluster management
As we’ve already touched on briefly, a Kafka Connect cluster is represented by a KafkaConnect custom resource. Streaming Data Manager deploys the Kubernetes resources necessary to set up Kafka Connect clusters in accordance with the specifications of these custom resources. As a result, any Kafka Connect configuration changes made in a custom resource are automatically propagated to the corresponding Kafka Connect cluster.
Streaming Data Manager computes and maintains the following KafkaConnect configurations automatically. You cannot override these settings:
- bootstrap.servers
- group.id
- config.storage.topic
- offset.storage.topic
- status.storage.topic
- key.converter.schema.registry.url
- value.converter.schema.registry.url