Manage Kafka Connectors
Streaming Data Manager automates the deployment of Kafka Connect clusters and the creation of connectors through KafkaConnect custom resource instances and KafkaConnector custom resource instances, respectively.
You can manage these resources both imperatively and declaratively.
Imperative Kafka Connect connector management
The Streaming Data Manager CLI provides commands that deploy Kafka Connect clusters, and connectors to these clusters. The easiest way to quickly set up a Kafka Connect cluster is to run the:
smm install -a --install-sdm
command which will set up a Kafka Connect cluster with default settings, linked to the provisioned demo Apache Kafka cluster and the deployed Schema Registry instance.
To create, update, or delete Kafka connectors, use the following command:
smm sdm cluster kafka-connector [command]
update sub-commands require KafkaConnector custom resource specifications as descriptors. For details on these commands, see smm sdm cluster kafka-connector.
Declarative Kafka Connect connector management
The pattern for managing connectors is similar to that of Kafka Connect. There is a KafkaConnector custom resource that represents a connector type which is used by Streaming Data Manager to instantiate connectors on Kafka Connect clusters.
apiVersion: kafka.banzaicloud.io/v1beta1 kind: KafkaConnector metadata: name: my-connector spec: # The connector class name (e.g. io.confluent.connect.s3.S3SinkConnector) class: # The maximum number of tasks the Kafka Connector can create tasksMax: # References the Kafka Connect cluster where this connector is to run clusterRef: name: my-kafka-connect # Defines the config values for Kafka Connector in key - value pairs format config: # Whether to pause or resume the connector execution pause:
- [Kafka Connect and Connector examples](/sdm/kafka-connect/examples/