Manage Kafka Connectors

Streaming Data Manager automates the deployment of Kafka Connect clusters and the creation of connectors through KafkaConnect custom resource instances and KafkaConnector custom resource instances, respectively.

You can manage these resources both imperatively and declaratively.

Imperative Kafka Connect connector management

​The Streaming Data Manager CLI provides commands that deploy Kafka Connect clusters, and connectors to these clusters. ​The easiest way to quickly set up a Kafka Connect cluster is to run the:

smm install -a --install-sdm

command which will set up a Kafka Connect cluster with default settings, linked to the provisioned demo Apache Kafka cluster and the deployed Schema Registry instance.

To create, update, or delete Kafka connectors, use the following command:

smm sdm cluster kafka-connector [command]

These create and update sub-commands require KafkaConnector custom resource specifications as descriptors. For details on these commands, see smm sdm cluster kafka-connector.

Declarative Kafka Connect connector management

The pattern for managing connectors is similar to that of Kafka Connect. There is a KafkaConnector custom resource that represents a connector type which is used by Streaming Data Manager to instantiate connectors on Kafka Connect clusters.

apiVersion: kafka.banzaicloud.io/v1beta1
kind: KafkaConnector
metadata:
  name: my-connector
spec:
  # The connector class name (e.g. io.confluent.connect.s3.S3SinkConnector)
  class:
  # The maximum number of tasks the Kafka Connector can create
  tasksMax:
  # References the Kafka Connect cluster where this connector is to run
  clusterRef:
    name: my-kafka-connect
  # Defines the config values for Kafka Connector in key - value pairs format
  config:
  # Whether to pause or resume the connector execution
  pause:

Further information

• [Kafka Connect and Connector examples](/sdm/kafka-connect/examples/