Using Kafka Connect with Streaming Data Manager

Kafka Connect is a tool that allows for the scalable and reliable streaming of data between Apache Kafka and other systems. Streaming Data Manager deploys Kafka Connect with Confluent’s Community Connectors included, and supports the use of schemas via Schema Registry.

Streaming Data Manager automates the deployment of Kafka Connect clusters and the creation of connectors through KafkaConnect custom resource instances and KafkaConnector custom resource instances, respectively.

Kafka Connect API endpoints

You can access the deployed Kafka Connect clusters at the kafka-connect-svc-<kafka-connect-name>.<namespace>.svc:<servicePort> endpoint from within the Kubernetes cluster.

Security

Users that connect to Kafka Connect are authenticated using mTLS by default. You can disable this using the MTLS field in the KafkaConnect custom resource.

Kafka Connect and connector ACLs

Streaming Data Manager takes care of setting up all the ACLs necessary for Kafka Connect itself. However, you must set up the ACLs for the connectors separately. Typically:

• A connector of the sink type requires READ access to the topics it reads, and to the consumer group named connect-{connector-name}.
• A connector of the source type requires WRITE access to the topics it writes to.

A Kafka Connect cluster and the connectors it hosts are authenticated by the service account the Kafka Connect deployment runs with, so you must configure the connector-specific ACLs with that in mind.