Using Kafka Connect with Streaming Data Manager
Kafka Connect is a tool that allows for the scalable and reliable streaming of data between Apache Kafka and other systems. Streaming Data Manager deploys Kafka Connect with Confluent’s Community Connectors included, and supports the use of schemas via Schema Registry.
Streaming Data Manager automates the deployment of Kafka Connect clusters and the creation of connectors through KafkaConnect custom resource instances and KafkaConnector custom resource instances, respectively.
Kafka Connect API endpoints
You can access the deployed Kafka Connect clusters at the
kafka-connect-svc-<kafka-connect-name>.<namespace>.svc:<servicePort> endpoint from within the Kubernetes cluster.
Users that connect to Kafka Connect are authenticated using mTLS by default. You can disable this using the
MTLS field in the KafkaConnect custom resource.
Kafka Connect and connector ACLs
Streaming Data Manager takes care of setting up all the ACLs necessary for Kafka Connect itself. However, you must set up the ACLs for the connectors separately. Typically:
- A connector of the sink type requires
READaccess to the topics it reads, and to the consumer group named
- A connector of the source type requires
WRITEaccess to the topics it writes to.
A Kafka Connect cluster and the connectors it hosts are authenticated by the service account the Kafka Connect deployment runs with, so you must configure the connector-specific ACLs with that in mind.