Client applications outside the Kubernetes cluster

This scenario covers using Kafka ACLs when your client applications are outside the Kubernetes cluster. As a result, the client applications must connect to the endpoint bound to the Kafka cluster’s external listener. In this scenario, the client applications must present a client certificate to authenticate themselves.

Prerequisites

To use Kafka ACLs with Istio mTLS, you need:

• a Kubernetes cluster (version 1.19 and above), with
• at least 12 vCPU and 12 GB of memory, and
• with the capability to provision LoadBalancer Kubernetes services.
• A Kafka cluster.

There are two ways in this documentation to sign certificate for the clients:

• Use our CSR operator
• Use cert-manager

Use our CSR operator

1. Enable ACLs and configure an external listener using Streaming Data Manager. Complete the following steps.

   1. Verify that your deployed Kafka cluster is up and running:

      supertubes cluster get --namespace <namespace-of-your-cluster> --kafka-cluster <name-of-your-kafka-cluster> --kubeconfig <path-to-kubeconfig-file>
      

      Expected output:

      Namespace  Name   State           Image                               Alerts  Cruise Control Topic Status  Rolling Upgrade Errors  Rolling Upgrade Last Success
      kafka      kafka  ClusterRunning  banzaicloud/kafka:2.13-2.5.0-bzc.1  0       CruiseControlTopicReady      0
      

   2. Enable ACLs and configure an external listener. The deployed Kafka cluster has no ACLs, and external access is disabled by default. Enable them by applying the following changes:

      supertubes cluster update --namespace kafka --kafka-cluster kafka --kubeconfig <path-to-kubeconfig-file> -f -<<EOF
      apiVersion: kafka.banzaicloud.io/v1beta1
      kind: KafkaCluster
      spec:
        ingressController: "istioingress"
        istioIngressConfig:
          gatewayConfig:
            mode: PASSTHROUGH
      readOnlyConfig: |
          auto.create.topics.enable=false
          offsets.topic.replication.factor=2
          authorizer.class.name=kafka.security.authorizer.AclAuthorizer
          allow.everyone.if.no.acl.found=false    
      listenersConfig:
          externalListeners:
          - type: "plaintext"
            name: "external"
            externalStartingPort: 19090
            containerPort: 9094
      EOF
      

   3. The update in the previous step reconfigures the Kafka cluster to receive rolling updates. Verify that this is reflected in the state of the cluster.

      supertubes cluster get --namespace kafka --kafka-cluster kafka --kubeconfig <path-to-kubeconfig-file>
      

      Expected output:

      Namespace  Name   State                    Image                               Alerts  Cruise Control Topic Status  Rolling Upgrade Errors  Rolling Upgrade Last Success
      kafka      kafka  ClusterRollingUpgrading  banzaicloud/kafka:2.13-2.5.0-bzc.1  0       CruiseControlTopicReady      0
      

   4. Wait until the reconfiguration is finished and the cluster is in the ClusterRunning state. This can take a while, as the rolling upgrade applies changes on a broker-by-broker basis.

2. Get the endpoint bound to the external listener of the Kafka cluster:

   kubectl get service -n kafka meshgateway-external-kafka
   

   Example output:

   NAME                         TYPE           CLUSTER-IP     EXTERNAL-IP                                                                PORT(S)
   meshgateway-external-kafka   LoadBalancer   10.10.44.209   aff4c6887766440238fb19c381779eae-1599690198.eu-north-1.elb.amazonaws.com   19090:32480/TCP,19091:30972/TCP,29092:30748/TCP
   

3. Create a Kafka user that the client application will use to identify itself.

   kubectl create -f - <<EOF
   apiVersion: kafka.banzaicloud.io/v1alpha1
   kind: KafkaUser
   metadata:
     name: external-kafkauser
     namespace: default
   spec:
     clusterRef:
       name: kafka
       namespace: kafka
     secretName: external-kafkauser-secret
     pkiBackendSpec:
       pkiBackend: "k8s-csr"
       signerName: "csr.banzaicloud.io/privateca"
   EOF
   

   Grant this user access to the topics it needs.

4. Export the public certificate of the CA.

   • If you are using the CSR operator:

     kubectl get secret external-kafkauser-secret -o 'go-template={{index .data "chain.pem"}}' | base64 -D > /var/tmp/ca.crt
     

   • If you are using cert-manager:

     kubectl get secret external-kafkauser-secret -o 'go-template={{index .data "ca.crt"}}' | base64 -D > /var/tmp/ca.crt 
     

   Alternatively, you can download the CA certificate and the client certificate from the Streaming Data Manager web interface.

5. Export the client certificate stored in external-kafkauser-secret which represents Kafka user external-kafkauser:

   kubectl get secret external-kafkauser-secret -o 'go-template={{index .data "tls.crt"}}' | base64 -D > /var/tmp/tls.crt
   kubectl get secret external-kafkauser-secret -o 'go-template={{index .data "tls.key"}}' | base64 -D > /var/tmp/tls.key
   

6. Use the exported client credentials and the CA certificate in your application to connect to the external listener of the Kafka cluster. (Otherwise, Istio automatically rejects the client application.) The following command is an example to connect with the kcat client application:

   kcat -L -b aff4c6887766440238fb19c381779eae-1599690198.eu-north-1.elb.amazonaws.com:29092 -X security.protocol=SSL -X ssl.key.location=/var/tmp/tls.key -X ssl.certificate.location=/var/tmp/tls.crt -X ssl.ca.location=/var/tmp/ca.crt
   Metadata for all topics (from broker -1: ssl://aff4c6887766440238fb19c381779eae-1599690198.eu-north-1.elb.amazonaws.com:29092/bootstrap):
   2 brokers:
   broker 0 at aff4c6887766440238fb19c381779eae-1599690198.eu-north-1.elb.amazonaws.com:19090 (controller)
   broker 1 at aff4c6887766440238fb19c381779eae-1599690198.eu-north-1.elb.amazonaws.com:19091
   1 topics:
   topic "example-topic" with 3 partitions:
   partition 0, leader 0, replicas: 0,1, isrs: 0,1
   partition 1, leader 1, replicas: 1,0, isrs: 0,1
   partition 2, leader 0, replicas: 0,1, isrs: 0,1
   

Use cert-manager

1. Enable ACLs and configure an external listener using Streaming Data Manager. Complete the following steps.

   1. Verify that your deployed Kafka cluster is up and running:

      supertubes cluster get --namespace <namespace-of-your-cluster> --kafka-cluster <name-of-your-kafka-cluster> --kubeconfig <path-to-kubeconfig-file>
      

      Expected output:

      Namespace  Name   State           Image                               Alerts  Cruise Control Topic Status  Rolling Upgrade Errors  Rolling Upgrade Last Success
      kafka      kafka  ClusterRunning  banzaicloud/kafka:2.13-2.5.0-bzc.1  0       CruiseControlTopicReady      0
      

   2. Enable ACLs and configure an external listener. The deployed Kafka cluster has no ACLs, and external access is disabled by default. Enable them by applying the following changes:

      supertubes cluster update --namespace kafka --kafka-cluster kafka --kubeconfig <path-to-kubeconfig-file> -f -<<EOF
      apiVersion: kafka.banzaicloud.io/v1beta1
      kind: KafkaCluster
      spec:
        ingressController: "istioingress"
        istioIngressConfig:
          gatewayConfig:
            mode: PASSTHROUGH
      readOnlyConfig: |
          auto.create.topics.enable=false
          offsets.topic.replication.factor=2
          authorizer.class.name=kafka.security.authorizer.AclAuthorizer
          allow.everyone.if.no.acl.found=false    
      listenersConfig:
          externalListeners:
          - type: "plaintext"
            name: "external"
            externalStartingPort: 19090
            containerPort: 9094
      EOF
      

   3. The update in the previous step reconfigures the Kafka cluster to receive rolling updates. Verify that this is reflected in the state of the cluster.

      supertubes cluster get --namespace kafka --kafka-cluster kafka --kubeconfig <path-to-kubeconfig-file>
      

      Expected output:

      Namespace  Name   State                    Image                               Alerts  Cruise Control Topic Status  Rolling Upgrade Errors  Rolling Upgrade Last Success
      kafka      kafka  ClusterRollingUpgrading  banzaicloud/kafka:2.13-2.5.0-bzc.1  0       CruiseControlTopicReady      0
      

   4. Wait until the reconfiguration is finished and the cluster is in the ClusterRunning state. This can take a while, as the rolling upgrade applies changes on a broker-by-broker basis.

2. Install cert-manager.

   1. Install cert-manager on the cluster. The cert-manager application will issue the client certificates for the client applications. If you already have cert-manager installed and configured on the cluster, skip this step.

      kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.3/cert-manager.yaml
      

   2. Specify a cluster issuer for cert-manager that has the same CA or root certificate as the Istio mesh, otherwise, the application’s client certificate won’t be valid for the mTLS enforced by Istio.

      1. Get the CA certificate used by Istio:

         kubectl get secrets -n istio-system istio-ca-secret -o yaml
         

         This secret has different fields than what cert-manager expects.

      2. Create a new secret from this in a format that works for cert-manager.

         kubectl create -f - <<EOF
         apiVersion: v1
         kind: Secret
         metadata:
           name: ca-key-pair
           namespace: cert-manager
         data:
           tls.crt: <tls-crt-from-istio-ca-secret>
           tls.key: <your-tls-key-from-istio-ca-secret>
         EOF
         

         kubectl create -f - <<EOF
         apiVersion: cert-manager.io/v1alpha2
         kind: ClusterIssuer
         metadata:
           name: ca-issuer
           namespace: cert-manager
         spec:
           ca:
             secretName: ca-key-pair
         EOF
         

3. Get the endpoint bound to the external listener of the Kafka cluster:

   kubectl get service -n kafka meshgateway-external-kafka
   

   Example output:

   NAME                         TYPE           CLUSTER-IP     EXTERNAL-IP                                                                PORT(S)
   meshgateway-external-kafka   LoadBalancer   10.10.44.209   aff4c6887766440238fb19c381779eae-1599690198.eu-north-1.elb.amazonaws.com   19090:32480/TCP,19091:30972/TCP,29092:30748/TCP
   

4. Create a Kafka user that the client application will use to identify itself.

   kubectl create -f - <<EOF
   apiVersion: kafka.banzaicloud.io/v1alpha1
   kind: KafkaUser
   metadata:
     name: external-kafkauser
     namespace: default
   spec:
     clusterRef:
       name: kafka
       namespace: kafka
     secretName: external-kafkauser-secret
     pkiBackendSpec:
       pkiBackend: "cert-manager"
       issuerRef:
         name: "ca-issuer"
         kind: "ClusterIssuer"
   EOF
   

   Grant this user access to the topics it needs.

5. Export the public certificate of the CA.

   • If you are using the CSR operator:

     kubectl get secret external-kafkauser-secret -o 'go-template={{index .data "chain.pem"}}' | base64 -D > /var/tmp/ca.crt
     

   • If you are using cert-manager:

     kubectl get secret external-kafkauser-secret -o 'go-template={{index .data "ca.crt"}}' | base64 -D > /var/tmp/ca.crt 
     

   Alternatively, you can download the CA certificate and the client certificate from the Streaming Data Manager web interface.

6. Export the client certificate stored in external-kafkauser-secret which represents Kafka user external-kafkauser:

   kubectl get secret external-kafkauser-secret -o 'go-template={{index .data "tls.crt"}}' | base64 -D > /var/tmp/tls.crt
   kubectl get secret external-kafkauser-secret -o 'go-template={{index .data "tls.key"}}' | base64 -D > /var/tmp/tls.key
   

7. Use the exported client credentials and the CA certificate in your application to connect to the external listener of the Kafka cluster. (Otherwise, Istio automatically rejects the client application.) The following command is an example to connect with the kcat client application:

   kcat -L -b aff4c6887766440238fb19c381779eae-1599690198.eu-north-1.elb.amazonaws.com:29092 -X security.protocol=SSL -X ssl.key.location=/var/tmp/tls.key -X ssl.certificate.location=/var/tmp/tls.crt -X ssl.ca.location=/var/tmp/ca.crt
   Metadata for all topics (from broker -1: ssl://aff4c6887766440238fb19c381779eae-1599690198.eu-north-1.elb.amazonaws.com:29092/bootstrap):
   2 brokers:
   broker 0 at aff4c6887766440238fb19c381779eae-1599690198.eu-north-1.elb.amazonaws.com:19090 (controller)
   broker 1 at aff4c6887766440238fb19c381779eae-1599690198.eu-north-1.elb.amazonaws.com:19091
   1 topics:
   topic "example-topic" with 3 partitions:
   partition 0, leader 0, replicas: 0,1, isrs: 0,1
   partition 1, leader 1, replicas: 1,0, isrs: 0,1
   partition 2, leader 0, replicas: 0,1, isrs: 0,1