Recommendations demo application

This page shows you how to install a demo application that creates Apache Kafka brokers and topics on your cluster, and how to test the Apache Kafka deployment.

In the demo application, the Recommendations service acts as the Consumer. The Analytics, Bookings, and Catalog services are the Producers that publish data streams to the topic recommendations-topic.

Recommendations app overview Recommendations app overview

Prerequisites

  • Streaming Data Manager must already be installed on Service Mesh Manager.

    CAUTION:

    To install Streaming Data Manager on an existing Service Mesh Manager installation, the cluster must run Service Mesh Manager version 1.11.0 or later. If your cluster is running an earlier Service Mesh Manager version, you must upgrade it first.

    CAUTION:

    When using Streaming Data Manager on Amazon EKS, you must install the EBS CSI driver add-on on your cluster.

Install the demo application

  1. Set the your KUBECONFIG file to the primary Service Mesh Manager cluster.

  2. Run the following command install the recommendations application. The command checks if the ApplicationManifest Custom Resource object is available on the cluster, and installs the recommendations application.

    smm demoapp install
    

    Expected output:

    ...
    ✓ provision-registry-access-secrets ❯ done
    ✓ found valid license, checking violations name=free, cluste...
    ✓ ApplicationManifest is ready, enabling Recommendations app
    ...
    
  3. Installing the recommendations application creates the following Custom Resources:

    • ZookeeperCluster (Name: zookeeper-server Namespace: zookeeper)
    • KafkaCluster (Name: kafka Namespace: kafka)
    • KafkaTopic (Name:: recommendations-topic, Namespace: kafka)
    • KafkaResourceSelector (Name: recommendation-topic, recommendation-group Namespace: kafka)
    • KafkaACL (Name: recommendations, analytics, bookings Namespace: kafka)

    You can check that they have been properly created by using kubectl, for example:

    # Get a list of KafkaTopic objects under kafka namespace
    kubectl get kafkatopics -n kafka
    

    Expected output:

    NAME                    AGE
    recommendations-topic   22m
    

Monitor Kafka traffic on the UI

You can display the traffic between each Kafka component (Producer, Consumer, Topic, and so on) on the Service Mesh Manager dashboard.

  1. Run the following command to open the Service Mesh Manager dashboard. (For details, see Dashboard).

    smm dashboard
    
  2. Navigate to the MENU > TOPOLOGY page, then in the Namespaces field, select the smm-demo and kafka namespaces. The traffic between our Consumer service and the Producers services.

    Kafka traffic on the Service Mesh Manager dashboard Kafka traffic on the Service Mesh Manager dashboard

Kafka deployment details

The dashboard gives you access to detailed information about the deployed Kafka Custom Resource objects.

The MENU > TOPICS page shows information about your Kafka topics, for example, the offset and size for each partition in recommendation-topic topic. For details, see Topics dashboard.

Kafka topics on the dashboard Kafka topics on the dashboard

The MENU > BROKERS page shows information about your Kafka Brokers, for example, the amount of partitions and log size for your brokers. For details, see Brokers dashboard.

Kafka brokers on the dashboard Kafka brokers on the dashboard

For further Kafka-related information about the Streaming Data Manager dashboard, see Dashboard.

Run a simple kcat consumer

The kcat application can act as a Kafka producer and consumer, sending messages to topics and reading them. This example creates a kcat pod and reads messages from a topic.

  1. Open a terminal and run the following command to deploy kcat pod.

    kubectl apply -n kafka -f- <<EOF
    apiVersion: v1
    kind: Pod
    metadata:
      name: kcat
    spec:
      containers:
      - name: kafka-test
        image: "edenhill/kcat:1.7.1"
        # Just spin & wait forever
        command: [ "/bin/sh", "-c", "--" ]
        args: [ "while true; do sleep 3000; done;" ]
    EOF
    
  2. Run the following command to find the Broker service of your Kafka deployment.

    kubectl get services -n kafka
    

    Expected output:

    NAME                          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                        AGE
    kafka-0                       ClusterIP   10.10.207.64    <none>        29092/TCP,29093/TCP,9020/TCP   3d18h
    kafka-1                       ClusterIP   10.10.196.238   <none>        29092/TCP,29093/TCP,9020/TCP   3d18h
    kafka-all-broker              ClusterIP   10.10.145.238   <none>        29092/TCP,29093/TCP            3d18h
    kafka-cruisecontrol-svc       ClusterIP   10.10.29.105    <none>        8090/TCP,9020/TCP              3d18h
    kafka-kminion                 ClusterIP   10.10.80.242    <none>        8080/TCP                       3d18h
    kafka-operator-alertmanager   ClusterIP   10.10.12.10     <none>        9001/TCP                       3d19h
    kafka-operator-authproxy      ClusterIP   10.10.241.133   <none>        8443/TCP                       3d19h
    kafka-operator-operator       ClusterIP   10.10.23.68     <none>        443/TCP                        3d19h
    

    The kafka-all-broker service binds to Kafka broker pods on ports 29092 and 29093.

  3. Next, run the following command to execute into the kcat pod.

    kubectl exec -n kafka -it kcat sh
    
  4. Start the kcat application by running the following command. The -C flag runs kcat as a consumer, reading messages from the recommendations-topic (published by the analytics, bookings, and catalog services of the demo application).

    kcat -b kafka-all-broker:29092 -C -t recommendations-topic
    

    Expected output:

    ...
    analytics-message
    analytics-message
    analytics-message
    bookings-message
    bookings-message
    catalog-message
    catalog-message
    ...
    
  5. You can list the metadata of the recommendations-topic by running the following command.

    kcat -b kafka-all-broker:29092 -L -t recommendations-topic
    

    Expected output:

    Metadata for recommendations-topic (from broker -1: kafka-all-broker:29092/bootstrap):
    2 brokers:
      broker 0 at kafka-0.kafka.svc.cluster.local:29092 (controller)
      broker 1 at kafka-1.kafka.svc.cluster.local:29092
    1 topics:
      topic "recommendations-topic" with 6 partitions:
        partition 0, leader 1, replicas: 1,0, isrs: 1,0
        partition 1, leader 0, replicas: 0,1, isrs: 0,1
        partition 2, leader 1, replicas: 1,0, isrs: 1,0
        partition 3, leader 0, replicas: 0,1, isrs: 0,1
        partition 4, leader 1, replicas: 1,0, isrs: 1,0
        partition 5, leader 0, replicas: 0,1, isrs: 0,1
    

Delete the recommendations application

If you have finished testing and want to delete the recommendations application, complete the following steps.

  1. Uninstall the demo application.

    CAUTION:

    This command removes the entire demo application, not only the components specific to Streaming Data Manager.
    smm demoapp uninstall
    
  2. Remove the Kafka topic created by the recommendations application.

    kubectl delete kafkatopic recommendations-topic -n kafka