Recommendations demo application
This page shows you how to install a demo application that creates Apache Kafka brokers and topics on your cluster, and how to test the Apache Kafka deployment.
In the demo application, the Recommendations service acts as the Consumer. The Analytics, Bookings, and Catalog services are the Producers that publish data streams to the topic
- Streaming Data Manager must already be installed on Service Mesh Manager.
CAUTION:To install Streaming Data Manager on an existing Service Mesh Manager installation, the cluster must run Service Mesh Manager version 1.11.0 or later. If your cluster is running an earlier Service Mesh Manager version, you must upgrade it first.
CAUTION:When using Streaming Data Manager on Amazon EKS, you must install the EBS CSI driver add-on on your cluster.
Install the demo application
Set the your KUBECONFIG file to the primary Service Mesh Manager cluster.
Run the following command install the recommendations application. The command checks if the
ApplicationManifestCustom Resource object is available on the cluster, and installs the recommendations application.
smm demoapp install
... ✓ provision-registry-access-secrets ❯ done ✓ found valid license, checking violations name=free, cluste... ✓ ApplicationManifest is ready, enabling Recommendations app ...
Installing the recommendations application creates the following Custom Resources:
ZookeeperCluster(Name: zookeeper-server Namespace: zookeeper)
KafkaCluster(Name: kafka Namespace: kafka)
KafkaTopic(Name:: recommendations-topic, Namespace: kafka)
KafkaResourceSelector(Name: recommendation-topic, recommendation-group Namespace: kafka)
KafkaACL(Name: recommendations, analytics, bookings Namespace: kafka)
You can check that they have been properly created by using kubectl, for example:
# Get a list of KafkaTopic objects under kafka namespace kubectl get kafkatopics -n kafka
NAME AGE recommendations-topic 22m
Monitor Kafka traffic on the UI
You can display the traffic between each Kafka component (Producer, Consumer, Topic, and so on) on the Service Mesh Manager dashboard.
Run the following command to open the Service Mesh Manager dashboard. (For details, see Dashboard).
Navigate to the MENU > TOPOLOGY page, then in the Namespaces field, select the smm-demo and kafka namespaces. The traffic between our Consumer service and the Producers services.
Kafka deployment details
The dashboard gives you access to detailed information about the deployed Kafka Custom Resource objects.
The MENU > TOPICS page shows information about your Kafka topics, for example, the offset and size for each partition in recommendation-topic topic. For details, see Topics dashboard.
The MENU > BROKERS page shows information about your Kafka Brokers, for example, the amount of partitions and log size for your brokers. For details, see Brokers dashboard.
For further Kafka-related information about the Streaming Data Manager dashboard, see Dashboard.
Run a simple kcat consumer
The kcat application can act as a Kafka producer and consumer, sending messages to topics and reading them. This example creates a kcat pod and reads messages from a topic.
Open a terminal and run the following command to deploy kcat pod.
kubectl apply -n kafka -f- <<EOF apiVersion: v1 kind: Pod metadata: name: kcat spec: containers: - name: kafka-test image: "edenhill/kcat:1.7.1" # Just spin & wait forever command: [ "/bin/sh", "-c", "--" ] args: [ "while true; do sleep 3000; done;" ] EOF
Run the following command to find the Broker service of your Kafka deployment.
kubectl get services -n kafka
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kafka-0 ClusterIP 10.10.207.64 <none> 29092/TCP,29093/TCP,9020/TCP 3d18h kafka-1 ClusterIP 10.10.196.238 <none> 29092/TCP,29093/TCP,9020/TCP 3d18h kafka-all-broker ClusterIP 10.10.145.238 <none> 29092/TCP,29093/TCP 3d18h kafka-cruisecontrol-svc ClusterIP 10.10.29.105 <none> 8090/TCP,9020/TCP 3d18h kafka-kminion ClusterIP 10.10.80.242 <none> 8080/TCP 3d18h kafka-operator-alertmanager ClusterIP 10.10.12.10 <none> 9001/TCP 3d19h kafka-operator-authproxy ClusterIP 10.10.241.133 <none> 8443/TCP 3d19h kafka-operator-operator ClusterIP 10.10.23.68 <none> 443/TCP 3d19h
kafka-all-brokerservice binds to Kafka broker pods on ports
Next, run the following command to execute into the kcat pod.
kubectl exec -n kafka -it kcat sh
Start the kcat application by running the following command. The
-Cflag runs kcat as a consumer, reading messages from the
recommendations-topic(published by the analytics, bookings, and catalog services of the demo application).
kcat -b kafka-all-broker:29092 -C -t recommendations-topic
... analytics-message analytics-message analytics-message bookings-message bookings-message catalog-message catalog-message ...
You can list the metadata of the
recommendations-topicby running the following command.
kcat -b kafka-all-broker:29092 -L -t recommendations-topic
Metadata for recommendations-topic (from broker -1: kafka-all-broker:29092/bootstrap): 2 brokers: broker 0 at kafka-0.kafka.svc.cluster.local:29092 (controller) broker 1 at kafka-1.kafka.svc.cluster.local:29092 1 topics: topic "recommendations-topic" with 6 partitions: partition 0, leader 1, replicas: 1,0, isrs: 1,0 partition 1, leader 0, replicas: 0,1, isrs: 0,1 partition 2, leader 1, replicas: 1,0, isrs: 1,0 partition 3, leader 0, replicas: 0,1, isrs: 0,1 partition 4, leader 1, replicas: 1,0, isrs: 1,0 partition 5, leader 0, replicas: 0,1, isrs: 0,1
Delete the recommendations application
If you have finished testing and want to delete the recommendations application, complete the following steps.
Uninstall the demo application.
CAUTION:This command removes the entire demo application, not only the components specific to Streaming Data Manager.
smm demoapp uninstall
Remove the Kafka topic created by the recommendations application.
kubectl delete kafkatopic recommendations-topic -n kafka