Recommendations demo application

This page shows you how to install a demo application that creates Apache Kafka brokers and topics on your cluster, and how to test the Apache Kafka deployment.

In the demo application, the Recommendations service acts as the Consumer. The Analytics, Bookings, and Catalog services are the Producers that publish data streams to the topic recommendations-topic.

Prerequisites

Streaming Data Manager must already be installed on Service Mesh Manager.

CAUTION:
To install Streaming Data Manager on an existing Service Mesh Manager installation, the cluster must run Service Mesh Manager version 1.11.0 or later. If your cluster is running an earlier Service Mesh Manager version, you must upgrade it first.

CAUTION:
When using Streaming Data Manager on Amazon EKS, you must install the EBS CSI driver add-on on your cluster.

Install the demo application

Set the your KUBECONFIG file to the primary Service Mesh Manager cluster.
Run the following command install the recommendations application. The command checks if the ApplicationManifest Custom Resource object is available on the cluster, and installs the recommendations application.
```
smm demoapp install
```
Expected output:
```
...
✓ provision-registry-access-secrets ❯ done
✓ found valid license, checking violations name=free, cluste...
✓ ApplicationManifest is ready, enabling Recommendations app
...
```
Installing the recommendations application creates the following Custom Resources:
- ZookeeperCluster (Name: zookeeper-server Namespace: zookeeper)
- KafkaCluster (Name: kafka Namespace: kafka)
- KafkaTopic (Name:: recommendations-topic, Namespace: kafka)
You can check that they have been properly created by using kubectl, for example:
```
# Get a list of KafkaTopic objects under kafka namespace
kubectl get kafkatopics -n kafka
```
Expected output:
```
NAME                    AGE
recommendations-topic   22m
```

Monitor Kafka traffic on the UI

You can display the traffic between each Kafka component (Producer, Consumer, Topic, and so on) on the Service Mesh Manager dashboard.

Run the following command to open the Service Mesh Manager dashboard. (For details, see Dashboard).
```
smm dashboard
```
Navigate to the MENU > TOPOLOGY page, then in the Namespaces field, select the smm-demo and kafka namespaces. The traffic between our Consumer service and the Producers services.

Kafka deployment details

The dashboard gives you access to detailed information about the deployed Kafka Custom Resource objects.

The MENU > TOPICS page shows information about your Kafka topics, for example, the offset and size for each partition in recommendation-topic topic. For details, see Topics dashboard.

The MENU > BROKERS page shows information about your Kafka Brokers, for example, the amount of partitions and log size for your brokers. For details, see Brokers dashboard.

For further Kafka-related information about the Streaming Data Manager dashboard, see Dashboard.

Run a simple kcat consumer

The kcat application can act as a Kafka producer and consumer, sending messages to topics and reading them. This example creates a kcat pod and reads messages from a topic.

Open a terminal and run the following command to deploy kcat pod.

kubectl apply -n kafka -f- <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: kcat
spec:
  containers:
  - name: kafka-test
    image: "edenhill/kcat:1.7.1"
    # Just spin & wait forever
    command: [ "/bin/sh", "-c", "--" ]
    args: [ "while true; do sleep 3000; done;" ]
EOF

Run the following command to find the Broker service of your Kafka deployment.

kubectl get services -n kafka

Expected output:

NAME                          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                        AGE
kafka-0                       ClusterIP   10.10.207.64    <none>        29092/TCP,29093/TCP,9020/TCP   3d18h
kafka-1                       ClusterIP   10.10.196.238   <none>        29092/TCP,29093/TCP,9020/TCP   3d18h
kafka-all-broker              ClusterIP   10.10.145.238   <none>        29092/TCP,29093/TCP            3d18h
kafka-cruisecontrol-svc       ClusterIP   10.10.29.105    <none>        8090/TCP,9020/TCP              3d18h
kafka-kminion                 ClusterIP   10.10.80.242    <none>        8080/TCP                       3d18h
kafka-operator-alertmanager   ClusterIP   10.10.12.10     <none>        9001/TCP                       3d19h
kafka-operator-authproxy      ClusterIP   10.10.241.133   <none>        8443/TCP                       3d19h
kafka-operator-operator       ClusterIP   10.10.23.68     <none>        443/TCP                        3d19h

The kafka-all-broker service binds to Kafka broker pods on ports 29092 and 29093.

Next, run the following command to execute into the kcat pod.
```
kubectl exec -n kafka -it kcat sh
```
Start the kcat application by running the following command. The -C flag runs kcat as a consumer, reading messages from the recommendations-topic (published by the analytics, bookings, and catalog services of the demo application).
```
kcat -b kafka-all-broker:29092 -C -t recommendations-topic
```
Expected output:
```
...
analytics-message
analytics-message
analytics-message
bookings-message
bookings-message
catalog-message
catalog-message
...
```

You can list the metadata of the recommendations-topic by running the following command.

kcat -b kafka-all-broker:29092 -L -t recommendations-topic

Expected output:

Metadata for recommendations-topic (from broker -1: kafka-all-broker:29092/bootstrap):
2 brokers:
  broker 0 at kafka-0.kafka.svc.cluster.local:29092 (controller)
  broker 1 at kafka-1.kafka.svc.cluster.local:29092
1 topics:
  topic "recommendations-topic" with 6 partitions:
    partition 0, leader 1, replicas: 1,0, isrs: 1,0
    partition 1, leader 0, replicas: 0,1, isrs: 0,1
    partition 2, leader 1, replicas: 1,0, isrs: 1,0
    partition 3, leader 0, replicas: 0,1, isrs: 0,1
    partition 4, leader 1, replicas: 1,0, isrs: 1,0
    partition 5, leader 0, replicas: 0,1, isrs: 0,1

Delete the recommendations application

If you have finished testing and want to delete the recommendations application, complete the following steps.

Uninstall the demo application.

CAUTION:
This command removes the entire demo application, not only the components specific to Streaming Data Manager.
```
smm demoapp uninstall
```
Remove the Kafka topic created by the recommendations application.
```
kubectl delete kafkatopic recommendations-topic -n kafka
```