FAQ

What is Streaming Data Manager?

Calisti’s Streaming Data Manager is the deployment tool for setting up and operating production-ready Apache Kafka clusters on Kubernetes, leveraging a Cloud Native technology stack. Streaming Data Manager includes Zookeeper, Koperator, Envoy, and many other components hosted in a managed service mesh. All components are automatically installed, configured, and managed in order to operate a production-ready Kafka cluster on Kubernetes.

What are the key features?

  • Designed for Cloud Natives, Streaming Data Manager provides declarative topic and user management through custom resources (CRs), and automates the setup and management of Apache Kafka on Kubernetes. It also allows you to fine-tune the configurations of your brokers individually for heterogeneous cluster layouts.
  • You can deploy managed Kafka in your own environment, including on-premises, cloud, multi-cloud, and hybrid-cloud scenarios. Streaming Data Manager can automatically scale and self-heal your cluster based on Prometheus alerts. To minimize downtime and keep your operations functional, Streaming Data Manager provides rolling upgrades and advanced Grafana dashboards to monitor all Streaming Data Manager components.
  • Our products focus on secure operations, so Streaming Data Manager automatically uses mTLS-based encrypted and authenticated communication between all components. It also provides Kubernetes-native RBAC access control, integration with Kafka ACLs, and supports multi-tenant operation.
  • To keep your services highly availability, you can deploy and manage cross-cluster replication using MirrorMaker2, and also use Kubernetes-native volume snapshots for disaster recovery. Up- and down-scaling your brokers and volumes automatically allows you to use optimal resources for the incoming traffic without manual intervention.
  • For making sure getting that getting started with Streaming Data Manager is easier our product also supports data migration from basically any Kafka cluster into Streaming Data Manager. This allows you to get up and running quicker, even if you’ve already been using Kafka.

What does the Streaming Data Manager architecture look like?

Streaming Data Manager allows you to run large Apache Kafka clusters not only in the cloud, but also on-premises, multi-cloud, and hybrid-cloud environments.

Streaming Data Manager architecture

Why is Streaming Data Manager using Istio?

Running Kafka over Istio brings additional security benefits, scalability and durability, locality based load balancing, and many other useful features, including:

  • Out of the box tracing and monitoring within a Kafka mesh
  • Consumer group metrics
  • Information about apps and their version of the client libraries
  • Request validation
  • Protocol version translations
  • Automatic topic name conversions without having to modify the clients
  • Mirroring topics to another clusters (we run many hybrid Kubernetes clusters)
  • Functional parity across runtimes
  • External Access via a single LoadBalancer

Do Service Mesh Manager and Streaming Data Manager use the same mesh?

Currently Service Mesh Manager and Streaming Data Manager use separate service meshes with separate control planes. The Streaming Data Manager service mesh is used only for the Apache Kafka brokers and the control-plane services of Streaming Data Manager. They are tied together in the sense that they are managed by the same Istio operator and use the same version of Istio.

Note that currently you cannot manage the Streaming Data Manager service mesh from the Service Mesh Manager UI, only from the command line.

What is Koperator?

Koperator (formerly called Banzai Cloud Kafka operator) is a Kubernetes operator to automate provisioning, management, autoscaling and operations of Apache Kafka clusters deployed to K8s.

Should I use Streaming Data Manager or the Koperator?

is an open-source component of the commercial Streaming Data Manager product. In addition to , Streaming Data Manager installs, configures, and manages several other components that are needed for the reliable operation and management of a Kafka cluster, and also provides several other features, including commercial support.

How does Streaming Data Manager help me scale?

Streaming Data Manager exposes Cruise-Control and Kafka JMX metrics to Prometheus, and acts as a Prometheus Alert Manager. It receives alerts defined in Prometheus, and creates actions based on Prometheus alert annotations, so it can handle and react to alerts automatically, without having to involve human operators.

Streaming Data Manager can gracefully scale your Kafka clusters both up and down, and also supports vertical capacity scaling individually for each broker, including adding new disks.

Does Streaming Data Manager support SSL?

Streaming Data Manager fully automates managed mutual TLS (mTLS) encryption and authentication. You don’t need to configure your brokers to use SSL, as Streaming Data Manager provides mTLS out-of-the box at the network layer (implemented through a light-weight, managed Istio mesh). All services deployed by Streaming Data Manager (Zookeeper, Koperator, the Kafka cluster, Cruise Control, MirrorMaker2, and so on) interact with each other using mTLS.

Kafka SSL

How does Streaming Data Manager protect my data?

Nowadays every enterprise must be prepared for an eventual service disruption. Streaming Data Manager provides multiple ways for you to get your Kafka clusters ready for disaster recovery, including MirrorMaker2 and Container Storage Interface (CSI).

Do Streaming Data Manager and Service Mesh Manager use the same components?

Some of the components are shared between Service Mesh Manager and Streaming Data Manager, while others are separate or different.

Both Service Mesh Manager and Streaming Data Manager use the following components, but have separate instances of them:

  • IstioControlPlane (called cp-v115x and sdm-icp-v115x, respectively)
  • Grafana
  • kube-state-metrics
  • node-exporter
  • Prometheus
  • prometheus-operator

Shared instances:

  • cluster-registry
  • image-pull-secrets-operator
  • the same istio-operator instance is used, but separate istio control planes are instantiated

For managing certificates of the Istio control plane, Streaming Data Manager uses the csr-operator (certificate-signing-request operator) , while Service Mesh Manager uses cert-manager.

The Istio used by Streaming Data Manager is slightly different from the one used by Service Mesh Manager, because Streaming Data Manager uses custom istio-proxy builds to handle Kafka traffic, so the sidecars are actually different.

Can I move data from an existing Kafka cluster into Streaming Data Manager?

Yes, it is possible to move data from an existing Kafka install into Streaming Data Manager. For more details and limitations of this use-case this see the page for data migration.

Why did my KafkaTopic fail when trying to create?

KafkaTopic resource instances can only be created when the KafkaCluster has enough brokers running. The Streaming Data Manager configuration interface for managing Kafka resources is unable to create or modify KafkaTopic resources until the corresponding KafkaCluster has enough brokers to fulfill the topic replication. The create operation will result in an error returned during the KafkaTopic configuration validation if the KafkaTopic configuration is inconsistent with the KafkaCluster configuration or broker status.

It is expected that different administrative workflows, likely with different personas, configure KafkaClusters and KafkaTopics. Sequencing of the operations workflows handling these configuration tasks is required.