What's new

Release 1.8.0 (2022-11-07)

Streaming Data Manager is now part of Calisti.

Release 1.7.0 (2022-04-28)

Create Kafka users on the UI

Starting with Streaming Data Manager version 1.7.0, you can create and delete Kafka users from the Streaming Data Manager UI, and download the certificates of the users in various formats. For details, see Kafka users.

Kafka on multi-cluster Kubernetes

Your Apache Kafka clusters can now span multiple Kubernetes clusters. Streaming Data Manager makes it easy to set up your Kafka clusters to have nodes in multiple Kubernetes clusters, and automatically keeps the resources synchronized between the member clusters. For details, see Synchronizing SDM resources across multiple Kubernetes Clusters.

Koperator upgrade

The koperator component has been upgraded from version 0.20.2 to version 0.21.2 to include the latest bugfixes and features.

Prometheus upgrade

The Prometheus component has been upgraded from version 12.10.6 to version 34.10.0 to include the latest bugfixes and features.

Istio 1.11 support

Streaming Data Manager now supports Istio 1.11.

Release 1.6.3 (2022-02-04)

Fix: MirrorMaker2 unable to connect to all brokers on remote cluster

Koperator upgrade

The koperator component has been upgraded from version 0.20.1 to version 0.20.2 to incorporate the bugfix that addresses the multiple Kafka topic creation issue.

Release 1.6.2 (2021-12-21)

Koperator upgrade

The koperator component has been upgraded from version 0.20.0 to version 0.20.1 to include the latest security and bugfixes.

Other component upgrades

  • Bump up embedded Pomerium version from v0.14.7 to v0.15.6 to avoid CVE-2021-41230
  • Updated jmx-exporter used for publishing metrics to Prometheus from version 0.14.0 to version 0.16.1
  • Bump up Cruise Control version used by the Kafka demo cluster to v2.5.79 to avoid CVE-2021-44228

Release 1.6.1 (2021-11-26)

Upstream Istio support

Now you can run Streaming Data Manager using its own service mesh together with an upstream Istio service mesh, allowing you to you to manage Istio separately from Streaming Data Manager. For details, see How to run SDM with upstream Istio in the same Kubernetes cluster.

Koperator upgrade

The koperator component has been upgraded from version 0.19.0 to version 0.20.0 to include the latest bugfixes.

Release 1.6.0 (2021-10-27)

Use a custom Certificate Authority with Istio

You can modify the default Istio Custom Resource (CR) provided by Streaming Data Manager to provision workload certificates using a custom certificate authority (CA) which integrates with the Kubernetes Certificate Signing Request (CSR) API. For details, see Use a custom Certificate Authority with Istio.

Topic edit, create, delete through SDM UI

You can now create, delete, and edit topics using the Streaming Data Manager web interface as well. For details, see Topics dashboard.

Apache Kafka support

Streaming Data Manager now supports Kafka 2.8.1 as well.

Koperator upgrade

The koperator component has been upgraded from version 0.18.3 to version 0.19.0. The main new features and changes of this version are the following:

  • Support for TopologySpreadConstraints in Envoy ingress deployments
  • Expose an http healthcheck endpoint to Envoy anycast ingress
  • Vault support has been removed from the operator

Release 1.5.0 (2021-09-22)

OIDC login support

In version 1.5.0, you can secure Streaming Data Manager access using an external ingress, and authenticate and authorize user connections. The solution is based on Pomerium, allowing you to use the identity provider of your choice. You can also apply role based authorization, and assign read-only or read and write access to Streaming Data Manager. For details, see Accessing the UI.

Kubernetes support

Streaming Data Manager now supports Kubernetes version 1.21. Note that the minimal required Kubernetes version has been increased to 1.19.

Istio 1.10 support

Streaming Data Manager now supports Istio 1.10.

Koperator upgrade

The kafka-operator component has been upgraded from version 0.17.0 to koperator version 0.18.3. This upgrade brings you the following new features:

  • downScale alert supports broker id based scaling
  • You can define a sidecar container to the operator
  • Enable Affinity settings for the envoy deployment
  • Enable Resource settings for the initcontainers
  • Enable Pod Disruption budget to Envoy deployment
  • Support for K8s-CSR when creating new KafkaUser
  • You can define a sidecar container for the broker pods

Other component upgrades

The underlying components of Streaming Data Manager have been upgraded to:

  • cert-manager 1.5.3
  • Cruise Control Module version 2.5.68-1.0
  • Kafka Minion component to version KMinion 2.1
  • Zookeeper operator version 0.2.11

Bug fixes

  • After enabling ACLs on Kafka 2.8 the cluster is not usable until all broker are not restarted manually
  • IMPS operator should run outside of the Istio mesh
  • For details on bugfixes related to the Koperator component, see the Koperator release notes.

Release 1.4.1 (2021-08-09)

This patch release contains the following changes:

  • The IMPS operator runs outside the mesh.
  • For demo purposes, Apache Kafka version 2.6.2 is used instead of 2.8., as 2.8.0 has a regression. This regression impacts Kafka 2.7.1 as well, thus it’s safer to use Kafka 2.6.2.

Release 1.4 (2021-06-16)

Release 1.4 focuses on compliance, component upgrades, and various bugfixes across all components.

Kafka authentication based on OAuth2 JWT tokens

Streaming Data Manager allows you to use Kafka ACLs with a third-party identity provider using a OAuth 2.0 Bearer Token based authentication. Streaming Data Manager supports OAuth Bearer Token authentication for both external listeners and internal listeners. For details, see Enable OAuth Bearer Token Authentication for listeners.

Kafka ACLs dashboard

To make troubleshooting errors related to client permissions easier and faster, the Streaming Data Manager web interface now has a dashboard that shows Kafka ACLs. The ACL Dashboard gives you an overview of the KafkaACLs, Roles, Resource Selectors and Classic ACLs of your Kafka clusters.

Kafka ACLs dashboard Kafka ACLs dashboard

For details, see ACL dashboard.

The Grafana dashboard for Kafka clusters has also been improved.

Installation images

The private images and binaries of Streaming Data Manager are now accessible from Cisco’s ECR repository. For details, see Accessing the Streaming Data Manager binaries.

Apache Kafka and Koperator support

Streaming Data Manager now supports Kafka versions 2.6.2, 2.7.1 and 2.8.0, and uses Koperator version 0.17.0.

Istio 1.9.5 support

Streaming Data Manager now supports Istio 1.9.

Release 1.3.2 (2020-04-19)

SASL authentication on external listeners

For external listeners, Streaming Data Manager now supports the SASL Plaintext protocol. For details, see SASL authentication on external listeners.

Apache Kafka and Istio support

Streaming Data Manager now supports Istio 1.9.1 and Kafka 2.7.0.

Other changes

  • The Kafka JMX exporter now collects only necessary Kafka metrics.
  • The Streaming Data Manager CLI can handle the inter.broker.protocol.version Kafka configuration.

Release 1.2.1 (2020-04-06)

Metrics improvements

Streaming Data Manager now uses Zookeeper’s native Prometheus metrics. The related Zookeeper Grafana Helm charts have been updated.

Apache Kafka and Istio support

Streaming Data Manager now supports Istio 1.8.2 and Kafka 2.6.1.

Upgraded components

This release upgrades the following notable components:

  • Upgrade Schema Registry version to 6.0.0
  • Upgrade ksqlDB version to 0.13.0
  • Upgraded the embedded Prometheus Operator Helm chart to version 12.10.6
  • cert-manager v0.15.2 is now embedded into Streaming Data Manager

Release 1.1 (2020-11-10)

ksqlDB support

Streaming Data Manager supports, automates, and secures ksqlDB both in headless and interactive mode. With Streaming Data Manager, ksqlDB is pre-configured to use mTLS and advanced authorization, out-of-the-box. In addition, it uses Istio’s Authorization Policy instead of the usual HTTP-Basic authorization.

To support operating ksqlDB deployments, Streaming Data Manager not only supports scaling through CPU or memory usage, but also allows scaling based on consumer lag.

Release 1.0 (2020-09-16)

Setting up and managing Kafka Connect clusters

Kafka Connect is a tool that allows for the scalable and reliable streaming of data between Apache Kafka and other systems. Streaming Data Manager deploys Kafka Connect with Confluent’s Community Connectors included, and supports the use of schemas via Schema Registry.

Streaming Data Manager automates the deployment of Kafka Connect clusters and the creation of connectors through KafkaConnect custom resource instances and KafkaConnector custom resource instances, respectively. For details, see Using Kafka Connect with Streaming Data Manager.

Streaming Data Manager dashboard

The dashboard helps you monitor all the key metrics of your Kafka cluster at a glance and in a single place. Colors indicate potential issues and what you should keep an eye on, so you can prepare for anomalies and then act accordingly. You can observe the core components of your cluster under the labels of Topics, Brokers and Consumer Groups, and dig deeper into each of these for further information - like number of messages, partition sizes, low and high watermark offsets, in-sync replicas, and so on. For details, see Dashboard.

Streaming Data Manager dashboard Streaming Data Manager dashboard