Calisti

• 1: What's new
• 2: Service Mesh Manager
• 2.1: Overview
• 2.1.1: Features
• 2.1.1.1: Istio distribution
• 2.1.1.2: Mesh lifecycle management
• 2.1.1.3: Observability toolbox
• 2.1.1.4: Multi-cluster topologies
• 2.1.2: Modes of operation
• 2.1.3: Istio-operator feature comparison
• 2.2: Getting started with the Free Tier
• 2.3: Installation
• 2.3.1: Prerequisites
• 2.3.1.1: Accessing the Service Mesh Manager binaries
• 2.3.1.2: Create a test cluster
• 2.3.2: Licensing options
• 2.3.3: Create single cluster mesh
• 2.3.4: Create multi-cluster mesh
• 2.3.5: SMM Operator helm charts
• 2.3.5.1: Install SMM with the SMM Operator chart
• 2.3.5.2: The ControlPlane Custom Resource
• 2.3.6: Install SMM - GitOps - single cluster
• 2.3.7: Install SMM - GitOps - multi-cluster
• 2.3.8: Install FIPS images
• 2.3.9: Customize installation
• 2.4: Upgrade
• 2.4.1: Upgrading SMM and SDM
• 2.4.2: Canary control plane upgrades
• 2.4.3: Upgrade SMM - GitOps - single cluster
• 2.4.4: Upgrade SMM - GitOps - multi-cluster
• 2.4.5: Multi-cluster upgrade from 1.10.0 to 1.11.0
• 2.5: Dashboard
• 2.5.1: Dashboard overview
• 2.5.2: Mesh
• 2.5.2.1: Validation
• 2.5.3: Topology view
• 2.5.3.1: Drill-down
• 2.5.3.2: Generate test load
• 2.5.3.3: Pod logs
• 2.5.4: Workloads
• 2.5.5: Services
• 2.5.6: Gateways
• 2.5.6.1: Create ingress gateway
• 2.5.6.2: Create egress gateway
• 2.5.6.3: Manage host and port configuration
• 2.5.6.4: Routes and traffic management
• 2.5.6.5: Using Let's Encrypt with your own domain name
• 2.5.7: Traffic tap
• 2.5.8: Managing Kafka clusters
• 2.5.9: Configuring the Dashboard
• 2.5.9.1: Exposing the Dashboard
• 2.5.9.2: OIDC authentication
• 2.5.9.2.1: Using Dex for authentication
• 2.6: Mesh Management
• 2.6.1: Multi-cluster - single mesh
• 2.6.1.1: Cluster network
• 2.6.1.2: Attach a new cluster to the mesh
• 2.6.1.3: Deploy applications on multiple clusters
• 2.6.1.4: Detach a cluster from the mesh
• 2.6.1.5: Cluster registry controller
• 2.6.2: Traffic management
• 2.6.2.1: Circuit Breaking
• 2.6.2.2: External Services
• 2.6.2.3: Fault Injection
• 2.6.2.4: Ingress gateways
• 2.6.2.5: Mirroring
• 2.6.2.6: Restrict Outbound Traffic of Workloads
• 2.6.2.7: Routing
• 2.6.2.8: Sidecar injection
• 2.6.3: Istio configuration validation
• 2.7: Convenience features
• 2.7.1: Tracking Service Level Objectives (SLOs)
• 2.7.1.1: Create new Service Level Objectives (SLOs)
• 2.7.1.2: Create new Alerting Policy
• 2.7.1.3: Service Level Indicator template
• 2.7.1.4: Create custom Service Level Objective
• 2.7.2: Flagger Canary
• 2.7.3: Integrated monitoring in SMM
• 2.7.4: Integrated tracing in SMM
• 2.7.5: Let's Encrypt support for gateways
• 2.8: Integrating Virtual Machines into the mesh
• 2.8.1: Prerequisites
• 2.8.2: Quickstart
• 2.8.3: Dashboard
• 2.8.4: Use-cases
• 2.8.4.1: Add a virtual machine to the mesh
• 2.8.4.2: VM to Kubernetes migration
• 2.8.4.3: Upgrading Istio
• 2.8.4.4: Autoscaling VM groups
• 2.8.4.5: Maintenance
• 2.8.4.6: Remove VM from the mesh
• 2.8.5: Istio resources
• 2.8.6: Known limitations
• 2.9: Operations
• 2.9.1: Scaling and performance tuning
• 2.9.1.1: Finding bottlenecks
• 2.9.1.2: Scale a specific Workload
• 2.9.1.3: Scale the Istio proxy sidecar of a workload
• 2.9.1.4: Scale the Istio Control Plane
• 2.9.1.5: Scale SMM Control Plane
• 2.9.1.6: Scale the built-in Prometheus
• 2.9.2: SLO-based alerting in Production
• 2.9.2.1: Set up Persistent Volumes for Prometheus
• 2.9.2.2: Set up High Availability for the Monitoring Stack
• 2.9.2.3: Deploy a new Alertmanager
• 2.9.2.4: Use an existing Alertmanager
• 2.9.3: Global DNS
• 2.9.4: GitOps
• 2.10: Security
• 2.10.1: Zero Trust Security
• 2.10.2: Authentication
• 2.10.3: Mutual TLS
• 2.10.4: Authorization
• 2.10.5: FIPS-compliant service mesh
• 2.10.6: Open Port Inventory
• 2.10.7: Recovery procedure
• 2.10.8: Fetch CA certificate from Vault
• 2.11: Frequently Asked Questions
• 2.11.1: Cisco Service Mesh Manager
• 2.11.2: Service mesh FAQ
• 2.11.3: Observability and debugging
• 2.11.4: Security and compliance
• 2.11.5: Multi-cluster support
• 2.11.6: Licensing
• 2.11.7: Streaming Data Manager
• 2.12: Reference
• 2.12.1: API
• 2.12.2: CRD
• 2.12.2.1: ControlPlane CRD schema reference (group smm.cisco.com)
• 2.12.2.2: IstioOperator CRD schema reference (group smm.cisco.com)
• 2.12.2.3: License CRD schema reference (group smm.cisco.com)
• 2.12.2.4: SRE API
• 2.12.2.4.1: AlertingPolicy CRD schema reference (group sre.smm.cisco.com)
• 2.12.2.4.2: AlertingTemplate CRD schema reference (group sre.smm.cisco.com)
• 2.12.2.4.3: ServiceLevelIndicatorTemplate CRD schema reference (group sre.smm.cisco.com)
• 2.12.2.4.4: ServiceLevelObjective CRD schema reference (group sre.smm.cisco.com)
• 2.12.3: The command line interface
• 2.12.3.1: Command reference
• 2.12.3.1.1: smm
• 2.12.3.1.2: smm activate
• 2.12.3.1.3: smm activate-ecr
• 2.12.3.1.4: smm analyze
• 2.12.3.1.5: smm cert-manager
• 2.12.3.1.6: smm cert-manager install
• 2.12.3.1.7: smm cert-manager uninstall
• 2.12.3.1.8: smm clouddoor
• 2.12.3.1.9: smm clouddoor install
• 2.12.3.1.10: smm clouddoor uninstall
• 2.12.3.1.11: smm config
• 2.12.3.1.12: smm config delete
• 2.12.3.1.13: smm config edit
• 2.12.3.1.14: smm config view
• 2.12.3.1.15: smm dashboard
• 2.12.3.1.16: smm demoapp
• 2.12.3.1.17: smm demoapp install
• 2.12.3.1.18: smm demoapp load
• 2.12.3.1.19: smm demoapp load start
• 2.12.3.1.20: smm demoapp load stop
• 2.12.3.1.21: smm demoapp uninstall
• 2.12.3.1.22: smm graph
• 2.12.3.1.23: smm install
• 2.12.3.1.24: smm istio
• 2.12.3.1.25: smm istio cluster
• 2.12.3.1.26: smm istio cluster attach
• 2.12.3.1.27: smm istio cluster detach
• 2.12.3.1.28: smm istio cluster status
• 2.12.3.1.29: smm istio install
• 2.12.3.1.30: smm istio outbound-traffic-policy
• 2.12.3.1.31: smm istio overview
• 2.12.3.1.32: smm istio uninstall
• 2.12.3.1.33: smm license
• 2.12.3.1.34: smm license apply
• 2.12.3.1.35: smm license list
• 2.12.3.1.36: smm license remove
• 2.12.3.1.37: smm login
• 2.12.3.1.38: smm mirror-maker2
• 2.12.3.1.39: smm operator
• 2.12.3.1.40: smm operator reconcile
• 2.12.3.1.41: smm prometheus
• 2.12.3.1.42: smm registry
• 2.12.3.1.43: smm registry add
• 2.12.3.1.44: smm registry ecr
• 2.12.3.1.45: smm registry list
• 2.12.3.1.46: smm registry override
• 2.12.3.1.47: smm registry override reset
• 2.12.3.1.48: smm registry override set
• 2.12.3.1.49: smm registry override status
• 2.12.3.1.50: smm registry reconcile
• 2.12.3.1.51: smm registry remove
• 2.12.3.1.52: smm sdm
• 2.12.3.1.53: smm sdm cluster
• 2.12.3.1.54: smm sdm cluster kafka-connect
• 2.12.3.1.55: smm sdm cluster kafka-connect create
• 2.12.3.1.56: smm sdm cluster kafka-connect delete
• 2.12.3.1.57: smm sdm cluster kafka-connect get
• 2.12.3.1.58: smm sdm cluster kafka-connect update
• 2.12.3.1.59: smm sdm cluster kafka-connector
• 2.12.3.1.60: smm sdm cluster kafka-connector
• 2.12.3.1.61: smm sdm cluster kafka-connector
• 2.12.3.1.62: smm sdm cluster kafka-connector delete
• 2.12.3.1.63: smm sdm cluster kafka-connector get
• 2.12.3.1.64: smm sdm cluster update
• 2.12.3.1.65: smm sidecar-proxy
• 2.12.3.1.66: smm sidecar-proxy auto-inject
• 2.12.3.1.67: smm sidecar-proxy auto-inject off
• 2.12.3.1.68: smm sidecar-proxy auto-inject on
• 2.12.3.1.69: smm sidecar-proxy egress
• 2.12.3.1.70: smm sidecar-proxy egress recommend
• 2.12.3.1.71: smm tap
• 2.12.3.1.72: smm uninstall
• 2.12.3.1.73: smm version
• 2.12.3.1.74: smm zookeeper
• 2.12.3.2: Upgrade Service Mesh Manager CLI
• 3: Streaming Data Manager
• 3.1: Overview
• 3.1.1: What's new
• 3.1.2: Feature comparison
• 3.1.3: Modes of operation
• 3.1.3.1: Operator mode
• 3.1.3.2: The ApplicationManifest Custom Resource
• 3.2: Getting started with the Free Tier
• 3.2.1: Accessing the Streaming Data Manager binaries
• 3.3: Installation
• 3.3.1: Recommendations demo application
• 3.3.2: Install SDM - GitOps
• 3.4: Dashboard
• 3.4.1: Accessing the UI
• 3.4.2: Brokers dashboard
• 3.4.3: Consumers dashboard
• 3.4.4: Topics dashboard
• 3.4.4.1: Create topic
• 3.4.4.2: Edit topic
• 3.4.4.3: Delete topic
• 3.4.5: ACL dashboard
• 3.4.5.1: KafkaACL dashboard
• 3.4.5.2: Role dashboard
• 3.4.5.3: Resource Selector dashboard
• 3.4.5.4: Classic ACL dashboard
• 3.4.5.5: Kafka users
• 3.4.5.5.1: Create user
• 3.4.5.5.2: Delete user
• 3.5: Disaster recovery
• 3.5.1: Disaster recovery on Kubernetes using MirrorMaker2
• 3.6: Managing custom Kafka distribution with Streaming Data Manager
• 3.6.1: Updating the Kafka distribution with Streaming Data Manager
• 3.7: Apache Kafka over Istio
• 3.7.1: Use a custom Certificate Authority with Istio
• 3.7.2: Setup Istio to use CSR operator as external CA
• 3.8: Kafka ACLs over Istio overview
• 3.8.1: Client applications in the same mesh
• 3.8.2: Client applications outside the Istio mesh
• 3.8.3: Client applications outside the Kubernetes cluster
• 3.8.4: Enable OAuth Bearer Token Authentication for listeners
• 3.8.4.1: OAuth Bearer Token Authentication for listeners (Apache Kafka 3.1.0 and above)
• 3.8.4.2: OAuth Bearer Token Authentication for listeners (Apache Kafka pre 3.1.0)
• 3.8.5: Declarative Kafka ACLs
• 3.9: Koperator
• 3.9.1: Install the operator
• 3.9.2: Upgrade the operator
• 3.9.3: Test provisioned Kafka Cluster
• 3.9.4: CruiseControlOperation to manage Cruise Control
• 3.9.5: Features
• 3.9.6: Provisioning Kafka Topics
• 3.9.7: Securing Kafka With SSL
• 3.9.8: Scenarios
• 3.9.9: Troubleshooting the operator
• 3.9.9.1: Common errors
• 3.9.10: Monitoring Apache Kafka on Kubernetes
• 3.9.11: Expose the Kafka cluster to external applications
• 3.9.12: Configure rack awareness
• 3.9.13: Supported versions and compatibility matrix
• 3.9.14: Support
• 3.9.15: Benchmarking Kafka
• 3.9.16: Delete the operator
• 3.9.17: Tips and tricks for the Koperator
• 3.9.18: CRD
• 3.9.18.1: KafkaCluster CRD schema reference (group kafka.banzaicloud.io)
• 3.9.18.2: KafkaTopic CRD schema reference (group kafka.banzaicloud.io)
• 3.9.18.3: KafkaUser CRD schema reference (group kafka.banzaicloud.io)
• 3.9.19: Developer Guide
• 3.9.20: License of Koperator
• 3.10: Manage Schema Registry instances
• 3.11: Using ksqlDB
• 3.11.1: Running ksqlDB in headless mode
• 3.12: Monitoring and alerting
• 3.12.1: Storage settings
• 3.12.2: Alerting
• 3.12.3: Changing Prometheus resource limits
• 3.13: Security
• 3.13.1: Open Port Inventory
• 3.13.2: Recovery procedure
• 3.14: Using Kafka Connect with Streaming Data Manager
• 3.14.1: Manage Kafka Connect clusters
• 3.14.2: Manage Kafka Connectors
• 3.14.3: Kafka Connect and Connector examples
• 3.15: Synchronizing SDM resources across multiple Kubernetes Clusters
• 3.15.1: How to synchronize SDM resources between Kubernetes clusters
• 3.15.2: Cluster Registry Custom Resources
• 3.15.3: Operator Mode
• 3.15.4: How to manage multiple Apache Kafka clusters using the dashboard
• 3.16: Tips and tricks
• 3.16.1: How to switch to declarative mode
• 3.16.2: Add more disk to the brokers
• 3.16.3: How to run SDM with upstream Istio in the same Kubernetes cluster
• 3.16.4: Create SuperUser with JKS key
• 3.17: FAQ
• 3.18: Reference
• 3.18.1: CRD
• 3.18.1.1: ApplicationManifest CRD schema reference (group supertubes.banzaicloud.io)
• 3.18.1.2: KafkaACL CRD schema reference (group kafka.banzaicloud.io)
• 3.18.1.3: KafkaBackup CRD schema reference (group kafka.banzaicloud.io)
• 3.18.1.4: KafkaConnect CRD schema reference (group kafka.banzaicloud.io)
• 3.18.1.5: KafkaConnector CRD schema reference (group kafka.banzaicloud.io)
• 3.18.1.6: KafkaResourceSelector CRD schema reference (group kafka.banzaicloud.io)
• 3.18.1.7: KafkaRole CRD schema reference (group kafka.banzaicloud.io)
• 3.18.1.8: KsqlDB CRD schema reference (group kafka.banzaicloud.io)
• 3.18.1.9: MirrorMaker2 CRD schema reference (group kafka.banzaicloud.io)
• 3.18.1.10: SchemaRegistry CRD schema reference (group kafka.banzaicloud.io)
• 3.18.2: The command line interface
• 3.18.2.1: Installing the CLI
• 4: Support
• 4.1: Known Issues
• 4.2: Common CLI issues

1 - What's new

Release 1.11.0 (2022-11-07)

Streaming Data Manager

Calisti now has a new component called Streaming Data Manager. Streaming Data Manager is a cloud-native, turnkey solution for deploying and managing Apache Kafka over Istio, providing:

• Security and encryption
• out-of-the-box observability
• RBAC integration
• Scaling

For details, see Overview.

Note: When using Streaming Data Manager on Amazon EKS, you must install the EBS CSI driver add-on on your cluster.

GitOps support

Service Mesh Manager and Streaming Data Manager can be used in GitOps environments as well. For details, see Install SMM - GitOps - single cluster, Install SMM - GitOps - multi-cluster, and Install SDM - GitOps.

Istio 1.15 support

Service Mesh Manager now supports Istio 1.15 and provides our Istio distribution based on that codebase.

This also means that Service Mesh Manager is fully compatible with Kubernetes v1.24.x.

Other changes

• The health views of the Services and Workloads pages now have fixed URLs to make sharing easier.

• If the name of your cluster doesn’t comply with the RFC 1123 DNS labels/subdomain restrictions, Service Mesh Manager now automatically converts it to a compliant format and sets it as the name of the cluster. In earlier versions, you had to manually set a compliant name for clusters with non-compliant names, otherwise certain operations (like smm install and smm attach) failed. Service Mesh Manager now automatically applies the following conversions if needed:

  • Replace ‘_’ characters with ‘-’
  • Replace ‘.’ characters with ‘-’
  • Replace ‘:’ characters with ‘-’
  • Truncate the name to 63 characters

• The Service Mesh Manager CLI now returns an error message when trying to run a command on a cluster that’s running an unsupported Kubernetes version.

• In Kubernetes 1.24 or newer, token secrets for service accounts aren’t created automatically. If Service Mesh Manager is running on a Kubernetes 1.24 (or newer) cluster, then when adding virtual machines to the mesh, you must create the token secrets manually. For details, see Add a virtual machine to the mesh.

Release 1.10.0 (2022-08-09)

RedHat-based virtual machines

Service Mesh Manager now supports attaching virtual machines running RedHat Enterprise Linux 8 to the mesh. For details, see Integrating Virtual Machines into the mesh.

Istio 1.13 support

Service Mesh Manager now supports Istio 1.13 and provides our Istio distribution based on that codebase.

Enterprise licenses

Paid-tier and enterprise licenses are now available for Service Mesh Manager.

• If you are interested in purchasing a license, contact us.
• If you have already purchased a license, apply it to your Service Mesh Manager deployments. For details, see Licensing options.

Other changes

• The smm CLI tool now supports MacOS running on M1 chips.
• The Prometheus node exporter service now uses port 19101 instead of 19100. That way, the Prometheus deployment of Service Mesh Manager can work side-by-side with a pre-existing Prometheus deployment. For details on other ports used by Service Mesh Manager, see Open Port Inventory.

Release 1.9.1 (2022-04-11)

Service Mesh Manager now supports attaching virtual machines to the mesh. After a virtual machine has been integrated into the mesh, Service Mesh Manager automatically updates the configuration of the virtual machine to ensure that it remains a part of the mesh and receives every configuration updates it needs to operate in teh mesh. In addition, the observability features available for Kubernetes pods are available for the virtual machines as well, for example:

• Virtual machine workloads and their health information are shown on the MENU > TOPOLOGY and MENU > WORKLOADS dashboard pages.
• On both pages, you can drill down to get detailed information and monitoring data about the virtual machine.
• Troubleshooting features like tracing and traffic tapping work for virtual machines as well.

For details, see Integrating Virtual Machines into the mesh.

Release 1.9.0 (2022-03-08)

Free tier

From now on, after a free registration, you can use Service Mesh Manager to manage your mesh of up to ten nodes. For details, see Licensing options and Getting started with the Free Tier.

Istio 1.12 support

Service Mesh Manager now supports Istio 1.12 and provides our Istio distribution based on that codebase.

Other changes

This release includes the following fixes:

• All custom resources used by Service Mesh Manager had been moved to the smm.cisco.com group. CLI is capable of migrating the objects to the new group.
• Topology:
  • Mesh gateways are now fully visible on the topology page even in timeline mode
  • Topology view now shows pod counts in timeline mode
• Fix an issue causing new SLOs to not to start calculating on creation
• IstioControlPlane settings can be overridden from Service Mesh Manager’s ControlPlane resource using the .spec.meshManager.istio.istioCRDOverrides key (which contains a YAML string).

Removed features

The following commands have been removed from the Service Mesh Manager command-line tool. You can configure the related features from the dashboard.

• smm sidecar-proxy egress get
• smm sidecar-proxy egress set
• smm sidecar-proxy egress delete
• smm routing
• smm mtls
• Integrated support for canary deployments. You can use the Flagger operator instead.

Release 1.8.2 (2021-12-14)

This release includes the following fixes:

Active-active fixes

• Fix secret cleanup for Istio in active-active setups.
• Update istio-operator to latest.
• Multiple active Istio control-plane support.
• Cluster name is now visible in istio status command.
• Control plane list now shows clusters as well.

Mesh view

• Stabilize the ordering of Istio clusters to prevent changed ordering on the UI.

cert-manager

• Update to v1 API.

Auth

• Fix an issue where 1.7 specific authentication tokens were generated during upgrade scenarios.

UI

• Fix an issue which caused topology to crash for ingress gateways.

Operators

• Add RBAC for Coordination resources so that operator leader election can use the resources.
• In case there is a merge conflict during reconciliation the smm operator will retry the reconciliation without failing.
• 1.7 Istio operators will be properly removed during uninstall.

Let’s Encrypt

• Validate DNS records on let’s encrypt enabled ingresses to ensure that the ingress and DNS records are matching.

Registry access

• Sort secret names to prevent changes always happening during reconciliation.

Release 1.8 (2021-10-26)

The primary goal of this release was to have a modern way to orchestrate Istio and the multi-cluster topologies Service Mesh Manager supports. As part of this work, the Cisco Istio Operator has been restructured from the ground up so that you can benefit from an API that has been adjusted to the modern Istio versions.

As this new version of the operator supports not just the Primary-Remote cluster topology, but also Multi-Primary both on the same and different network, this change paves the way for subsequent releases to add support into Service Mesh Manager for meshes with any number of Primary and Remote clusters.

Istio 1.11 support

Service Mesh Manager now supports Istio 1.11 and provides our Istio distribution based on that codebase.

This also means that Service Mesh Manager is fully compatible with Kubernetes v1.22.x.

OIDC and external dashboard access support

This release provides support for exposing the Service Mesh Manager dashboard via a public, https URL. For the required configuration please check out the Exposing the Dashboard page.

To entirely remove the need for downloading the Service Mesh Manager CLI and to better integrate with existing OIDC-enabled Kubernetes deployments, we are also supporting OIDC Authentication.

Release 1.7 (2021-07-28)

Release 1.7 is focusing on compliance, integrations, tech-debt and reusability.

GraphQL federation

The Service Mesh Manager GraphQL API is now broken down into separate components to increase reusability, and to provide the ability to switch components on/off in Service Mesh Manager in the future.

Protocol-specific observability

Istio provides several useful metrics for the TCP, HTTP, and GRPC protocols. To give you better observability and more insight into the traffic of your services, Service Mesh Manager displays protocol-specific metrics normally not available in Istio for MySQL and PostgreSQL traffic. Support for more protocols is planned in future releases.

Istio 1.10 support

Service Mesh Manager now supports Istio 1.10.

Cluster registry

A generic, distributed Kubernetes cluster registry is now serving as the base for keeping multi-cluster metadata. Cluster metadata is replicated across clusters using a gossip-like protocol.

Unified Istio distribution with SecureCN

SecureCN and Service Mesh Manager are now using the same Istio distribution that enables better integration between the two products.

CSDL Compliance

Service Mesh Manager has now reached CSDL “Planned” status.

DevNet Sandbox

Service Mesh Manager is now available on DevNet sandbox for design partners for solution testing.

Release 1.6.1 (2021-05-06)

This release is a security and bugfix release.

Included changes are:

• Add support for Istio 1.8.5 for customers still using the old version of Istio instead of 1.9
• Fix an issue in the Istio operator that required permissions for the authentication.istio.io and config.istio.io groups, while those are only needed for Istio versions < 1.8
• smm activate command now resets all of the user’s registry settings, making changing IAM credentials easier. Previously the end user-needed to remove the registry access credentials manually using the smm registry remove command

Release 1.6 (2021-04-09)

Group your clusters into networks to optimize your mesh topology using a mix of gateway-based and flat-network connections between your clusters, decreasing cross-cluster latencies and transfer costs. Clusters belonging to the same network can access each other directly, without using the cluster gateway. For details, see Cluster network and Attach a new cluster to the mesh.

UI improvements

• The Mesh Overview page of the UI shows information about your service mesh and the control planes.

  

• The new Dashboard Overview page now shows health and SLO information as well.

  

• The Topology View now shows the health status of the services.

  

• You can now filter on health and alert status on the services and workloads lists.

  

Istio 1.9 support

Service Mesh Manager now supports Istio 1.9.

2 - Service Mesh Manager

Service Mesh Manager is a multi and hybrid-cloud enabled service mesh platform for constructing modern applications. Built on Kubernetes and our Istio distribution, Service Mesh Manager enables flexibility, portability and consistency across on-premise datacenters and cloud environments.

Service Mesh Manager helps you to confidently scale your microservices over single- and multi-cluster environments and to make daily operational routines standardized and more efficient. The componentization and scaling of modern applications inevitably leads to a number of optimization and management issues:

• How do you spot bottlenecks? Are all components functioning correctly?
• How are connections between components secured?
• How does one reliably upgrade service components?

Service Mesh Manager helps you accomplish these tasks and many others in a simple and scalable way, by leveraging the Istio service mesh and building many automations around it. Our tag-line for the product captures this succinctly:

Service Mesh Manager operationalizes the service mesh to bring deep observability, convenient management, and policy-based security to modern container-based applications.

Learn more about Service Mesh Manager.

2.1 - Overview

Service Mesh Manager is a multi and hybrid-cloud enabled service mesh platform for constructing modern applications. Built on Kubernetes and our Istio distribution, Service Mesh Manager enables flexibility, portability and consistency across on-premise datacenters and cloud environments.

Service Mesh Manager helps you to confidently scale your microservices over single- and multi-cluster environments and to make daily operational routines standardized and more efficient. The componentization and scaling of modern applications inevitably leads to a number of optimization and management issues:

• How do you spot bottlenecks? Are all components functioning correctly?
• How are connections between components secured?
• How does one reliably upgrade service components?

Service Mesh Manager helps you accomplish these tasks and many others in a simple and scalable way, by leveraging the Istio service mesh and building many automations around it. Our tag-line for the product captures this succinctly:

Service Mesh Manager operationalizes the service mesh to bring deep observability, convenient management, and policy-based security to modern container-based applications.

Key features

Service Mesh Manager takes the pain out of Istio by offering great UX from installation and mesh management to runtime diagnostics and more.

Istio distribution

Service Mesh Manager is built on Istio, but offers enhanced functionality, for example, operator-based Istio management, a full-featured CLI tool, and an intuitive and easy to use UI. It is not a new abstraction layer on top of Istio, and stays fully compatible with the upstream. Service Mesh Manager is designed for enterprise users and comes with commercial support.

For a detailed list of changes compared to upstream Istio, see Istio distribution.

Observability

The Service Mesh Manager UI gives you insight into the operation of your services. It not only shows the service topology with real-time and historical metrics, but also allows you to drill-down and analyze the metrics in context. Service Mesh Manager automatically calculates the health of your services and workloads based on the available metrics. If you still need additional details, you can access the related Grafana dashboards with a single click.

You can also monitor communications with services that are external to your mesh.

Root cause diagnostics

Root cause diagnostics help you efficiently isolate and solve operational issues related to your services. Service Mesh Manager offers:

• Real-time tracing of inter-service traffic through the Tap view
• Service mesh configuration and cluster state validation
• Drill-down view to analyze metrics in context
• One-click access to Jaeger distributed traces)

Control

You can manage Istio through the Service Mesh Manager UI and the CLI. Service Mesh Manager gives you easy access to the configuration of the Istio service mesh and its underlying traffic-management features, including:

• Routing
• Circuit breaking
• Fault injection
• Ingress gateway management
• Restricting Outbound Traffic

With Service Mesh Manager, you can manage service-updates using automated, industry-standard upgrade strategies, like canary releases.

Multi-cluster

With Service Mesh Manager, you can monitor and manage your hybrid multi-cloud service infrastructure from a single pane of glass. You can easily attach and detach clusters using the CLI, and take advantage of enhanced multi-cluster telemetry.

Service Mesh Manager supports multiple mesh topologies, so you can use the one that best fits for your use-cases. In multi-cluster configurations it provides automatic locality load-balancing.

Service Level Objectives and burn-rate alerts

Service Mesh Manager helps SREs and operation engineers to observe and control the health of their services and applications. You can create and track service level objectives and corresponding alerting rules on the Service Mesh Manager dashboard.

Security & Compliance

Service Mesh Manager helps you secure your services through industry-standard authorization and authentication practices, including:

• Automatically-secured communication channels between service components: advanced mTLS configuration per service, per namespace, or cluster-wide
• Seamless Kubernetes-native RBAC authentication
• Available in FIPS-140 compliant edition as well

High-level architecture

Service Mesh Manager consists of the following components:

• Service mesh management: The open source Cisco Istio operator helps to install/upgrade/manage Istio deployments. Its unique features include managing multiple ingress/egress gateways in a declarative fashion, and automated and fine-tuned multi-cluster management.

• The core components of Service Mesh Manager are:

  • the Service Mesh Manager backend (exposing a GraphQL API)
  • the Service Mesh Manager UI, a web interface
  • the Service Mesh Manager CLI
  • the Service Mesh Manager operator

  Service Mesh Manager’s soul is its backend, which exposes a GraphQL API. The Service Mesh Manager UI (dashboard) and CLI interact with this API. The Service Mesh Manager operator is an optional component which helps with a declarative installation method to support GitOps workflows.

• External out-of-the-box integrations:

  • Prometheus
  • Grafana
  • Jaeger
  • Cert manager

  These components are automatically installed and configured by Service Mesh Manager by default to be able to work with Istio. You can also integrate Service Mesh Manager with your own Prometheus, Grafana, Jaeger, or Cert manager - Service Mesh Manager follow the batteries included but replaceable paradigm.

Istio-operator and Service Mesh Manager

The Calisti team actively maintains its fully upstream-compatible Istio distribution and several open-source projects and integrations that serve as the basis for Cisco Service Mesh Manager. From the perspective of Istio management, the Calisti team maintains the following:

• The Istio operator is an open source project, which is the core component involved in Istio control plane and gateway lifecycle management for Cisco Service Mesh Manager.
• Cisco Service Mesh Manager is a commercial product that includes all the features mentioned in this guide, enterprise support, and optionally integration support for customer environments.

Read the detailed comparison.

Next steps

• Jump to the Installation guide or get started with the Free Tier
• Read the FAQ
• View Service Mesh Manager related blogposts
• Schedule a demo

2.1.1 - Features

Service Mesh Manager addresses the whole cloud-native lifecycle of a service mesh-based solution by providing various tools starting from day 0 to day 2 operations. As such a solution requires quite many components to provide the core service mesh functionality, tracing, metrics or safe canary-based deployments (just to name a few) we are dividing Service Mesh Manager into the following layers:

Now let’s see how these layers add up to a complete solution over the whole lifecycle of the product:

Day 0

Day 0, in software development, represents the design phase, during which project requirements are specified and the architecture of the solution is decided. Service Meshes, even if they are offloading the burden of security and traffic routing from the microservices' native side, are complex in nature.

Service Mesh Manager is designed with Day 0 experimentation in mind: we are providing a CLI tool that allows to install Service Mesh Manager without prior experience: you can have an Istio-based service mesh up and running in 15 minutes - with monitoring and tracing included. The user interface allows for rapid experimentation with Istio features via an intuitive dashboard, so during the design phase Engineers can focus on what matters the most: finding the right architecture.

Day 1

Day 1 involves developing and deploying software that was designed in the Day 0 phase. In this phase you create not only the application itself, but also its infrastructure, network, and external services, then implement the initial configuration of it all.

After the initial experimentation, Service Mesh Manager aids this process by not just providing facilities for configuring the service mesh, but also by providing [validations]/docs/mesh-management/validation/ to check for any issues in the deployed settings, and integrated metrics and outlier-detection information to pinpoint any issues with the freshly changed services.

In case of interoperability issues, the traffic tap and automated tracing feature provides more detailed insight into the real-time traffic.

Day 2

Day 2 is the time when the product is shipped or made available to the customer. Here, most of the effort is focused on maintaining, monitoring, and optimizing the system. Analyzing the behavior of the system and reacting correctly are of crucial importance, as the resulting feedback loop is applied until the end of the application’s life.

Service meshes and Istio in particular are developing fast. This is reflected in the N-1 support model it uses: a new Istio version is released every 3 months, and only the last two are supported. Service Mesh Manager helps decrease the risk of these upgrades by providing canary-like control plane upgrades: SREs can gradually upgrade their services to the new version even on a Workload level, and in case an issue happens, the old version of Istio is always available in the cluster to fall back to.

Service Mesh Manager provides a Service Level Objective feature that allows to ensure the solution works within its expected operational parameters. In case of an issue, the automated outlier detection system detects failures and shows them on the topology view. We are aiding postmortems using our timeline feature, that allows for checking out the past state of the deployment, including health data.

2.1.1.1 - Istio distribution

Service Mesh Manager is built on Istio, but offers enhanced functionality, for example, operator-based Istio management, a full-featured CLI tool, and an intuitive and easy to use UI. It is not a new abstraction layer on top of Istio, and stays fully compatible with the upstream.

Service Mesh Manager is designed for enterprise users and comes with commercial support.

Notable changes compared to upstream Istio

FIPS 140-2 Level 1 compliant build

The FIPS build uses Google’s BoringCrypto for the go-based components and Envoy. All components are recompiled with the necessary configuration to provide Level 1 compliance. Also the allowed ciphers are restricted even more than FIPS would allow. For details, see FIPS-compliant service mesh.

Multiple control plane support

The upstream Istio does not have the proper support for having properly isolated multiple control planes within one cluster. Various changes (ENV name overrides, ConfigMap name overrides, and so on) were made to support proper isolation between control planes.

Protocol specific observability

Istio uses Envoy proxy under the hood, which has support for various data protocols and provides protocol-specific metrics for them. The upstream Istio can enable those metrics if a supported protocol is detected. That list has been extended with PostgreSQL, and other protocols are coming soon.

Direct connect through gateways

Direct connect means that a workload can be exposed through an Istio ingress gateway in a way that the internal mTLS is not terminated, but rather the workload proxy port is directly accessible through the gateway. This allows communication to a workload with mTLS from an external client. This feature is mainly used in the Streaming Data Manager (formerly called Supertubes) product.

DNS capture and report

With this feature the Istio proxy is able to capture DNS requests and responses and report them to an API endpoint. This feature is used in the SecureCN product.

TLS interception

The mesh Certificate Authority (CA) can issue TLS certificates for arbitrary domain names, to be able to look into TLS encrypted traffic. This feature is used in the SecureCN product.

Store arbitrary key/value information in certificates

The certificates issued by Istio CA can store arbitrary, workload-specific key-value attributes in the certificates' subject directory attribute property. This is used in Panoptica, the Cisco Secure Application Cloud to propagate workload-specific information between clusters, without the need for a central database.

Standalone sidecar injector component

The standalone sidecar injector is used in multi-cluster topologies on the peer clusters to have the sidecar-injection functionality of Istiod with much smaller resource requirements.

2.1.1.2 - Mesh lifecycle management

Operating a Service Mesh at scale is hard due to the inherent complexity of Mesh configurations. To ensure the most optimal operations of Service Mesh Manager based solutions, we provide [validations]/docs/mesh-management/validation/ to highlight any common issues with the existing configuration.

When it comes to supporting a service mesh-based solution on the long run, Service Mesh Manager provides support to dynamically extend the existing cluster with zero downtime to additional clusters.

Given that Istio is a fast-moving project (a new release is available every 3 months), Service Mesh Manager needs to bridge the gap between a rapidly changing Cloud Native solution and the requirements of an enterprise deployment. To decrease the blast radius of these upgrades we have introduced canary control plane upgrades based on Cisco’s open source Istio operator.

2.1.1.3 - Observability toolbox

Service Mesh Manager includes integrated observability by default. This allows you to take the most out of your service mesh deployments, by not just using the security and traffic management features of Istio, but also by having access to all the monitoring and tracing features Istio is capable of.

Integrated monitoring

Service Mesh Manager includes Prometheus to ensure faster troubleshooting and recovery. For further information on our monitoring capabilities, see the Dashboard guide.

Integrated tracing

Distributed tracing is the process of tracking individual requests throughout their whole call stack in the system.

With distributed tracing in place, you can visualize full call stacks, to see:

• which service called which service,
• how long each call took, and
• the network latencies between them.

That way you can tell where a request failed, or which service took too much time to respond.

To access traces and see real-time traffic flowing thru the cluster, see the traffic tap feature.

Automated outlier detection system

Complex systems might be hard to understand. Service Mesh Manager provides an automated (zero configuration required) outlier detection system available on the topology, workloads, and services pages of the Service Mesh Manager UI.

Service Level Objectives

To ensure, and most importantly to measure the deployed workloads availability, you can use an integrated Service Level Objective based alerting system.

For defining SLOs, see Tracking Service Level Objectives (SLOs).

For alerting settings, see SLO-based alerting in Production.

2.1.1.4 - Multi-cluster topologies

Multi-cluster topologies overview

Service Mesh Manager is able to construct an Istio service mesh that spans multiple clusters. In this scenario you combine multiple clusters into a single service mesh that you can manage from either a single or from multiple Istio control planes.

Single mesh scenarios are best suited to use cases where clusters are configured together, sharing resources, and are generally treated as one infrastructural component within an organization.

Service Mesh Manager not only automates setting up multi-cluster topologies, but also:

• Updates resources to keep everything running in case a cluster changes (for example, the IP address of a load balancer changes, and so on)
• Keeps the Istio CRs in sync between the clusters
• Creates federated trust between the clusters (this is a difference compared to Istio)
• Provides observability, tracing, traffic tapping and other features over multiple clusters

Supported multi-cluster topologies

Istio supports a variety of mesh topologies, as detailed in the official documentation.

Service Mesh Manager implements the Primary-Remote model, either using the different or the same network model. Service Mesh Manager also supports the Primary-Primary model, either using the different or the same network model.

Moreover, the topologies can be combined and one can have multiple primaries and multiple remotes using Service Mesh Manager.

Creating a multi-cluster mesh

Read the multi-cluster installation guide for details on how to set up a multi-cluster mesh.

2.1.2 - Modes of operation

To support the different use-cases from Day 0 to Day 2 operations, Service Mesh Manager has different modes of operation. The same binary can act as:

• a CLI tool for imperative use,
• a local reconciler, and
• a Kubernetes operator.

You can also use the operator in GitOps scenarios.

Imperative mode

The main purpose of the imperative mode is to install Service Mesh Manager, get you started, and help you experiment with the various components. You can access only a small subset of the available configuration options and features (mostly just the default settings and some of the most important configuration flags) to avoid getting overloaded with command line flags.

Most notably, you can install and delete Service Mesh Manager from the command line. Internally, the install and delete commands change the component-specific parts of the main configuration, then trigger the reconciliation of the affected components.

Other commands do not necessarily change the main configuration, nor trigger reconciliation of any component. Such commands create dynamic resources which are out of scope for the reconcilers, but are convenient for getting started without having to leave the CLI.

Once you are finished experimenting with Service Mesh Manager, the recommended way forward is to start using the reconcile command, and apply all configuration through the custom resource directly. This is analogous to how you use kubectl create and then switch to using kubectl apply when you already have a full configuration and just want to apply changes incrementally. If you are an experienced Kubernetes user, you probably skip the imperative mode and start using the reconcile command from the beginning.

The drawback of the imperative mode is that there is no overall state of components, so it can’t tell what has already been installed.

Also, it it not suitable for automation. CD systems typically require Helm charts, Kustomize, or pure YAML resources to operate with. Although the imperative commands of Service Mesh Manager have a --dump-resources flag that generates YAML files instead of applying them, you would still have to run the install command locally for each component, and commit the generated resources into version control. The CD workflow would then have to specify sequential steps for each component separately, making the whole flow difficult to extend over time.

Using the imperative mode

To use Service Mesh Manager in imperative mode, install the smm-cli command-line tool, then use its commands to install Service Mesh Manager and perform other actions. For a list of available commands, see the CLI reference.

Note: You can also configure many aspects of your service mesh using the Service Mesh Manager web interface. To access the web interface run the smm dashboard command (if your KUBECONFIG file is set properly), the smm dashboard command automatically performs the login).

Install/Uninstall components

The following components can be installed/uninstalled individually. The -a flag installs/uninstalls them all. For details on installing and uninstalling the Service Mesh Manager operator, see Operator mode.

• istio: smm istio [install|uninstall]
• cert-manager: smm cert-manager [install|uninstall]
• smm (backend and UI): smm [install|uninstall]
• demo application: smm demoapp [install|uninstall]

Reconciler mode

Reconciler mode is a declarative CLI mode. The reconcile command is a one-shot version of an operator’s reconcile flow. It executes the component reconcilers in order, and can decide whether they require another reconciliation round, or are already finished. Reconciling can apply new configuration, and remove disabled components from the system.

Note: In this mode, the operator is not installed on the cluster. The controller code runs from the CLI on the client side.

A component can be anything that receives the whole configuration, understands its own part from it to configure itself, and is able to delete its managed resources when disabled or removed. Service Mesh Manager uses two different implementations:

• The native reconciler triggers a “resource builder” to create Kubernetes resources along with their desired state (present or absent) based on the configuration of the component. Such resource builders create CRDs, RBAC, and a Deployment resource to be able to run an operator.

• The other implementation is the Helm reconciler that basically acts as a Helm operator. It installs and upgrades an embedded chart if it has changed, or uninstalls it if it has been removed from the main configuration.

Compared to kubectl apply, these solutions add ordering, and allow executing custom logic if required. Also, they remove resources that are not present in the config anymore. The CLI in this case executes the control logic as well.

Compared to terraform, the dependencies are managed in a predefined execution order and have static binding using deterministic names. Lower performance, but easier to follow. Remote state is the CR saved to the API server.

Using the reconciler mode

To use Service Mesh Manager in reconciler mode, complete the following steps. In this scenario, the manifest is read from a file, allowing you to declaratively provide custom configuration for the various components.

1. Login to your Service Mesh Manager installation.

2. Prepare the configuration settings you want to apply in a YAML file, and run the following command. For details on the configuration settings, see the ControlPlane Custom Resource.

   smm reconcile --from-file <path-to-file>
   

3. The settings applied to the components are the result of merging the default settings + valuesOverride + managed settings. You cannot change the managed settings to avoid misconfiguration and possible malfunction.

Operator mode

The operator mode follows the familiar operator pattern. In operator mode, Service Mesh Manager watches events on the ControlPlane Custom Resource, and triggers a reconciliation for all components in order, the same way you can trigger the reconcile command locally.

Note: Unlike in the declarative CLI mode, in operator mode the Service Mesh Manager operator is running inside Kubernetes, and not on a client machine. This naturally means that this mode is exclusive with the install, delete, and reconcile commands.

Using the operator mode is the recommended way to integrate the Service Mesh Manager installer into a Kubernetes-native continuous delivery solution, for example, Argo, where the integration boils down to applying YAML files to get the installer deployed as an operator.

Existing configurations managed using the reconcile command work out-of-the box after switching to the operator mode.

Using the operator mode

To use Service Mesh Manager in operator mode, Install Service Mesh Manager in operator mode. In this scenario, the reconcile flow runs on the Kubernetes cluster as an operator that watches the ControlPlane custom resources. Any changes made to the watched custom resource triggers the reconcile flow.

GitOps

GitOps is a way of implementing Continuous Deployment for cloud native applications. Based on Git and Continuous Deployment tools, GitOps provides a declarative way to store the desired state of your infrastructure and automated processes to realize the desired state in your production environment.

For example, to deploy a new application you update the repository, and the automated processes perform the actual deployment steps.

When used in operator mode, Service Mesh Manager works flawlessly with GitOps solutions such as Argo CD, and can be used to declaratively manage your service mesh. For a detailed tutorial on setting up Argo CD with Service Mesh Manager, see Install SMM - GitOps - single cluster.

2.1.3 - Istio-operator feature comparison

The Calisti team actively maintains its fully upstream-compatible Istio distribution and several open-source projects and integrations that serve as the basis for Cisco Service Mesh Manager. From the perspective of Istio management, the Calisti team maintains the following:

• The Istio operator is an open source project, which is the core component involved in Istio control plane and gateway lifecycle management for Cisco Service Mesh Manager.
• Cisco Service Mesh Manager is a commercial product that includes all the features mentioned in this guide, enterprise support, and optionally integration support for customer environments.

2.2 - Getting started with the Free Tier

This Getting started guide helps you access and install the free version of Service Mesh Manager. If you are a paying customer, see Installation for installation options.

To get started with Service Mesh Manager, you will install Service Mesh Manager and a demo application on a single cluster. After that, you can attach other clusters to the mesh and redeploy the demo application to run on multiple clusters.

Free tier limitations

• The free tier of Service Mesh Manager allows you to use Service Mesh Manager on maximum of two Kubernetes clusters where the total number of worker nodes in your clusters is 10. For details, see Licensing options.
• SMM Operator helm charts is not supported.

To buy an enterprise license, contact your Cisco sales representative, or directly Cisco Emerging Technologies and Incubation.

Prerequisites

You need a Kubernetes cluster to run Service Mesh Manager. If you don’t already have a Kubernetes cluster to work with, then:

1. Create a cluster that meets the following resource requirements with your favorite provider.

   CAUTION:

   Supported providers and Kubernetes versions

   The cluster must run a Kubernetes version that Service Mesh Manager supports: Kubernetes 1.21, 1.22, 1.23, 1.24.

   Service Mesh Manager is tested and known to work on the following Kubernetes providers:

   • Amazon Elastic Kubernetes Service (Amazon EKS)
   • Google Kubernetes Engine (GKE)
   • Azure Kubernetes Service (AKS)
   • On-premises installation of stock Kubernetes with load balancer support (and optionally PVCs for persistence)

   Resource requirements

   Make sure that your Kubernetes cluster has sufficient resources. The default installation (with Service Mesh Manager and demo application) requires the following amount of resources on the cluster:

   	Only Service Mesh Manager	Service Mesh Manager and Streaming Data Manager
   CPU	- 12 vCPU in total - 4 vCPU available for allocation per worker node (If you are testing on a cluster at a cloud provider, use nodes that have at least 4 CPUs, for example, c5.xlarge on AWS.)	- 24 vCPU in total - 4 vCPU available for allocation per worker node (If you are testing on a cluster at a cloud provider, use nodes that have at least 4 CPUs, for example, c5.xlarge on AWS.)
   Memory	- 16 GB in total - 2 GB available for allocation per worker node	- 36 GB in total - 2 GB available for allocation per worker node
   Storage	12 GB of ephemeral storage on the Kubernetes worker nodes (for Traces and Metrics)	12 GB of ephemeral storage on the Kubernetes worker nodes (for Traces and Metrics)

   These minimum requirements need to be available for allocation within your cluster, in addition to the requirements of any other loads running in your cluster (for example, DaemonSets and Kubernetes node-agents). If Kubernetes cannot allocate sufficient resources to Service Mesh Manager, some pods will remain in Pending state, and Service Mesh Manager will not function properly.

   Enabling additional features, such as High Availability increases this value.

   The default installation, when enough headroom is available in the cluster, should be able to support at least 150 running Pods with the same amount of Services. For setting up Service Mesh Manager for bigger workloads, see scaling Service Mesh Manager.

2. Set Kubernetes configuration and context.

   The Service Mesh Manager command-line tool uses your current Kubernetes context, from the file named in the KUBECONFIG environment variable (~/.kube/config by default). Check if this is the cluster you plan to deploy the product by running the following command:

   kubectl config get-contexts
   

   If there are multiple contexts in the Kubeconfig file, specify the one you want to use with the use-context parameter, for example:

   kubectl config use-context <context-to-use>
   

Preparation

To access and install the free version of Service Mesh Manager, complete the following steps.

1. You’ll need a Cisco Customer account to download Service Mesh Manager. If you don’t already have one here’s how to sign up:

   1. Visit the Cisco Account registration page and complete the registration form.
   2. Look out for an email from no-reply@mail-id.cisco.com titled Activate Account and click on the Activate Account button to activate your account.

2. Download the Service Mesh Manager command-line tool.

   1. Visit the Service Mesh Manager download center.
   2. If you’re redirected to the home page, check the upper right-hand corner to see if you’re signed in. If you see a login button go ahead and login using your Cisco Customer account credentials. If, instead, you see “welcome, ” then you are already logged in.
   3. Once you have logged in, navigate to the Service Mesh Manager download center again.
   4. Read and accept the End-User License Agreement (EULA).
   5. Download the Service Mesh Manager command-line tool (CLI) suitable for your system. The CLI supports macOS and Linux (x86_64). On Windows, install the Windows Subsystem for Linux (WSL) and use the Linux binary.
   6. Extract the archive. The archive contains two binaries, smm for Service Mesh Manager, and supertubes for Streaming Data Manager.
   7. Navigate to the directory where you have extracted the CLI.

3. The Service Mesh Manager download page shows your credentials that you can use to access the Service Mesh Manager docker images.

   Open a terminal and login to the image registries of Service Mesh Manager by running:

   SMM_REGISTRY_PASSWORD=<your-password> ./smm activate \
     --host=registry.eticloud.io \
     --prefix=smm \
     --user='<your-username>'
   

   Where the <your-password> and <your-username> parts contain the access credentials to the registries.

Install Service Mesh Manager on a single cluster

1. Run the following command. This will install the main Service Mesh Manager components.

   smm install -a --cluster-name <name-of-your-cluster>
   

   Note: If you are installing Service Mesh Manager on a managed Kubernetes solution of a public cloud provider (for example, Amazon EKS, AKS, or GKE) or kOps, the cluster name auto-discovered by Service Mesh Manager is incompatible with Kubernetes resource naming restrictions and Istio’s method of identifying clusters in a multicluster mesh.

   In earlier Service Mesh Manager versions, you had to manually use the --cluster-name parameter to set a cluster name that complies with the RFC 1123 DNS subdomain/label format (alphanumeric string without “_” or “.” characters). Starting with Service Mesh Manager version 1.11, non-compliant names are automatically converted using the following rules:

   • Replace ‘_’ characters with ‘-’
   • Replace ‘.’ characters with ‘-’
   • Replace ‘:’ characters with ‘-’
   • Truncate the name to 63 characters

   Service Mesh Manager supports KUBECONFIG contexts having the following authentication methods:

   • certfile and keyfile
   • certdata and keydata
   • bearer token
   • exec/auth provider

   Username-password pairs are not supported. If you are installing Service Mesh Manager in a test environment, you can install it without requiring authentication by running:

   smm install --anonymous-auth -a --run-demo
   

   If you experience errors during the installation, try running the installation in verbose mode: smm install -v

2. Wait until the installation is completed. This can take a few minutes.

3. (Optional) If you don’t already have Istio workload and traffic on this cluster, install the demo application:

   smm demoapp install
   

4. Run the following command to open the dashboard. If you don’t already have Istio workload and traffic, the dashboard will be empty.

   smm dashboard
   

   

5. (Optional)

   If you are installing Service Mesh Manager on a managed Kubernetes solution of a public cloud provider (for example, AWS, Azure, or Google Cloud), assign admin roles so that you can tail the logs of your containers from the Service Mesh Manager UI and perform various tasks from the CLI that require custom permissions. Run the following command:

   kubectl create clusterrolebinding user-cluster-admin --clusterrole=cluster-admin --user=<gcp/aws/azure username>
   

6. At this point, Service Mesh Manager is up and running. On the dashboard select MENU > TOPOLOGY to see how the traffic flows through your mesh, and experiment with any of the available features described in the documentation.

   

7. To evaluate Streaming Data Manager, see Getting tarted with Streaming Data Manager.

Get help

If you run into errors, experience problems, or just have a question or feedback while using the Free Tier of Service Mesh Manager, visit our Application Networking and Observability community site.

Support details for the Pro and Enterprise Tiers are provided in the purchased plan.

2.3 - Installation

To evaluate the services Service Mesh Manager offers, we recommend using the free edition of Service Mesh Manager in a test environment and using our demo application. This way you can start over any time, and try all the options you are interested in without having to worry about changes made to your existing environment, even if it’s not used in production.

Production installation is very similar, but of course you won’t need to deploy the demo application, and you must exactly specify which components you want to use.

2.3.1 - Prerequisites

Before deploying Service Mesh Manager on your cluster, complete the following tasks.

Create a cluster

You need a Kubernetes cluster to run Service Mesh Manager (and optionally, Streaming Data Manager). If you don’t already have a Kubernetes cluster to work with, create one with one of the methods described in Create a test cluster.

Install the Service Mesh Manager tool

Install the Service Mesh Manager command-line tool. You can use the Service Mesh Manager CLI tool to install Service Mesh Manager and other components on your cluster.

Note: The Service Mesh Manager CLI supports macOS and Linux (x86_64). On Windows, install the Windows Subsystem for Linux (WSL) and use the Linux binary.

1. Install the Service Mesh Manager CLI for your environment.

   • If you have already received access to the Service Mesh Manager binaries, see Accessing the Service Mesh Manager binaries.
   • If you are new to Service Mesh Manager, you can also use the free edition of Service Mesh Manager for evaluation.

2. Set Kubernetes configuration and context.

   The Service Mesh Manager command-line tool uses your current Kubernetes context, from the file named in the KUBECONFIG environment variable (~/.kube/config by default). Check if this is the cluster you plan to deploy the product by running the following command:

   kubectl config get-contexts
   

   If there are multiple contexts in the Kubeconfig file, specify the one you want to use with the use-context parameter, for example:

   kubectl config use-context <context-to-use>
   

Deploy Service Mesh Manager

After you have completed the previous steps, you can install Service Mesh Manager on a single cluster, or you can form a multi-cluster mesh right away.

Note: The default version of Service Mesh Manager is built with the standard SSL libraries. To use a FIPS-compliant version of Istio, see Install FIPS images.

Select the installation method you want to use:

• Install Service Mesh Manager on a single cluster
• Install Service Mesh Manager on multiple cluster

You can install Service Mesh Manager on a single cluster first, and attach additional clusters later to form a multi-cluster mesh.

2.3.1.1 - Accessing the Service Mesh Manager binaries

To evaluate Service Mesh Manager we recommend using the free tier option.

If you don’t already have a Cisco Customer Identity (CCI) account, you’ll also have to complete a brief sign-up procedure.

To access the CLI binaries, you can either download it from the Service Mesh Manager download page or from registry.eticloud.io using ORAS.

Download the CLI

1. Visit the Service Mesh Manager download center.
2. If you’re redirected to the home page, check the upper right-hand corner to see if you’re signed in. If you see a login button go ahead and login using your Cisco Customer account credentials. If, instead, you see “welcome, ” then you are already logged in.
3. Once you have logged in, navigate to the Service Mesh Manager download center again.
4. Read and accept the End-User License Agreement (EULA).
5. Download the Service Mesh Manager command-line tool (CLI) suitable for your system. The CLI supports macOS and Linux (x86_64). On Windows, install the Windows Subsystem for Linux (WSL) and use the Linux binary.
6. Extract the archive. The archive contains two binaries, smm for Service Mesh Manager, and supertubes for Streaming Data Manager.
7. Navigate to the directory where you have extracted the CLI.

Download the CLI using ORAS

To install the Service Mesh Manager CLI using ORAS, complete the following steps.

1. Install OCI Registry As Storage (ORAS). For details, see the ORAS installation guide for your operating system. For example, on macOS you can run brew install oras

2. Log in to registry.eticloud.io using ORAS. You can find your credentials and the activation command on the Service Mesh Manager download page. (If you haven’t registered yet, sign up on the Service Mesh Manager page).

   Run the following command to log in, then enter your username and password.

   oras login registry.eticloud.io
   

3. Download the Service Mesh Manager CLI by running:

   oras pull registry.eticloud.io/smm/smm-cli:v1.11.0
   

4. To manage Apache Kafka installations using Streaming Data Manager, download the Streaming Data Manager command-line tool (called supertubes-cli) as well.

   oras pull registry.eticloud.io/sdm/supertubes-cli:v1.11.0
   

5. Extract the archive for your operating system.

6. Navigate to the directory where you have extracted the CLI.

Activate the CLI

Due to legal requirements the docker images for Service Mesh Manager are stored in a docker registry requiring authentication. Service Mesh Manager has built-in support for transparently performing this authentication. For this feature to work you must “activate” the CLI on every workstation that will be used to install, upgrade, or change the Service Mesh Manager deployment. For using the dashboard or any other CLI command this activation step can be skipped.

You can find your credentials and the activation command on the Service Mesh Manager download page.

Open a terminal and login to the image registries of Service Mesh Manager by running:

SMM_REGISTRY_PASSWORD=<your-password> ./smm activate \
  --host=registry.eticloud.io \
  --prefix=smm \
  --user='<your-username>'

Where the <your-password> and <your-username> parts contain the access credentials to the registries.

After the activation, you can install Service Mesh Manager on a single cluster or multiple clusters, or manage an existing installation.

Upgrading an already activated 1.8.x SMM

In case your local 1.8.x CLI is already activated using the ECR repositories (the old activate command is still available as activate-ecr command), feel free to continue using the existing ECR repositories, they will remain supported.

If you’d like to start using the new repositories, execute the activate command as shown above. That updates the local environment to rely on the new repositories. When you use the install or operator reconcile command for the next time, the Kubernetes cluster will be automatically updated to use the new access credentials.

2.3.1.2 - Create a test cluster

You need a Kubernetes cluster to run Service Mesh Manager. If you don’t already have a Kubernetes cluster to work with, create one with one of the following methods.

• Run locally (~5 minutes): Deploy Service Mesh Manager to a single-node Kubernetes cluster running on your development machine.
• Run on a Kubernetes cluster (~10 minutes): Deploy Service Mesh Manager to a Kubernetes cluster of your choice.

Run Service Mesh Manager locally

Recommended if you don’t have or don’t want to create a Kubernetes cluster, but want to try out Service Mesh Manager quickly.

1. Install one of the following tools to run a Kubernetes cluster locally:

   • minikube (recommended for Linux)
   • Docker for Desktop (recommended for Mac/Windows) provides Kubernetes support as noted here
   • kind

2. Ensure that the local Kubernetes cluster meets the following requirements:

1. Launch the local Kubernetes cluster with one of the following tools:

   • minikube:

     minikube start
     

   • Docker for Desktop: In preferences, choose Enable Kubernetes.

   • kind:

     kind create cluster
     

2. Proceed to Install Service Mesh Manager.

3. When you’re done experimenting, you can remove the demo application, Service Mesh Manager, and Istio from your cluster with the following command, which removes all of these components in the correct order:

   smm uninstall -a
   

   Note: Uninstalling Service Mesh Manager does not remove the Custom Resource Definitions (CRDs) from the cluster, because removing a CRD removes all related resources. Since Service Mesh Manager uses several external components, this could remove things not belonging to Service Mesh Manager.

Run on a Kubernetes cluster

Recommended if you have a Kubernetes cluster and want to try out Service Mesh Manager quickly.

1. Create a cluster that meets the following resource requirements with your favorite provider.

   CAUTION:

   Supported providers and Kubernetes versions

   The cluster must run a Kubernetes version that Service Mesh Manager supports: Kubernetes 1.21, 1.22, 1.23, 1.24.

   Service Mesh Manager is tested and known to work on the following Kubernetes providers:

   • Amazon Elastic Kubernetes Service (Amazon EKS)
   • Google Kubernetes Engine (GKE)
   • Azure Kubernetes Service (AKS)
   • On-premises installation of stock Kubernetes with load balancer support (and optionally PVCs for persistence)

   Resource requirements

   Make sure that your Kubernetes cluster has sufficient resources. The default installation (with Service Mesh Manager and demo application) requires the following amount of resources on the cluster:

   	Only Service Mesh Manager	Service Mesh Manager and Streaming Data Manager
   CPU	- 12 vCPU in total - 4 vCPU available for allocation per worker node (If you are testing on a cluster at a cloud provider, use nodes that have at least 4 CPUs, for example, c5.xlarge on AWS.)	- 24 vCPU in total - 4 vCPU available for allocation per worker node (If you are testing on a cluster at a cloud provider, use nodes that have at least 4 CPUs, for example, c5.xlarge on AWS.)
   Memory	- 16 GB in total - 2 GB available for allocation per worker node	- 36 GB in total - 2 GB available for allocation per worker node
   Storage	12 GB of ephemeral storage on the Kubernetes worker nodes (for Traces and Metrics)	12 GB of ephemeral storage on the Kubernetes worker nodes (for Traces and Metrics)

   These minimum requirements need to be available for allocation within your cluster, in addition to the requirements of any other loads running in your cluster (for example, DaemonSets and Kubernetes node-agents). If Kubernetes cannot allocate sufficient resources to Service Mesh Manager, some pods will remain in Pending state, and Service Mesh Manager will not function properly.

   Enabling additional features, such as High Availability increases this value.

   The default installation, when enough headroom is available in the cluster, should be able to support at least 150 running Pods with the same amount of Services. For setting up Service Mesh Manager for bigger workloads, see scaling Service Mesh Manager.

2. Set Kubernetes configuration and context.

   The Service Mesh Manager command-line tool uses your current Kubernetes context, from the file named in the KUBECONFIG environment variable (~/.kube/config by default). Check if this is the cluster you plan to deploy the product by running the following command:

   kubectl config get-contexts
   

   If there are multiple contexts in the Kubeconfig file, specify the one you want to use with the use-context parameter, for example:

   kubectl config use-context <context-to-use>
   

3. Proceed to Install Service Mesh Manager.

2.3.2 - Licensing options

Service Mesh Manager is available in two different editions:

• To register for free-tier access, see Getting started with the Free Tier.
• To buy a Pro (paid-tier) license, visit the Service Mesh Manager website.
• For an enterprise license, contact your Cisco sales representative, or directly the Service Mesh Manager sales team.

Free tier

The free tier allows using 10 nodes in maximum 2 clusters inside the same mesh (active/active or active/passive setups are allowed).

For example, you can use a single cluster with 10 worker nodes (counting the active nodes too), or you can have an active cluster with 6 worker nodes and a passive cluster attached with 4 additional nodes.

If you exceed these limits, the management functionalities of Service Mesh Manager become restricted until the number of nodes and clusters are back to the allowed values:

• CLI commands for installing Service Mesh Manager and attaching clusters to the mesh will fail until the node count has been decreased.

  Note: The uninstall command works regardless of the node count.

• The dashboard shows an error detailing the license violations when over limits. As Kubernetes is dynamic in nature, you can exceed the node limit by 1 for one day, so you can keep using the Service Mesh Manager dashboard during node rotations.

The Istio data plane is available regardless the number of nodes, ensuring that no production outage happens due to license violation.

To register for free-tier access, see Getting started with the Free Tier.

Paid tier

To buy a Pro (paid-tier) license, visit the Service Mesh Manager website. If you have purchased a Pro license for Service Mesh Manager, you have to apply the license to your Service Mesh Manager installations. To achieve that, complete the following steps.

1. Copy your license token into a file (for example, license.key).

2. Apply the license to your Service Mesh Manager installation. If you have a multi-cluster setup, apply it to the primary cluster (where the Service Mesh Manager control plane is running). Run the following command:

   smm license apply --licenseKeyPath <license.key>
   

   Note: If you are using Service Mesh Manager with a commercial license in a multi-cluster scenario, Service Mesh Manager automatically synchronizes the license to the attached clusters. If the peer cluster already has a license, it is automatically deleted and replaced with the license of the primary Service Mesh Manager cluster. Detaching a peer cluster automatically deletes the license from the peer cluster.

3. Run the following command to verify that the new license has been added to the Service Mesh Manager installation.

   smm license list
   

   Alternatively, you can open the Service Mesh Manager dashboard, open the user account in the top-right, then select License to display the details of the license.

   The details of the license include:

   • the number of permitted clusters (MaxClusters) in the mesh,
   • the total number of permitted nodes in the mesh (MaxNodes), and
   • the number of maximum permitted nodes for a cluster (MaxNodesPerCluster).

Exceeding the paid-tier license limit

In case you exceed the license limit for a paid-tier license, you lose access to the Service Mesh Manager dashboard until you decrease the size of the mesh to comply with the license limits.

The Istio data plane is available regardless the number of nodes, ensuring that no production outage happens due to license violation.

Enterprise license

To buy an enterprise license, contact your Cisco sales representative, or directly the Service Mesh Manager sales team.

To apply an enterprise license to your Service Mesh Manager installations, follow the steps described for the Paid tier.

2.3.3 - Create single cluster mesh

Prerequisites

You need the Service Mesh Manager CLI tool installed on your computer and a Kubernetes cluster as described in the Prerequisites section.

Install Service Mesh Manager

For a quick demo or evaluation, complete the following steps to install Service Mesh Manager with every component, including the demo application. If you prefer a more interactive installation, see Installing Service Mesh Manager interactively.

• If you have already received access to the Service Mesh Manager binaries, see Accessing the Service Mesh Manager binaries.
• If you are new to Service Mesh Manager, you can also use the free edition of Service Mesh Manager for evaluation.

Note: If you are installing Service Mesh Manager on a managed Kubernetes solution of a public cloud provider (for example, Amazon EKS, AKS, or GKE) or kOps, the cluster name auto-discovered by Service Mesh Manager is incompatible with Kubernetes resource naming restrictions and Istio’s method of identifying clusters in a multicluster mesh.

In earlier Service Mesh Manager versions, you had to manually use the --cluster-name parameter to set a cluster name that complies with the RFC 1123 DNS subdomain/label format (alphanumeric string without “_” or “.” characters). Starting with Service Mesh Manager version 1.11, non-compliant names are automatically converted using the following rules:

• Replace ‘_’ characters with ‘-’
• Replace ‘.’ characters with ‘-’
• Replace ‘:’ characters with ‘-’
• Truncate the name to 63 characters

1. Run the following command. This will install the main Service Mesh Manager components.

   • If you are installing Service Mesh Manager on a managed Kubernetes solution of a public cloud provider (for example, Amazon EKS, AKS, or GKE) or kOps, run

     smm install -a --cluster-name <name-of-your-cluster>
     

   • Otherwise, run

     smm install -a
     

   Service Mesh Manager supports KUBECONFIG contexts having the following authentication methods:

   • certfile and keyfile
   • certdata and keydata
   • bearer token
   • exec/auth provider

   Username-password pairs are not supported. If you are installing Service Mesh Manager in a test environment, you can install it without requiring authentication by running:

   smm install --anonymous-auth -a --run-demo
   

   If you experience errors during the installation, try running the installation in verbose mode: smm install -v

   Note: If you are installing Service Mesh Manager on a local cluster (for example, using MiniKube) and you don’t have a local LoadBalancer setup, disable the meshexpansion gateway support. To do that, create a file called local_icp_cr.yaml with the following content:

   apiVersion: servicemesh.cisco.com/v1alpha1
   kind: IstioControlPlane
   metadata:
       name: mesh
       namespace: istio-system
   spec:
     meshExpansion:
       enabled: false
   

   Then, run the following command: smm install --istio-cr-file local_icp_cr.yaml

2. Wait until the installation is completed. This can take a few minutes. Run the following command to open the dashboard.

   smm dashboard
   

   

   If you don’t already have Istio workload and traffic, the dashboard will be empty. To install the demo application, run:

   smm demoapp install
   

   After installation, the demo application automatically starts generating traffic, and the dashboard draws a picture of the data flow. (If it doesn’t, run the smm demoapp load start command, or Generate load on the UI. If you want to stop generating traffic, run smm demoapp load stop.)

3. If you are installing Service Mesh Manager on a managed Kubernetes solution of a public cloud provider (for example, AWS, Azure, or Google Cloud), assign admin roles so that you can tail the logs of your containers from the Service Mesh Manager UI and perform various tasks from the CLI that require custom permissions. Run the following command:

   kubectl create clusterrolebinding user-cluster-admin --clusterrole=cluster-admin --user=<gcp/aws/azure username>
   

4. At this point, Service Mesh Manager is up and running. On the dashboard select MENU > TOPOLOGY to see how the traffic flows through your mesh, and experiment with any of the available features described in the documentation.

5. If you have purchased a commercial license for Service Mesh Manager, apply the license. For details, see Paid tier.

Install Service Mesh Manager interactively

With the interactive installation, you can:

• Install the Service Mesh Manager core, which provides a dashboard and an internal API for handling the service mesh.
• Install and execute the Istio operator.
• Install a demo application (optional).

Complete the following steps.

1. Start the installation.

   • If you are installing Service Mesh Manager on a managed Kubernetes solution of a public cloud provider (for example, AWS, Azure, or GCP), run

     smm install --cluster-name <name-of-your-cluster>
     

   • Otherwise, run

     smm install
     

   If you experience errors during the installation, try running the installation in verbose mode: smm install -v

   During installation, answer the interactive questions in the terminal.

   ? Install istio-operator (recommended). Press enter to accept Yes
   ? Install cert-manager (recommended). Press enter to accept Yes
   ? Install canary-operator (recommended). Press enter to accept Yes
   ? Install and run demo application (optional). Press enter to skip (y/N) y
   

   Note: If you don’t need the demo application, you can simply accept the defaults by pressing enter for each question as it will only install the core components. You can install additional components later.

2. Wait until the installation is completed. This can take a few minutes. If you have selected to install the demo application, the Service Mesh Manager dashboard automatically opens in your browser. Otherwise, run the following command to open the dashboard.

   smm dashboard
   

   If you don’t already have Istio workload and traffic, the dashboard will be empty. To install the demo application, run:

   smm demoapp install
   

   After installation, the demo application automatically starts generating traffic, and the dashboard draws a picture of the data flow. (If it doesn’t, run the smm demoapp load start command, or Generate load on the UI. If you want to stop generating traffic, run smm demoapp load stop.)

3. If you are installing Service Mesh Manager on a managed Kubernetes solution of a public cloud provider (for example, AWS, Azure, or Google Cloud), assign admin roles so that you can tail the logs of your containers from the Service Mesh Manager UI and perform various tasks from the CLI that require custom permissions. Run the following command:

   kubectl create clusterrolebinding user-cluster-admin --clusterrole=cluster-admin --user=<gcp/aws/azure username>
   

4. At this point, Service Mesh Manager is up and running. On the dashboard select MENU > TOPOLOGY to see how the traffic flows through your mesh, and experiment with any of the available features described in the documentation.

5. If you have purchased a commercial license for Service Mesh Manager, apply the license. For details, see Paid tier.

2.3.4 - Create multi-cluster mesh

Prerequisites

To create a multi-cluster mesh with Service Mesh Manager, you need:

• At least two Kubernetes clusters, with access to their kubeconfig files.
• The Service Mesh Manager CLI tool installed on your computer.
• Network connectivity properly configured between the participating clusters.

Create a multi-cluster mesh

To create a multi-cluster mesh with Service Mesh Manager, complete the following steps.

1. Install Service Mesh Manager to the primary cluster using the following command. This will install all Service Mesh Manager components to the cluster. Run smm install -a --cluster-name <name-of-your-cluster> or smm install -a

   Note: If you are installing Service Mesh Manager on a managed Kubernetes solution of a public cloud provider (for example, Amazon EKS, AKS, or GKE) or kOps, the cluster name auto-discovered by Service Mesh Manager is incompatible with Kubernetes resource naming restrictions and Istio’s method of identifying clusters in a multicluster mesh.

   In earlier Service Mesh Manager versions, you had to manually use the --cluster-name parameter to set a cluster name that complies with the RFC 1123 DNS subdomain/label format (alphanumeric string without “_” or “.” characters). Starting with Service Mesh Manager version 1.11, non-compliant names are automatically converted using the following rules:

   • Replace ‘_’ characters with ‘-’
   • Replace ‘.’ characters with ‘-’
   • Replace ‘:’ characters with ‘-’
   • Truncate the name to 63 characters

   If you experience errors during the installation, try running the installation in verbose mode: smm install -v

   Service Mesh Manager supports KUBECONFIG contexts having the following authentication methods:

   • certfile and keyfile
   • certdata and keydata
   • bearer token
   • exec/auth provider

   Username-password pairs are not supported. If you are installing Service Mesh Manager in a test environment, you can install it without requiring authentication by running:

   smm install --anonymous-auth -a --run-demo
   

2. On the primary Service Mesh Manager cluster, attach the peer cluster to the mesh using one of the following commands.

   Note: To understand the difference between the remote Istio and primary Istio clusters, see the Istio control plane models section in the official Istio documentation. The short summary is that remote Istio clusters do not have a separate Istio control plane, while primary Istio clusters do.

   The following commands automate the process of creating the resources necessary for the peer cluster, generate and set up the kubeconfig for that cluster, and attach the cluster to the mesh.

   • To attach a remote Istio cluster with the default options, run:

     smm istio cluster attach <PEER_CLUSTER_KUBECONFIG_FILE>
     

   • To attach a primary Istio cluster (one that has an active Istio control plane installed), run:

     smm istio cluster attach <PEER_CLUSTER_KUBECONFIG_FILE> --active-istio-control-plane
     

     Note: If the name of the cluster cannot be used as a Kubernetes resource name (for example, because it contains the underscore, colon, or another special character), you must manually specify a name to use when you are attaching the cluster to the service mesh. For example:

     smm istio cluster attach <PEER-CLUSTER-KUBECONFIG-FILE> --name <KUBERNETES-COMPLIANT-CLUSTER-NAME> --active-istio-control-plane
     

     Otherwise, the following error occurs when you try to attach the cluster:

     could not attach peer cluster: graphql: Secret "example-secret" is invalid: metadata.name: Invalid value: "gke_gcp-cluster_region": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.'**
     

3. Verify the name that will be used to refer to the cluster in the mesh. To use the name of the cluster, press Enter.

   Note: If you are installing Service Mesh Manager on a managed Kubernetes solution of a public cloud provider (for example, Amazon EKS, AKS, or GKE) or kOps, the cluster name auto-discovered by Service Mesh Manager is incompatible with Kubernetes resource naming restrictions and Istio’s method of identifying clusters in a multicluster mesh.

   In earlier Service Mesh Manager versions, you had to manually use the --cluster-name parameter to set a cluster name that complies with the RFC 1123 DNS subdomain/label format (alphanumeric string without “_” or “.” characters). Starting with Service Mesh Manager version 1.11, non-compliant names are automatically converted using the following rules:

   • Replace ‘_’ characters with ‘-’
   • Replace ‘.’ characters with ‘-’
   • Replace ‘:’ characters with ‘-’
   • Truncate the name to 63 characters

   ? Cluster must be registered. Please enter the name of the cluster (<current-name-of-the-cluster>)
   

4. Wait until the peer cluster is attached. Attaching the peer cluster takes some time, because it can be completed only after the ingress gateway address works. You can verify that the peer cluster is attached successfully with the following command:

   smm istio cluster status
   

   The process is finished when you see Available in the Status field of all clusters.

   To attach other clusters, or to customize the network settings of the cluster, see Attach a new cluster to the mesh.

5. Deploy the demo application. You can deploy the demo application in a distributed way to multiple clusters with the following commands:

   smm demoapp install -s frontpage,catalog,bookings,postgresql
   smm -c <PEER_CLUSTER_KUBECONFIG_FILE> demoapp install -s movies,payments,notifications,analytics,database,mysql --peer
   

   After installation, the demo application automatically starts generating traffic, and the dashboard draws a picture of the data flow. (If it doesn’t, run the smm demoapp load start command, or Generate load on the UI. If you want to stop generating traffic, run smm demoapp load stop.)

   If you are looking to deploy your own application, check out Deploy custom application for some guidelines.

6. If you are installing Service Mesh Manager on a managed Kubernetes solution of a public cloud provider (for example, AWS, Azure, or Google Cloud), assign admin roles so that you can tail the logs of your containers from the Service Mesh Manager UI and perform various tasks from the CLI that require custom permissions. Run the following command:

   kubectl create clusterrolebinding user-cluster-admin --clusterrole=cluster-admin --user=<gcp/aws/azure username>
   

7. Open the dashboard and look around.

   smm dashboard
   

8. If you have purchased a commercial license for Service Mesh Manager, apply the license. For details, see Paid tier.

Cleanup

1. To remove the demo application from a peer cluster, run the following command:

   smm -c <PEER_CLUSTER_KUBECONFIG_FILE> demoapp uninstall
   

2. To remove a peer cluster from the mesh, run the following command:

   smm istio cluster detach <PEER_CLUSTER_KUBECONFIG_FILE>
   

For details, see Detach a cluster from the mesh.

2.3.5 - SMM Operator helm charts

You can deploy Service Mesh Manager by using Helm with the SMM Operator chart.

SMM Operator is a Kubernetes operator to deploy and manage Service Mesh Manager. In this chart the CRD is not managed by the operator, and we expect CI/CD tools to take care of updating CRD.

2.3.5.1 - Install SMM with the SMM Operator chart

SMM Operator is a Kubernetes operator to deploy and manage Service Mesh Manager. In this chart the CRD is not managed by the operator, and we expect CI/CD tools to take care of updating CRD.

In case you have your own cluster deployed and are authorized to fetch images from the Cisco provided repositories, then you can rely on BasicAuth(url, username, password) for authentication required to pull images.

You can get a Username and Password by signing up for the Free tier version of Service Mesh Manager.

Prerequisites

Helm version 3.7 or newer.

Steps

1. Create two namespaces, one for smm-operator (called smm-registry-access), and one for cert-manager:

   kubectl create ns smm-registry-access
   kubectl create ns cert-manager
   

   (The smm-registry-access namespace is used because smm-operator should be in the same namespace as the imagepullsecrets-controller.)

2. Run the following helm commands. Replace <your-username> and <your-password> with the ones shown on your Service Mesh Manager download page.

   export HELM_EXPERIMENTAL_OCI=1
   echo <your-password> | helm registry login registry.eticloud.io -u '<your-username>' --password-stdin
   
   helm pull oci://registry.eticloud.io/smm-charts/smm-operator --version 1.11.0
   
   helm install \
     --namespace=smm-registry-access \
     --set "global.ecr.enabled=false" \
     --set "global.basicAuth.username=<your-username>" \
     --set "global.basicAuth.password=<your-password>" \
     smm-operator \
     oci://registry.eticloud.io/smm-charts/smm-operator --version 1.11.0
   

3. Install Service Mesh Manager by creating a ControlPlane resource. We recommend that you start with the following ControlPlane resource. This CR assumes that you are using docker-registry authentication, and the secrets referenced in the .spec.registryAccess is used to pull smm-operator image and sync across other namespaces created by the smm-operator chart.

   Replace <cluster-name> with the name of your cluster. The cluster name format must comply with the RFC 1123 DNS subdomain/label format (alphanumeric string without “_” or “.” characters). Otherwise, you get an error message starting with: Reconciler error: cannot determine cluster name controller=controlplane, controllerGroup=smm.cisco.com, controllerKind=ControlPlane

   kubectl apply -f - << EOF
   apiVersion: smm.cisco.com/v1alpha1
   kind: ControlPlane
   metadata:
     name: smm
   spec:
     clusterName: <cluster-name>
     certManager:
       enabled: true
       namespace: cert-manager
     clusterRegistry:
       enabled: true
       namespace: cluster-registry
     log: {}
     meshManager:
       enabled: true
       istio:
         enabled: true
         istioCRRef:
           name: cp-v115x
           namespace: istio-system
         operators:
           namespace: smm-system
       namespace: smm-system
     nodeExporter:
       enabled: true
       namespace: smm-system
       psp:
         enabled: false
       rbac:
         enabled: true
     oneEye: {}
     registryAccess:
       enabled: true
       imagePullSecretsController: {}
       namespace: smm-registry-access
       pullSecrets:
         - name: smm-registry.eticloud.io-pull-secret
           namespace: smm-registry-access
     repositoryOverride:
       host: registry.eticloud.io
       prefix: smm
     role: active
     smm:
       als:
         enabled: true
         log: {}
       application:
         enabled: true
         log: {}
       auth:
         mode: impersonation
       certManager:
         enabled: true
       enabled: true
       federationGateway:
         enabled: true
         name: smm
         service:
           enabled: true
           name: smm-federation-gateway
           port: 80
       federationGatewayOperator:
         enabled: true
       impersonation:
         enabled: true
       istio:
         revision: cp-v115x.istio-system
       leo:
         enabled: true
         log: {}
       log: {}
       namespace: smm-system
       prometheus:
         enabled: true
         replicas: 1
       prometheusOperator: {}
       releaseName: smm
       role: active
       sre:
         enabled: true
       useIstioResources: true
   EOF
   

Uninstalling the chart

To uninstall/delete the ControlPlane resource and smm-operator release, run:

kubectl delete controlplanes.smm.cisco.com smm
helm uninstall --namespace=smm-registry-access smm-operator

Chart configuration

The following table lists the configurable parameters of the Service Mesh Manager chart and their default values.

2.3.5.2 - The ControlPlane Custom Resource

Service Mesh Manager installs the ControlPlane Custom Resource with the following default values.

apiVersion: smm.cisco.com/v1alpha1
kind: ControlPlane
metadata:
 name: smm
spec:
  smm:
    als:
      enabled: true
      log: {}
    application:
      enabled: true
      log: {}
    auth:
      mode: impersonation
    certManager:
      enabled: true
    enabled: true
    highAvailability:
      enabled: true
    impersonation:
      enabled: true
    istio:
      revision: cp-v115x.istio-system
    leo:
      enabled: true
      log: {}
    log: {}
    namespace: smm-system
    prometheus:
      enabled: true
      replicas: 1
    releaseName: smm
    sre:
      enabled: true
    useIstioResources: true
    web:
      enabled: true
  canaryOperator:
    enabled: false
    namespace: smm-canary
    prometheusURL: http://smm-prometheus.smm-system.svc.cluster.local:59090/prometheus
    releaseName: ""
  certManager:
    manageNamespace: true
    enabled: true
    namespace: cert-manager
  clusterRegistry:
    enabled: true
    namespace: cluster-registry
  meshManager:
    enabled: true
    istio:
      istioCRRef:
        name: cp-v115x
        namespace: istio-system
    namespace: smm-system
    prometheusMetrics:
      authProxy:
        image:
          repository: quay.io/brancz/kube-rbac-proxy
          tag: v0.11.0
  log: {}
  nodeExporter:
    enabled: true
    namespace: smm-system
    psp:
      enabled: false
    rbac:
      enabled: true
  oneEye: {}
  registryAccess:
    enabled: true
    imagePullSecretsController: {}
    namespace: smm-registry-access
    pullSecrets:
    - name: ecr-smm-operator-eti-sre
      namespace: smm-system
    - name: ecr-smm-operator-banzai-customer
      namespace: smm-system
  role: active

To understand how Service Mesh Manager can be customized through its CRs, see Customize Installation.

2.3.6 - Install SMM - GitOps - single cluster

This guide details how to set up a GitOps environment for Service Mesh Manager using Argo CD. The same principles can be used for other tools as well.

Architecture

The high level architecture for Argo CD with a single-cluster Service Mesh Manager consists of the following components:

• A git repository that stores the various charts and manifests,
• a management cluster that runs the Argo CD server, and
• the Service Mesh Manager cluster managed by Argo CD.

Prerequisites

To complete this procedure, you need:

• A free registration for the Service Mesh Manager download page
• A Kubernetes cluster to deploy Argo CD on (called management-cluster in the examples).
• A Kubernetes cluster to deploy Service Mesh Manager on (called workload-cluster-1 in the examples).

Procedure overview

The high-level steps of the procedure are:

1. Install Argo CD and register the clusters
2. Prepare the Git repository
3. Deploy Service Mesh Manager

Install Argo CD

Complete the following steps to install Argo CD on the management cluster.

Set up the environment

1. Set the KUBECONFIG location and context name for the management-cluster cluster.

   MANAGEMENT_CLUSTER_KUBECONFIG=management_cluster_kubeconfig.yaml
   MANAGEMENT_CLUSTER_CONTEXT=management-cluster
   kubectl config --kubeconfig "${MANAGEMENT_CLUSTER_KUBECONFIG}" get-contexts "${MANAGEMENT_CLUSTER_CONTEXT}"
   

   Expected output:

   CURRENT   NAME                 CLUSTER              AUTHINFO   NAMESPACE
   *         management-cluster   management-cluster
   

2. Set the KUBECONFIG location and context name for the workload-cluster-1 cluster.

   WORKLOAD_CLUSTER_1_KUBECONFIG=workload_cluster_1_kubeconfig.yaml
   WORKLOAD_CLUSTER_1_CONTEXT=workload-cluster-1
   kubectl config --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" get-contexts "${WORKLOAD_CLUSTER_1_CONTEXT}"
   

   Expected output:

   CURRENT   NAME                 CLUSTER              AUTHINFO                                          NAMESPACE
   *         workload-cluster-1   workload-cluster-1
   

   Repeat this step for any additional workload clusters you want to use.

3. Make sure the management-cluster Kubernetes context is the current context.

   kubectl config --kubeconfig "${MANAGEMENT_CLUSTER_KUBECONFIG}" use-context "${MANAGEMENT_CLUSTER_CONTEXT}"
   

   Expected output:

   Switched to context "management-cluster".
   

Install Argo CD Server

1. Install the Argo CD Server. Run the following commands.

   kubectl create namespace argocd
   

   Expected output:

   namespace/argocd created
   

   kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
   

   Show expected output ↕

   customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io created
   customresourcedefinition.apiextensions.k8s.io/applicationsets.argoproj.io created
   customresourcedefinition.apiextensions.k8s.io/appprojects.argoproj.io created
   serviceaccount/argocd-application-controller created
   serviceaccount/argocd-applicationset-controller created
   serviceaccount/argocd-dex-server created
   serviceaccount/argocd-notifications-controller created
   serviceaccount/argocd-redis created
   serviceaccount/argocd-repo-server created
   serviceaccount/argocd-server created
   role.rbac.authorization.k8s.io/argocd-application-controller created
   role.rbac.authorization.k8s.io/argocd-applicationset-controller created
   role.rbac.authorization.k8s.io/argocd-dex-server created
   role.rbac.authorization.k8s.io/argocd-notifications-controller created
   role.rbac.authorization.k8s.io/argocd-server created
   clusterrole.rbac.authorization.k8s.io/argocd-application-controller created
   clusterrole.rbac.authorization.k8s.io/argocd-server created
   rolebinding.rbac.authorization.k8s.io/argocd-application-controller created
   rolebinding.rbac.authorization.k8s.io/argocd-applicationset-controller created
   rolebinding.rbac.authorization.k8s.io/argocd-dex-server created
   rolebinding.rbac.authorization.k8s.io/argocd-notifications-controller created
   rolebinding.rbac.authorization.k8s.io/argocd-redis created
   rolebinding.rbac.authorization.k8s.io/argocd-server created
   clusterrolebinding.rbac.authorization.k8s.io/argocd-application-controller created
   clusterrolebinding.rbac.authorization.k8s.io/argocd-server created
   configmap/argocd-cm created
   configmap/argocd-cmd-params-cm created
   configmap/argocd-gpg-keys-cm created
   configmap/argocd-notifications-cm created
   configmap/argocd-rbac-cm created
   configmap/argocd-ssh-known-hosts-cm created
   configmap/argocd-tls-certs-cm created
   secret/argocd-notifications-secret created
   secret/argocd-secret created
   service/argocd-applicationset-controller created
   service/argocd-dex-server created
   service/argocd-metrics created
   service/argocd-notifications-controller-metrics created
   service/argocd-redis created
   service/argocd-repo-server created
   service/argocd-server created
   service/argocd-server-metrics created
   deployment.apps/argocd-applicationset-controller created
   deployment.apps/argocd-dex-server created
   deployment.apps/argocd-notifications-controller created
   deployment.apps/argocd-redis created
   deployment.apps/argocd-repo-server created
   deployment.apps/argocd-server created
   statefulset.apps/argocd-application-controller created
   networkpolicy.networking.k8s.io/argocd-application-controller-network-policy created
   networkpolicy.networking.k8s.io/argocd-applicationset-controller-network-policy created
   networkpolicy.networking.k8s.io/argocd-dex-server-network-policy created
   networkpolicy.networking.k8s.io/argocd-notifications-controller-network-policy created
   networkpolicy.networking.k8s.io/argocd-redis-network-policy created
   networkpolicy.networking.k8s.io/argocd-repo-server-network-policy created
   networkpolicy.networking.k8s.io/argocd-server-network-policy created
   

2. Wait until the installation is complete, then check that the Argo CD pods are up and running.

   kubectl get pods -n argocd
   

   The output should be similar to:

   NAME                                                    READY   STATUS    RESTARTS   AGE
   pod/argocd-application-controller-0                     1/1     Running   0          7h59m
   pod/argocd-applicationset-controller-78b8b554f9-pgwbl   1/1     Running   0          7h59m
   pod/argocd-dex-server-6bbc85c688-8p7zf                  1/1     Running   0          16h
   pod/argocd-notifications-controller-75847756c5-dbbm5    1/1     Running   0          16h
   pod/argocd-redis-f4cdbff57-wcpxh                        1/1     Running   0          7h59m
   pod/argocd-repo-server-d5c7f7ffb-c8962                  1/1     Running   0          7h59m
   pod/argocd-server-76497676b-pnvf4                       1/1     Running   0          7h59m
   

3. For the Argo CD UI, set the argocd-server service type to LoadBalancer.

   kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'
   

   Expected output:

   service/argocd-server patched
   

4. Patch the App of Apps health check in Argo CD configuration to ignore diffs of controller/operator managed fields. For details about this patch, see the Argo CD documentation sections Resource Health and Diffing Customization.

   Apply the new Argo CD health check configurations:

   kubectl apply -f - <<EOF
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: argocd-cm
     namespace: argocd
     labels:
       app.kubernetes.io/name: argocd-cm
       app.kubernetes.io/part-of: argocd
   data:
     # App of app health check
     resource.customizations.health.argoproj.io_Application: |
       hs = {}
       hs.status = "Progressing"
       hs.message = ""
       if obj.status ~= nil then
         if obj.status.health ~= nil then
           hs.status = obj.status.health.status
           if obj.status.health.message ~= nil then
             hs.message = obj.status.health.message
           end
         end
       end
       return hs    
     # Ignoring RBAC changes made by AggregateRoles
     resource.compareoptions: |
       # disables status field diffing in specified resource types
       ignoreAggregatedRoles: true
   
       # disables status field diffing in specified resource types
       # 'crd' - CustomResourceDefinition-s (default)
       # 'all' - all resources
       # 'none' - disabled
       ignoreResourceStatusField: all    
   EOF
   

   Expected output:

   configmap/argocd-cm configured
   

5. Get the initial password for the admin user.

   kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
   

   Expected output:

   argocd-admin-password
   

6. Check the external-ip-or-hostname address of the argocd-server service.

   kubectl get service -n argocd argocd-server
   

   The output should be similar to:

   NAME                                      TYPE           CLUSTER-IP      EXTERNAL-IP               PORT(S)                      AGE
   argocd-server                             LoadBalancer   10.108.14.130   external-ip-or-hostname   80:31306/TCP,443:30063/TCP   7d13h
   

7. Open the https://external-ip-or-hostname URL and log in to the Argo CD server using the password received in the previous step.

   # Exactly one of hostname or IP will be available and used for the remote URL.
   open https://$(kubectl get service -n argocd argocd-server -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{.status.loadBalancer.ingress[0].ip}')
   

Install Argo CD CLI

1. Install Argo CD CLI on your computer. For details, see the Argo CD documentation.

2. Log in with the CLI:

   # Exactly one of hostname or IP will be available and used for the remote URL.
   argocd login $(kubectl get service -n argocd argocd-server -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{.status.loadBalancer.ingress[0].ip}') --insecure --username admin --password $(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)
   

   Expected output:

   'admin:login' logged in successfully
   

For more details about Argo CD installation, see the Argo CD getting started guide.

Register clusters

1. Register the clusters that will run Service Mesh Manager in Argo CD. In this example, register workload-cluster-1 using one of the following methods.

   • Register the cluster from the command line by running:

     argocd cluster add --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" "${WORKLOAD_CLUSTER_1_CONTEXT}"
     

     Expected output:

     WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `workload-cluster-1` with full cluster level privileges. Do you want to continue [y/N]? y
     INFO[0005] ServiceAccount "argocd-manager" created in namespace "kube-system"
     INFO[0005] ClusterRole "argocd-manager-role" created
     INFO[0005] ClusterRoleBinding "argocd-manager-role-binding" created
     INFO[0011] Created bearer token secret for ServiceAccount "argocd-manager"
     Cluster 'https://workload-cluster-1-ip-or-hostname' added
     

   • Alternatively, you can register clusters declaratively as Kubernetes secrets. Modify the following command for your environment and apply it. For details, see the Argo CD documentation.

     WORKLOAD_CLUSTER_1_IP="https://workload-cluster-1-IP" ARGOCD_BEARER_TOKEN="authentication-token" ARGOCD_CA_B64="base64 encoded certificate" ; kubectl apply -f - <<EOF
     apiVersion: v1
     kind: Secret
     metadata:
       name: workload-cluster-1-secret
       labels:
         argocd.argoproj.io/secret-type: cluster
     type: Opaque
     stringData:
       name: workload-cluster-1
       server: "${WORKLOAD_CLUSTER_1_IP}"
       config: |
         {
           "bearerToken": "${ARGOCD_BEARER_TOKEN}",
           "tlsClientConfig": {
             "insecure": false,
             "caData": "${ARGOCD_CA_B64}"
           }
         }    
     EOF
     

2. Make sure that the cluster is registered in Argo CD by running the following command:

   argocd cluster list
   

   The output should be similar to:

   SERVER                                      NAME                VERSION  STATUS   MESSAGE                                                  PROJECT
   https://kubernetes.default.svc              in-cluster                   Unknown  Cluster has no applications and is not being monitored.
   https://workload-cluster-1-ip-or-hostname   workload-cluster-1           Unknown  Cluster has no applications and is not being monitored.
   

Prepare Git repository

1. Create an empty repository called smm-gitops on GitHub (or another provider that Argo CD supports) and initialize it with a README.md file so that you can clone the repository. Because Service Mesh Manager credentials will be stored in this repository, make it a private repository.

   GITHUB_ID="github-id"
   GITHUB_REPOSITORY_NAME="calisti-gitops"
   

2. Obtain a personal access token to the repository (on GitHub, see Creating a personal access token), that has the following permissions:

   • admin:org_hook
   • admin:repo_hook
   • read:org
   • read:public_key
   • repo

3. Log in with your personal access token with git.

   export GH_TOKEN="github-personal-access-token" # Note: this environment variable needs to be exported so the `git` binary is going to use it automatically for authentication.
   

4. Clone the repository into your local workspace.

   git clone "https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git"
   

   Expected output:

   Cloning into 'calisti-gitops'...
   remote: Enumerating objects: 144, done.
   remote: Counting objects: 100% (144/144), done.
   remote: Compressing objects: 100% (93/93), done.
   remote: Total 144 (delta 53), reused 135 (delta 47), pack-reused 0
   Receiving objects: 100% (144/144), 320.08 KiB | 746.00 KiB/s, done.
   Resolving deltas: 100% (53/53), done.
   

5. Add the repository to Argo CD by running the following command. Alternatively, you can add it on Argo CD Web UI.

   argocd repo add "https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git" --name "${GITHUB_REPOSITORY_NAME}" --username "${GITHUB_ID}" --password "${GH_TOKEN}"
   

   Expected output:

   Repository 'https://github.com/github-id/calisti-gitops.git' added
   

6. Verify that the repository is connected by running:

   argocd repo list
   

   In the output, Status should be Successful:

   TYPE  NAME            REPO                                             INSECURE  OCI    LFS    CREDS  STATUS      MESSAGE  PROJECT
   git   calisti-gitops  https://github.com/github-id/calisti-gitops.git  false     false  false  true   Successful
   

7. Change into the root directory of the cloned repository and create the following directories.

   cd "${GITHUB_REPOSITORY_NAME}"
   

   mkdir -p apps/demo-app apps/smm-controlplane apps/smm-operator charts demo-app manifests
   

   The final structure of the repository will look like this:

   .
   ├── apps
   │   ├── demo-app
   │   │   └── demo-app.yaml
   │   ├── smm-controlplane
   │   │   └── smm-controlplane.yaml
   │   └── smm-operator
   │       └── smm-operator.yaml
   ├── charts
   │   └── smm-operator
   │       └── ...
   ├── demo-app
   │   ├── demo-app-ns.yaml
   │   └── demo-app.yaml
   └── manifests
       ├── cert-manager-namespace.yaml
       ├── smm-controlplane.yaml
       ├── istio-cp-v115x.yaml
       └── istio-system-namespace.yaml
   

   • The apps folder contains the Argo CD Application of the smm-operator, the smm-controlplane, and the demo-app.
   • The charts folder contains the Helm chart of the smm-operator.
   • The demo-app folder contains the manifest files of the demo application that represents your business application.
   • The manifests folder contains the smm-controlplane file, the istio-controlplane file, and the cert-manager and istio-system namespace files.

Prepare the helm charts

1. You need an active Service Mesh Manager registration to download the Service Mesh Manager charts and images. You can sign up for free, or obtain Enterprise credentials on the official Cisco Service Mesh Manager page. After registration, you can obtain your username and password from the Download Center. Set them as environment variables.

   CALISTI_USERNAME="<your-calisti-username>"
   

   CALISTI_PASSWORD="<your-calisti-password>"
   

2. Download the smm-operator chart from registry.eticloud.io into the charts directory of your Service Mesh Manager GitOps repository and extract it. Run the following commands:

   export HELM_EXPERIMENTAL_OCI=1 # Needed prior to Helm version 3.8.0
   
   echo "${CALISTI_PASSWORD}" | helm registry login registry.eticloud.io -u "${CALISTI_USERNAME}" --password-stdin
   

   Expected output:

   Login Succeeded
   

   helm pull oci://registry.eticloud.io/smm-charts/smm-operator --destination ./charts/ --untar --version 1.11.0
   

   Expected output:

   Pulled: registry.eticloud.io/smm-charts/smm-operator:latest-stable-version
   Digest: sha256:someshadigest
   

Deploy Service Mesh Manager

Deploy the smm-operator application

Complete the following steps to deploy the smm-operator chart using Argo CD.

1. Create an Argo CD Application CR for smm-operator.

   Before running the following command, edit it if needed:

   • If you are not using a GitHub repository, set the repoURL field to your repository.
   • Set the value of the PUBLIC_API_SERVER_ENDPOINT_ADDRESS variable to the public API endpoint of your cluster. Some managed Kubernetes solutions of public cloud providers have different API Server endpoints for internal and public access.

   ARGOCD_CLUSTER_NAME="${WORKLOAD_CLUSTER_1_CONTEXT}" PUBLIC_API_SERVER_ENDPOINT_ADDRESS="" ; cat > "apps/smm-operator/smm-operator-app.yaml" <<EOF
   # apps/smm-operator/smm-operator-app.yaml
   apiVersion: argoproj.io/v1alpha1
   kind: Application
   metadata:
     name: smm-operator
     namespace: argocd
     finalizers:
       - resources-finalizer.argocd.argoproj.io
   spec:
     project: default
     source:
       repoURL: https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git
       targetRevision: HEAD
       path: charts/smm-operator
       helm:
         parameters:
         - name: "global.ecr.enabled"
           value: 'false'
         - name: "global.basicAuth.username"
           value: "${CALISTI_USERNAME}"
         - name: "global.basicAuth.password"
           value: "${CALISTI_PASSWORD}"
         - name: "apiServerEndpointAddress"
           value: "${PUBLIC_API_SERVER_ENDPOINT_ADDRESS}" # The publicly accessible address of the k8s api server. Some Cloud providers have different API Server endpoint for internal and for public access. In that case the public endpoint needs to be specified here.
     destination:
       name: ${ARGOCD_CLUSTER_NAME}
       namespace: smm-registry-access
     syncPolicy:
       automated:
         prune: true
         selfHeal: true
       syncOptions:
         - Validate=false
         - PruneLast=true
         - CreateNamespace=true
         - Replace=true
   EOF
   

2. Commit and push the calisti-gitops repository.

   git add apps/smm-operator charts/smm-operator
   git commit -m "add smm-operator app"
   

   Show expected output ↕

   [main e6c4b4a] add smm-operator app
   36 files changed, 80324 insertions(+)
   create mode 100644 apps/smm-operator/smm-operator-app.yaml
   create mode 100644 charts/smm-operator/.helmignore
   create mode 100644 charts/smm-operator/Chart.yaml
   create mode 100644 charts/smm-operator/README.md
   create mode 100644 charts/smm-operator/crds/clusterfeature-crd.yaml
   create mode 100644 charts/smm-operator/crds/clusters-crd.yaml
   create mode 100644 charts/smm-operator/crds/crd-alertmanagerconfigs.yaml
   create mode 100644 charts/smm-operator/crds/crd-alertmanagers.yaml
   create mode 100644 charts/smm-operator/crds/crd-podmonitors.yaml
   create mode 100644 charts/smm-operator/crds/crd-probes.yaml
   create mode 100644 charts/smm-operator/crds/crd-prometheuses.yaml
   create mode 100644 charts/smm-operator/crds/crd-prometheusrules.yaml
   create mode 100644 charts/smm-operator/crds/crd-servicemonitors.yaml
   create mode 100644 charts/smm-operator/crds/crd-thanosrulers.yaml
   create mode 100644 charts/smm-operator/crds/crds.yaml
   create mode 100644 charts/smm-operator/crds/health.yaml
   create mode 100644 charts/smm-operator/crds/istio-operator-v1-crds.yaml
   create mode 100644 charts/smm-operator/crds/istio-operator-v2-crds.gen.yaml
   create mode 100644 charts/smm-operator/crds/istiooperator-crd.yaml
   create mode 100644 charts/smm-operator/crds/koperator-crds.yaml
   create mode 100644 charts/smm-operator/crds/metadata-crd.yaml
   create mode 100644 charts/smm-operator/crds/resourcesyncrules-crd.yaml
   create mode 100644 charts/smm-operator/crds/sre.yaml
   create mode 100644 charts/smm-operator/templates/_helpers.tpl
   create mode 100644 charts/smm-operator/templates/authproxy-rbac.yaml
   create mode 100644 charts/smm-operator/templates/authproxy-service.yaml
   create mode 100644 charts/smm-operator/templates/cert-manager-namespace.yaml
   create mode 100644 charts/smm-operator/templates/ecr.deployment.yaml
   create mode 100644 charts/smm-operator/templates/ecr.secret.yaml
   create mode 100644 charts/smm-operator/templates/ecr.service-account.yaml
   create mode 100644 charts/smm-operator/templates/namespace.yaml
   create mode 100644 charts/smm-operator/templates/operator-psp-basic.yaml
   create mode 100644 charts/smm-operator/templates/operator-rbac.yaml
   create mode 100644 charts/smm-operator/templates/operator-service.yaml
   create mode 100644 charts/smm-operator/templates/operator-statefulset.yaml
   create mode 100644 charts/smm-operator/values.yaml
   

   git push
   

   Expected output:

   Enumerating objects: 48, done.
   Counting objects: 100% (48/48), done.
   Delta compression using up to 12 threads
   Compressing objects: 100% (44/44), done.
   Writing objects: 100% (47/47), 282.18 KiB | 1.99 MiB/s, done.
   Total 47 (delta 20), reused 0 (delta 0), pack-reused 0
   remote: Resolving deltas: 100% (20/20), done.
   To github.com:pregnor/calisti-gitops.git
   + 8dd47c2...db9e7af main -> main (forced update)
   

3. Apply the Application manifest.

   kubectl apply -f "apps/smm-operator/smm-operator-app.yaml"
   

   Expected output:

   application.argoproj.io/smm-operator created
   

4. Verify that the applications have been added to Argo CD and are healthy.

   argocd app list
   

   Expected output:

   NAME          CLUSTER             NAMESPACE            PROJECT  STATUS  HEALTH   SYNCPOLICY  CONDITIONS  REPO                                             PATH                 TARGET
   smm-operator  workload-cluster-1  smm-registry-access  default  Synced  Healthy  Auto-Prune  <none>      https://github.com/github-id/calisti-gitops.git  charts/smm-operator  HEAD
   

5. Check the smm-operator application on the Argo CD Web UI.

   

Deploy the smm-controlplane application

1. Create the following namespace for the Service Mesh Manager ControlPlane.

   cat > manifests/cert-manager-namespace.yaml <<EOF
   # manifests/cert-manager-namespace.yaml
   apiVersion: v1
   kind: Namespace
   metadata:
     annotations:
       argocd.argoproj.io/sync-wave: "1"
     name: cert-manager
   EOF
   

2. Create the istio-system-ns.yaml file.

   cat > manifests/istio-system-namespace.yaml << EOF
   # manifests/istio-system-namespace.yaml
   apiVersion: v1
   kind: Namespace
   metadata:
     annotations:
       argocd.argoproj.io/sync-wave: "2"
     name: istio-system
   EOF
   

3. Create the istio-cp-v115x.yaml file.

   cat > manifests/istio-cp-v115x.yaml << EOF
   apiVersion: servicemesh.cisco.com/v1alpha1
   kind: IstioControlPlane
   metadata:
     annotations:
       argocd.argoproj.io/sync-wave: "5"
     name: cp-v115x
     namespace: istio-system
   spec:
     containerImageConfiguration:
       imagePullPolicy: Always
       imagePullSecrets:
       - name: smm-pull-secret
     distribution: cisco
     istiod:
       deployment:
         env:
         - name: ISTIO_MULTIROOT_MESH
           value: "true"
         image: registry.eticloud.io/smm/istio-pilot:v1.15.3-bzc.0
     k8sResourceOverlays:
     - groupVersionKind:
         group: apps
         kind: Deployment
         version: v1
       objectKey:
         name: istiod-cp-v115x
         namespace: istio-system
       patches:
       - path: /spec/template/spec/containers/0/args/-
         type: replace
         value: --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256
     meshConfig:
       defaultConfig:
         envoyAccessLogService:
           address: smm-als.smm-system.svc.cluster.local:50600
           tcpKeepalive:
             interval: 10s
             probes: 3
             time: 10s
           tlsSettings:
             mode: ISTIO_MUTUAL
         holdApplicationUntilProxyStarts: true
         proxyMetadata:
           ISTIO_META_ALS_ENABLED: "true"
           PROXY_CONFIG_XDS_AGENT: "true"
         tracing:
           tlsSettings:
             mode: ISTIO_MUTUAL
           zipkin:
             address: smm-zipkin.smm-system.svc.cluster.local:59411
       enableEnvoyAccessLogService: true
       enableTracing: true
     meshExpansion:
       enabled: true
       gateway:
         deployment:
           podMetadata:
             labels:
               app: istio-meshexpansion-gateway
               istio: meshexpansiongateway
         service:
           ports:
           - name: tcp-smm-als-tls
             port: 50600
             protocol: TCP
             targetPort: 50600
           - name: tcp-smm-zipkin-tls
             port: 59411
             protocol: TCP
             targetPort: 59411
     meshID: mesh1
     mode: ACTIVE
     networkName: network1
     proxy:
       image: registry.eticloud.io/smm/istio-proxyv2:v1.15.3-bzc.0
     proxyInit:
       cni:
         daemonset:
           image: registry.eticloud.io/smm/istio-install-cni:v1.15.3-bzc.0
       image: registry.eticloud.io/smm/istio-proxyv2:v1.15.3-bzc.0
     sidecarInjector:
       deployment:
         image: registry.eticloud.io/smm/istio-sidecar-injector:v1.15.3-bzc.0
     version: 1.15.3
   EOF
   

4. Create the smm-controlplane CR for the ControlPlane.

   ISTIO_MINOR_VERSION="1.15" ; cat > "manifests/smm-controlplane.yaml" <<EOF
   # manifests/smm-controlplane.yaml
   apiVersion: smm.cisco.com/v1alpha1
   kind: ControlPlane
   metadata:
     annotations:
       argocd.argoproj.io/sync-wave: "10"
     name: smm
   spec:
     certManager:
       enabled: true
       namespace: cert-manager
     clusterName: ${ARGOCD_CLUSTER_NAME}
     clusterRegistry:
       enabled: true
       namespace: cluster-registry
     log: {}
     meshManager:
       enabled: true
       istio:
         enabled: true
         istioCRRef:
           name: cp-v${ISTIO_MINOR_VERSION/.}x
           namespace: istio-system
         operators:
           namespace: smm-system
       namespace: smm-system
     nodeExporter:
       enabled: true
       namespace: smm-system
       psp:
         enabled: false
       rbac:
         enabled: true
     oneEye: {}
     registryAccess:
       enabled: true
       imagePullSecretsController: {}
       namespace: smm-registry-access
       pullSecrets:
       - name: smm-registry.eticloud.io-pull-secret
         namespace: smm-registry-access
     repositoryOverride:
       host: registry.eticloud.io
       prefix: smm
     role: active
     smm:
       exposeDashboard:
         meshGateway:
           enabled: true
       als:
         enabled: true
         log: {}
       application:
         enabled: true
         log: {}
       auth:
         forceUnsecureCookies: true
         mode: anonymous
       certManager:
         enabled: true
       enabled: true
       federationGateway:
         enabled: true
         name: smm
         service:
           enabled: true
           name: smm-federation-gateway
           port: 80
       federationGatewayOperator:
         enabled: true
       impersonation:
         enabled: true
       istio:
         revision: cp-v${ISTIO_MINOR_VERSION/.}x.istio-system
       leo:
         enabled: true
         log: {}
       log: {}
       namespace: smm-system
       prometheus:
         enabled: true
         replicas: 1
       prometheusOperator: {}
       releaseName: smm
       role: active
       sre:
         enabled: true
       useIstioResources: true
   EOF
   

5. Create the Argo CD Application CR for the smm-controlplane.

   ARGOCD_CLUSTER_NAME="${WORKLOAD_CLUSTER_1_CONTEXT}" ; cat > "apps/smm-controlplane/smm-controlplane-app.yaml" <<EOF
   # apps/smm-controlplane/smm-controlplane-app.yaml
   apiVersion: argoproj.io/v1alpha1
   kind: Application
   metadata:
     name: smm-controlplane
     namespace: argocd
     finalizers:
       - resources-finalizer.argocd.argoproj.io
   spec:
     project: default
     source:
       repoURL: https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git
       targetRevision: HEAD
       path: manifests
     destination:
       name: ${ARGOCD_CLUSTER_NAME}
     syncPolicy:
       automated:
         prune: true
         selfHeal: true
       syncOptions:
       - Validate=false
       - CreateNamespace=true
       - PrunePropagationPolicy=foreground
       - PruneLast=true
       - Replace=true
   EOF
   

6. Commit the changes and push the calisti-gitops repository.

   git add apps/smm-controlplane manifests
   git commit -m "add smm-controlplane app"
   

   Expected output:

   [main 25ba7e8] add smm-controlplane app
   5 files changed, 212 insertions(+)
   create mode 100644 apps/smm-controlplane/smm-controlplane-app.yaml
   create mode 100644 manifests/cert-manager-namespace.yaml
   create mode 100644 manifests/istio-cp-v115x.yaml
   create mode 100644 manifests/istio-system-namespace.yaml
   create mode 100644 manifests/smm-controlplane.yaml
   

   git push
   

   Expected output:

   Enumerating objects: 12, done.
   Counting objects: 100% (12/12), done.
   Delta compression using up to 10 threads
   Compressing objects: 100% (10/10), done.
   Writing objects: 100% (10/10), 2.70 KiB | 2.70 MiB/s, done.
   Total 10 (delta 1), reused 0 (delta 0), pack-reused 0
   remote: Resolving deltas: 100% (1/1), done.
   To github.com:<username>/calisti-gitops.git
     529545a..25ba7e8  main -> main
   

7. Apply the Application manifest.

   kubectl apply -f "apps/smm-controlplane/smm-controlplane-app.yaml"
   

   Expected output:

   application.argoproj.io/smm-controlplane created
   

8. Verify that the application has been added to Argo CD and is healthy.

   argocd app list
   

   Expected output:

   NAME              CLUSTER             NAMESPACE            PROJECT  STATUS     HEALTH   SYNCPOLICY  CONDITIONS  REPO                                             PATH                 TARGET
   smm-controlplane  workload-cluster-1                       default  Synced     Healthy  Auto-Prune  <none>      https://github.com/github-id/calisti-gitops.git  manifests            HEAD
   smm-operator      workload-cluster-1  smm-registry-access  default  Synced     Healthy  Auto-Prune  <none>      https://github.com/github-id/calisti-gitops.git  charts/smm-operator  HEAD
   

9. Check that all pods are healthy and running in the smm-system namespace of workload-cluster-1.

   kubectl get pods -n smm-system --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_1_CONTEXT}"
   

   Show expected output ↕

   NAME                                               READY   STATUS    RESTARTS        AGE
   istio-operator-v115x-85495cd76f-q7n22              2/2     Running   4 (7m36s ago)   17m
   mesh-manager-0                                     2/2     Running   4 (7m35s ago)   18m
   prometheus-smm-prometheus-0                        4/4     Running   0               15m
   smm-7f95479ff7-rzh2g                               2/2     Running   0               16m
   smm-7f95479ff7-v52vp                               2/2     Running   0               16m
   smm-als-8487fdf4f7-ddklg                           2/2     Running   0               16m
   smm-authentication-7888dfc6d7-w7tdq                2/2     Running   0               16m
   smm-federation-gateway-84f9fbf54d-7glvp            2/2     Running   0               16m
   smm-federation-gateway-operator-6cb99c5798-9fj25   2/2     Running   4 (7m36s ago)   16m
   smm-grafana-95ff96dd9-m6rx6                        3/3     Running   0               16m
   smm-health-86dc8c98d6-pv7bk                        2/2     Running   3 (7m35s ago)   16m
   smm-health-api-5df5b76bf5-lvbsp                    2/2     Running   0               16m
   smm-ingressgateway-7d59684cf7-jsj7f                1/1     Running   0               16m
   smm-ingressgateway-external-59f9874787-p55wr       1/1     Running   0               16m
   smm-kubestatemetrics-f4766d7b8-9mc9f               2/2     Running   0               16m
   smm-leo-9fc8db6db-vlzpw                            2/2     Running   0               16m
   smm-prometheus-operator-6558dbddc8-bgdh9           3/3     Running   1 (16m ago)     16m
   smm-sre-alert-exporter-6656f98dd8-8wvdx            2/2     Running   0               16m
   smm-sre-api-77b65ff6bd-spzk2                       2/2     Running   0               16m
   smm-sre-controller-59d6cdd588-7cvbk                2/2     Running   3 (7m35s ago)   16m
   smm-tracing-6c85986bfd-xjjqw                       2/2     Running   0               16m
   smm-vm-integration-cdd8d8688-sk79s                 2/2     Running   3 (7m35s ago)   16m
   smm-web-84d697fdb4-2fbkm                           3/3     Running   0               16m
   

10. Check the application on Argo CD Web UI.

    # Exactly one of hostname or IP will be available and used for the remote URL.
    open https://$(kubectl get service -n argocd argocd-server -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{.status.loadBalancer.ingress[0].ip}')
    

    

At this point, you have successfully installed smm-operator and smm-controlplane on workload-cluster-1.

Deploy an application

If you want to deploy an application into the service mesh, complete the following steps. The examples use the Service Mesh Manager demo application.

1. Create a namespace for the application: create the demo-app-ns.yaml file.

   cat > demo-app/demo-app-ns.yaml << EOF
   apiVersion: v1
   kind: Namespace
   metadata:
     labels:
       app.kubernetes.io/instance: smm-demo
       app.kubernetes.io/name: smm-demo
       app.kubernetes.io/part-of: smm-demo
       app.kubernetes.io/version: 0.1.4
       istio.io/rev: cp-v115x.istio-system
     name: smm-demo
   EOF
   

2. Create the demo-app.yaml file.

   cat > demo-app/demo-app.yaml << EOF
   apiVersion: smm.cisco.com/v1alpha1
   kind: DemoApplication
   metadata:
     name: smm-demo
     namespace: smm-demo
   spec:
     autoscaling:
       enabled: true
     controlPlaneRef:
       name: smm
     deployIstioResources: true
     deploySLOResources: true
     enabled: true
     enabledComponents:
     - frontpage
     - catalog
     - bookings
     - postgresql
     - payments
     - notifications
     - movies
     - analytics
     - database
     - mysql
     istio:
       revision: cp-v115x.istio-system
     load:
       enabled: true
       maxRPS: 30
       minRPS: 10
       swingPeriod: 1380000000000
     replicas: 1
     resources:
       limits:
         cpu: "2"
         memory: 192Mi
       requests:
         cpu: 40m
         memory: 64Mi
   EOF
   

3. Create an Argo CD Application file for the application. Create the demo-app.yaml file.

   ARGOCD_CLUSTER_NAME="${WORKLOAD_CLUSTER_1_CONTEXT}" ; cat > apps/demo-app/demo-app.yaml << EOF
   apiVersion: argoproj.io/v1alpha1
   kind: Application
   metadata:
     name: demo-app
     namespace: argocd
     finalizers:
       - resources-finalizer.argocd.argoproj.io
   spec:
     project: default
     source:
       repoURL: https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git
       targetRevision: HEAD
       path: demo-app
     destination:
       name: ${ARGOCD_CLUSTER_NAME}
       namespace: smm-demo
     syncPolicy:
       automated:
         prune: true
         selfHeal: true
       syncOptions:
       - Validate=false
       - CreateNamespace=true
       - PruneLast=true
       - Replace=true
   EOF
   

4. Commit and push the calisti-gitops repository.

   git add apps/demo-app demo-app
   git commit -m "add demo app"
   

   Expected output:

   [main 58a236e] add demo app
   3 files changed, 74 insertions(+)
   create mode 100644 apps/demo-app/demo-app.yaml
   create mode 100644 demo-app/demo-app-ns.yaml
   create mode 100644 demo-app/demo-app.yaml
   

   git push
   

   Expected output:

   Enumerating objects: 10, done.
   Counting objects: 100% (10/10), done.
   Delta compression using up to 10 threads
   Compressing objects: 100% (7/7), done.
   Writing objects: 100% (8/8), 1.37 KiB | 1.37 MiB/s, done.
   Total 8 (delta 0), reused 0 (delta 0), pack-reused 0
   To github.com:<username>/calisti-gitops.git
     e16549e..58a236e  main -> main
   

5. Deploy the application.

   kubectl apply -f apps/demo-app/demo-app.yaml
   

6. Wait until all the pods in the application namespace (smm-demo) are up and running.

   kubectl get pods -n smm-demo --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_1_CONTEXT}"
   

   Expected output:

   NAME                                READY   STATUS    RESTARTS   AGE
   analytics-v1-7899bd4d4-bnf24        2/2     Running   0          109s
   bombardier-6455fd74f6-jndpv         2/2     Running   0          109s
   bookings-v1-559768454c-7vhzr        2/2     Running   0          109s
   catalog-v1-99b7bb56d-fjvhl          2/2     Running   0          109s
   database-v1-5cb4b4ff67-95ttk        2/2     Running   0          109s
   frontpage-v1-5b4dcbfcb4-djr72       2/2     Running   0          108s
   movies-v1-78fcf666dc-z8c2z          2/2     Running   0          108s
   movies-v2-84d9f5658f-kc65j          2/2     Running   0          108s
   movies-v3-86bbbc9745-r84bl          2/2     Running   0          108s
   mysql-d6b6b78fd-b7dwb               2/2     Running   0          108s
   notifications-v1-794c5dd8f6-lndh4   2/2     Running   0          108s
   payments-v1-858d4b4ffc-vtxxl        2/2     Running   0          108s
   postgresql-555fd55bdb-jn5pq         2/2     Running   0          108s
   

7. Verify that the application appears on the Argo CD admin view, it is Healthy, and Synced.

   

Access the Service Mesh Manager dashboard

1. You can access the Service Mesh Manager dashboard via the smm-ingressgateway-external LoadBalancer external-ip-or-hostname address. Run the following command to retrieve the IP address:

   kubectl get services -n smm-system smm-ingressgateway-external --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_1_CONTEXT}"
   

   Expected output:

   NAME                          TYPE           CLUSTER-IP   EXTERNAL-IP                PORT(S)        AGE
   smm-ingressgateway-external   LoadBalancer   10.0.0.199   external-ip-or-hostname    80:32505/TCP   2m28s
   

2. Open the Service Mesh Manager dashboard using one of the following methods:

   • Open the http://<external-ip-or-hostname> URL in your browser.

   • Run the following command to open the dashboard with your default browser:

     # Exactly one of hostname or IP will be available and used for the remote URL.
     open http://$(kubectl get services -n smm-system smm-ingressgateway-external -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{.status.loadBalancer.ingress[0].ip}' --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_1_CONTEXT}")
     

   • If you have installed the Service Mesh Manager CLI on your machine, run the following command to open the Service Mesh Manager Dashboard in the default browser.

     smm dashboard --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_1_CONTEXT}"
     

     Expected output:

     ✓ validate-kubeconfig ❯ checking cluster reachability...
     ✓ opening Service Mesh Manager at http://127.0.0.1:50500
     

3. Check the deployments on the dashboard, for example, on the MENU > Overview, MENU > MESH, and MENU > TOPOLOGY pages.

2.3.7 - Install SMM - GitOps - multi-cluster

This guide details how to set up a multi-cluster Service Mesh Manager scenario in a GitOps environment for Service Mesh Manager using Argo CD. The same principles can be used for other tools as well.

Architecture

Service Mesh Manager supports multiple mesh topologies, so you can use the one that best fits your use cases. In multi-cluster configurations it provides automatic locality load-balancing.

The high level architecture for Argo CD with a multi-cluster Service Mesh Manager setup consists of the following components:

• A git repository that stores the various charts and manifests,
• a management cluster that runs the Argo CD server, and
• the Service Mesh Manager clusters managed by Argo CD.

Deployment models

When deploying Service Mesh Manager in a multi-cluster scenario you can deploy Service Mesh Manager in an active-passive model. For details on Service Mesh Manager clusters and their relationship to Istio clusters, see Istio clusters and SMM clusters.

Prerequisites

• A free registration for the Service Mesh Manager download page
• A Kubernetes cluster to deploy Argo CD on (called management-cluster in the examples).
• Two Kubernetes clusters to deploy Service Mesh Manager on (called workload-cluster-1 and workload-cluster-2 in the examples).

Install Argo CD

Complete the following steps to install Argo CD on the management cluster.

Set up the environment

1. Set the KUBECONFIG location and context name for the management-cluster cluster.

   MANAGEMENT_CLUSTER_KUBECONFIG=management_cluster_kubeconfig.yaml
   MANAGEMENT_CLUSTER_CONTEXT=management-cluster
   kubectl config --kubeconfig "${MANAGEMENT_CLUSTER_KUBECONFIG}" get-contexts "${MANAGEMENT_CLUSTER_CONTEXT}"
   

   Expected output:

   CURRENT   NAME                 CLUSTER              AUTHINFO   NAMESPACE
   *         management-cluster   management-cluster
   

2. Set the KUBECONFIG location and context name for the workload-cluster-1 cluster.

   WORKLOAD_CLUSTER_1_KUBECONFIG=workload_cluster_1_kubeconfig.yaml
   WORKLOAD_CLUSTER_1_CONTEXT=workload-cluster-1
   kubectl config --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" get-contexts "${WORKLOAD_CLUSTER_1_CONTEXT}"
   

   Expected output:

   CURRENT   NAME                 CLUSTER              AUTHINFO                                          NAMESPACE
   *         workload-cluster-1   workload-cluster-1
   

   Repeat this step for any additional workload clusters you want to use.

3. Make sure the management-cluster Kubernetes context is the current context.

   kubectl config --kubeconfig "${MANAGEMENT_CLUSTER_KUBECONFIG}" use-context "${MANAGEMENT_CLUSTER_CONTEXT}"
   

   Expected output:

   Switched to context "management-cluster".
   

Install Argo CD Server

1. Install the Argo CD Server. Run the following commands.

   kubectl create namespace argocd
   

   Expected output:

   namespace/argocd created
   

   kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
   

   Show expected output ↕

   customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io created
   customresourcedefinition.apiextensions.k8s.io/applicationsets.argoproj.io created
   customresourcedefinition.apiextensions.k8s.io/appprojects.argoproj.io created
   serviceaccount/argocd-application-controller created
   serviceaccount/argocd-applicationset-controller created
   serviceaccount/argocd-dex-server created
   serviceaccount/argocd-notifications-controller created
   serviceaccount/argocd-redis created
   serviceaccount/argocd-repo-server created
   serviceaccount/argocd-server created
   role.rbac.authorization.k8s.io/argocd-application-controller created
   role.rbac.authorization.k8s.io/argocd-applicationset-controller created
   role.rbac.authorization.k8s.io/argocd-dex-server created
   role.rbac.authorization.k8s.io/argocd-notifications-controller created
   role.rbac.authorization.k8s.io/argocd-server created
   clusterrole.rbac.authorization.k8s.io/argocd-application-controller created
   clusterrole.rbac.authorization.k8s.io/argocd-server created
   rolebinding.rbac.authorization.k8s.io/argocd-application-controller created
   rolebinding.rbac.authorization.k8s.io/argocd-applicationset-controller created
   rolebinding.rbac.authorization.k8s.io/argocd-dex-server created
   rolebinding.rbac.authorization.k8s.io/argocd-notifications-controller created
   rolebinding.rbac.authorization.k8s.io/argocd-redis created
   rolebinding.rbac.authorization.k8s.io/argocd-server created
   clusterrolebinding.rbac.authorization.k8s.io/argocd-application-controller created
   clusterrolebinding.rbac.authorization.k8s.io/argocd-server created
   configmap/argocd-cm created
   configmap/argocd-cmd-params-cm created
   configmap/argocd-gpg-keys-cm created
   configmap/argocd-notifications-cm created
   configmap/argocd-rbac-cm created
   configmap/argocd-ssh-known-hosts-cm created
   configmap/argocd-tls-certs-cm created
   secret/argocd-notifications-secret created
   secret/argocd-secret created
   service/argocd-applicationset-controller created
   service/argocd-dex-server created
   service/argocd-metrics created
   service/argocd-notifications-controller-metrics created
   service/argocd-redis created
   service/argocd-repo-server created
   service/argocd-server created
   service/argocd-server-metrics created
   deployment.apps/argocd-applicationset-controller created
   deployment.apps/argocd-dex-server created
   deployment.apps/argocd-notifications-controller created
   deployment.apps/argocd-redis created
   deployment.apps/argocd-repo-server created
   deployment.apps/argocd-server created
   statefulset.apps/argocd-application-controller created
   networkpolicy.networking.k8s.io/argocd-application-controller-network-policy created
   networkpolicy.networking.k8s.io/argocd-applicationset-controller-network-policy created
   networkpolicy.networking.k8s.io/argocd-dex-server-network-policy created
   networkpolicy.networking.k8s.io/argocd-notifications-controller-network-policy created
   networkpolicy.networking.k8s.io/argocd-redis-network-policy created
   networkpolicy.networking.k8s.io/argocd-repo-server-network-policy created
   networkpolicy.networking.k8s.io/argocd-server-network-policy created
   

2. Wait until the installation is complete, then check that the Argo CD pods are up and running.

   kubectl get pods -n argocd
   

   The output should be similar to:

   NAME                                                    READY   STATUS    RESTARTS   AGE
   pod/argocd-application-controller-0                     1/1     Running   0          7h59m
   pod/argocd-applicationset-controller-78b8b554f9-pgwbl   1/1     Running   0          7h59m
   pod/argocd-dex-server-6bbc85c688-8p7zf                  1/1     Running   0          16h
   pod/argocd-notifications-controller-75847756c5-dbbm5    1/1     Running   0          16h
   pod/argocd-redis-f4cdbff57-wcpxh                        1/1     Running   0          7h59m
   pod/argocd-repo-server-d5c7f7ffb-c8962                  1/1     Running   0          7h59m
   pod/argocd-server-76497676b-pnvf4                       1/1     Running   0          7h59m
   

3. For the Argo CD UI, set the argocd-server service type to LoadBalancer.

   kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'
   

   Expected output:

   service/argocd-server patched
   

4. Patch the App of Apps health check in Argo CD configuration to ignore diffs of controller/operator managed fields. For details about this patch, see the Argo CD documentation sections Resource Health and Diffing Customization.

   Apply the new Argo CD health check configurations:

   kubectl apply -f - <<EOF
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: argocd-cm
     namespace: argocd
     labels:
       app.kubernetes.io/name: argocd-cm
       app.kubernetes.io/part-of: argocd
   data:
     # App of app health check
     resource.customizations.health.argoproj.io_Application: |
       hs = {}
       hs.status = "Progressing"
       hs.message = ""
       if obj.status ~= nil then
         if obj.status.health ~= nil then
           hs.status = obj.status.health.status
           if obj.status.health.message ~= nil then
             hs.message = obj.status.health.message
           end
         end
       end
       return hs    
     # Ignoring RBAC changes made by AggregateRoles
     resource.compareoptions: |
       # disables status field diffing in specified resource types
       ignoreAggregatedRoles: true
   
       # disables status field diffing in specified resource types
       # 'crd' - CustomResourceDefinition-s (default)
       # 'all' - all resources
       # 'none' - disabled
       ignoreResourceStatusField: all    
   EOF
   

   Expected output:

   configmap/argocd-cm configured
   

5. Get the initial password for the admin user.

   kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
   

   Expected output:

   argocd-admin-password
   

6. Check the external-ip-or-hostname address of the argocd-server service.

   kubectl get service -n argocd argocd-server
   

   The output should be similar to:

   NAME                                      TYPE           CLUSTER-IP      EXTERNAL-IP               PORT(S)                      AGE
   argocd-server                             LoadBalancer   10.108.14.130   external-ip-or-hostname   80:31306/TCP,443:30063/TCP   7d13h
   

7. Open the https://external-ip-or-hostname URL and log in to the Argo CD server using the password received in the previous step.

   # Exactly one of hostname or IP will be available and used for the remote URL.
   open https://$(kubectl get service -n argocd argocd-server -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{.status.loadBalancer.ingress[0].ip}')
   

Install Argo CD CLI

1. Install Argo CD CLI on your computer. For details, see the Argo CD documentation.

2. Log in with the CLI:

   # Exactly one of hostname or IP will be available and used for the remote URL.
   argocd login $(kubectl get service -n argocd argocd-server -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{.status.loadBalancer.ingress[0].ip}') --insecure --username admin --password $(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)
   

   Expected output:

   'admin:login' logged in successfully
   

For more details about Argo CD installation, see the Argo CD getting started guide.

Register clusters

1. Register the clusters that will run Service Mesh Manager in Argo CD. In this example, register workload-cluster-1 and workload-cluster-2 using one of the following methods.

   • Register the cluster from the command line by running:

     argocd cluster add --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" "${WORKLOAD_CLUSTER_1_CONTEXT}"
     

     Expected output:

     WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `workload-cluster-1` with full cluster level privileges. Do you want to continue [y/N]? y
     INFO[0005] ServiceAccount "argocd-manager" created in namespace "kube-system"
     INFO[0005] ClusterRole "argocd-manager-role" created
     INFO[0005] ClusterRoleBinding "argocd-manager-role-binding" created
     INFO[0011] Created bearer token secret for ServiceAccount "argocd-manager"
     Cluster 'https://workload-cluster-1-ip-or-hostname' added
     

     argocd cluster add --kubeconfig "${WORKLOAD_CLUSTER_2_KUBECONFIG}" "${WORKLOAD_CLUSTER_2_CONTEXT}"
     

     Expected output:

     WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `workload-cluster-2` with full cluster level privileges. Do you want to continue [y/N]? y
     INFO[0005] ServiceAccount "argocd-manager" created in namespace "kube-system"
     INFO[0005] ClusterRole "argocd-manager-role" created
     INFO[0005] ClusterRoleBinding "argocd-manager-role-binding" created
     INFO[0011] Created bearer token secret for ServiceAccount "argocd-manager"
     Cluster 'https://workload-cluster-2-ip-or-hostname' added
     

   • Alternatively, you can register clusters declaratively as Kubernetes secrets. Modify the following command for your environment and apply it. For details, see the Argo CD documentation.

     WORKLOAD_CLUSTER_1_IP="https://workload-cluster-1-IP" ARGOCD_BEARER_TOKEN="authentication-token" ARGOCD_CA_B64="base64 encoded certificate" ; kubectl apply -f - <<EOF
     apiVersion: v1
     kind: Secret
     metadata:
       name: workload-cluster-1-secret
       labels:
         argocd.argoproj.io/secret-type: cluster
     type: Opaque
     stringData:
       name: workload-cluster-1
       server: "${WORKLOAD_CLUSTER_1_IP}"
       config: |
         {
           "bearerToken": "${ARGOCD_BEARER_TOKEN}",
           "tlsClientConfig": {
             "insecure": false,
             "caData": "${ARGOCD_CA_B64}"
           }
         }    
     EOF
     

     WORKLOAD_CLUSTER_2_IP="https://workload-cluster-2-IP" ARGOCD_BEARER_TOKEN="authentication-token" ARGOCD_CA_B64="base64 encoded certificate" ; kubectl apply -f - <<EOF
     apiVersion: v1
     kind: Secret
     metadata:
       name: workload-cluster-2-secret
       labels:
         argocd.argoproj.io/secret-type: cluster
     type: Opaque
     stringData:
       name: workload-cluster-2
       server: "${WORKLOAD_CLUSTER_2_IP}"
       config: |
         {
           "bearerToken": "${ARGOCD_BEARER_TOKEN}",
           "tlsClientConfig": {
             "insecure": false,
             "caData": "${ARGOCD_CA_B64}"
           }
         }    
     EOF
     

2. Make sure that the cluster is registered in Argo CD by running the following command:

   argocd cluster list
   

   The output should be similar to:

   SERVER                                      NAME                VERSION  STATUS   MESSAGE                                                  PROJECT
   https://kubernetes.default.svc              in-cluster                   Unknown  Cluster has no applications and is not being monitored.
   https://workload-cluster-1-ip-or-hostname   workload-cluster-1           Unknown  Cluster has no applications and is not being monitored.
   https://workload-cluster-2-ip-or-hostname   workload-cluster-2           Unknown  Cluster has no applications and is not being monitored.
   

Prepare Git repository

1. Create an empty repository called smm-gitops on GitHub (or another provider that Argo CD supports) and initialize it with a README.md file so that you can clone the repository. Because Service Mesh Manager credentials will be stored in this repository, make it a private repository.

   GITHUB_ID="github-id"
   GITHUB_REPOSITORY_NAME="calisti-gitops"
   

2. Obtain a personal access token to the repository (on GitHub, see Creating a personal access token), that has the following permissions:

   • admin:org_hook
   • admin:repo_hook
   • read:org
   • read:public_key
   • repo

3. Log in with your personal access token with git.

   export GH_TOKEN="github-personal-access-token" # Note: this environment variable needs to be exported so the `git` binary is going to use it automatically for authentication.
   

4. Clone the repository into your local workspace, for example:

   git clone "https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git"
   

   Expected output:

   Cloning into 'calisti-gitops'...
   remote: Enumerating objects: 144, done.
   remote: Counting objects: 100% (144/144), done.
   remote: Compressing objects: 100% (93/93), done.
   remote: Total 144 (delta 53), reused 135 (delta 47), pack-reused 0
   Receiving objects: 100% (144/144), 320.08 KiB | 746.00 KiB/s, done.
   Resolving deltas: 100% (53/53), done.
   

5. Add the repository to Argo CD by running the following command. Alternatively, you can add it on Argo CD Web UI.

   argocd repo add "https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git" --name "${GITHUB_REPOSITORY_NAME}" --username "${GITHUB_ID}" --password "${GH_TOKEN}"
   

   Expected output:

   Repository 'https://github.com/github-id/calisti-gitops.git' added
   

6. Verify that the repository is connected by running:

   argocd repo list
   

   In the output, Status should be Successful:

   TYPE  NAME            REPO                                             INSECURE  OCI    LFS    CREDS  STATUS      MESSAGE  PROJECT
   git   calisti-gitops  https://github.com/github-id/calisti-gitops.git  false     false  false  true   Successful
   

7. Change into the directory of the cloned repository (for example, smm-gitops) and create the following directories.

   cd "${GITHUB_REPOSITORY_NAME}"
   

   mkdir -p apps/smm-controlplane apps/smm-operator apps/demo-app charts manifests/smm-controlplane/base manifests/smm-controlplane/overlays/workload-cluster-1 manifests/smm-controlplane/overlays/workload-cluster-2 manifests/demo-app/base manifests/demo-app/overlays/workload-cluster-1 manifests/demo-app/overlays/workload-cluster-2
   

   The final structure of the repository will look like this:

   .
   ├── README.md
   ├── apps
   │   ├── smm-controlplane
   │   │   └── app-set.yaml
   │   ├── smm-operator
   │   │   └── app-set.yaml
   │   └── demo-app
   │       └── app-set.yaml
   ├── charts
   │   └── smm-operator
   │       ├── Chart.yaml
   │       └── ...
   ├── export-secrets.sh
   └── manifests
      ├── smm-controlplane
      │   ├── base
      │   │   ├── control-plane.yaml
      │   │   ├── cert-manager-namespace.yaml
      │   │   ├── istio-system-namespace.yaml
      │   │   ├── istio-cp-v115x.yaml
      │   │   └── kustomization.yaml
      │   └── overlays
      │       ├── workload-cluster-1
      │       │   ├── control-plane.yaml
      │       │   ├── istio-cp-v115x.yaml
      │       │   └── kustomization.yaml
      │       └── workload-cluster-2
      │           ├── control-plane.yaml
      │           ├── istio-cp-v115x.yaml
      │           └── kustomization.yaml
      └── demo-app
          ├── base
          │   ├── demo-app-namespace.yaml
          │   ├── demo-app.yaml
          │   └── kustomization.yaml
          └── overlays
              ├── workload-cluster-1
              │   ├── demo-app.yaml
              │   └── kustomization.yaml
              └── workload-cluster-2
                  ├── demo-app.yaml
                  └── kustomization.yaml
   

   • The apps folder contains the Argo CD Application of the smm-operator, the smm-controlplane, and the demo-app.
   • The charts folder contains the Helm chart of the smm-operator.
   • The manifests/demo-app folder contains the manifest files of the demo application that represents your business application.
   • The manifests/smm-controlplane folder contains the manifest files of the SMM ControlPlane.

Prepare the helm charts

1. You need an active Service Mesh Manager registration to download the Service Mesh Manager charts and images. You can sign up for free, or obtain Enterprise credentials on the official Cisco Service Mesh Manager page. After registration, you can obtain your username and password from the Download Center. Set them as environment variables.

   CALISTI_USERNAME="<your-calisti-username>"
   

   CALISTI_PASSWORD="<your-calisti-password>"
   

2. Download the smm-operator chart from registry.eticloud.io into the charts directory of your Service Mesh Manager GitOps repository and extract it. Run the following commands:

   export HELM_EXPERIMENTAL_OCI=1 # Needed prior to Helm version 3.8.0
   
   echo "${CALISTI_PASSWORD}" | helm registry login registry.eticloud.io -u "${CALISTI_USERNAME}" --password-stdin
   

   Expected output:

   Login Succeeded
   

   helm pull oci://registry.eticloud.io/smm-charts/smm-operator --destination ./charts/ --untar --version 1.11.0
   

   Expected output:

   Pulled: registry.eticloud.io/smm-charts/smm-operator:latest-stable-version
   Digest: sha256:someshadigest
   

Deploy Service Mesh Manager

Deploy the smm-operator application set

Complete the following steps to deploy the smm-operator chart using Argo CD.

1. Create the smm-operator’s Argo CD ApplicationSet CR. Argo CD ApplicationSet is perfect for deploying the same application on to different clusters. You can use list generators with cluster data in the ApplicationSet.

   Before running the following command, edit it if needed:

   • If you are not using a GitHub repository, set the repoURL field to your repository.
   • Set the value of the PUBLIC_API_SERVER_ENDPOINT_ADDRESS variable to the public API endpoint of your cluster. Some managed Kubernetes solutions of public cloud providers have different API Server endpoints for internal and public access.

   PUBLIC_API_SERVER_ENDPOINT_ADDRESS_1="" PUBLIC_API_SERVER_ENDPOINT_ADDRESS_2="" ;
   cat > "apps/smm-operator/app-set.yaml" <<EOF
   # apps/smm-operator/app-set.yaml
   apiVersion: argoproj.io/v1alpha1
   kind: ApplicationSet
   metadata:
     name: smm-operator-appset
     namespace: argocd
   spec:
     generators:
     - list:
         elements:
         - cluster: "${WORKLOAD_CLUSTER_1_CONTEXT}"
           apiServerEndpointAddress: "${PUBLIC_API_SERVER_ENDPOINT_ADDRESS_1}"
         - cluster: "${WORKLOAD_CLUSTER_2_CONTEXT}"
           apiServerEndpointAddress: "${PUBLIC_API_SERVER_ENDPOINT_ADDRESS_2}"
     template:
       metadata: 
         name: 'smm-operator-{{cluster}}'
         namespace: argocd
       spec:
         project: default
         source:
           repoURL: https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git
           targetRevision: HEAD
           path: charts/smm-operator
           helm:
             parameters:
             - name: "global.ecr.enabled"
               value: 'false'
             - name: "global.basicAuth.username"
               value: "${CALISTI_USERNAME}"
             - name: "global.basicAuth.password"
               value: "${CALISTI_PASSWORD}"
             - name: "apiServerEndpointAddress"
               value: '{{apiServerEndpointAddress}}'
         destination:
           namespace: smm-registry-access
           name: '{{cluster}}'
         ignoreDifferences:
         - kind: ValidatingWebhookConfiguration
           group: admissionregistration.k8s.io
           jsonPointers:
           - /webhooks
         syncPolicy:
           automated:
             prune: true
             selfHeal: true
           retry:
             limit: 5
             backoff:
               duration: 5s
               maxDuration: 3m0s
               factor: 2
           syncOptions:
             - Validate=false
             - PruneLast=true
             - CreateNamespace=true
             - Replace=true
   EOF
   

2. Commit and push the calisti-gitops repository.

   git add apps/smm-operator charts/smm-operator
   

   git commit -m "add smm-operator app"
   

   Show expected output ↕

   [main d4f6809] add smm-operator app
   35 files changed, 80310 insertions(+)
   create mode 100644 apps/smm-operator/app-set.yaml
   create mode 100644 charts/smm-operator/.helmignore
   create mode 100644 charts/smm-operator/Chart.yaml
   create mode 100644 charts/smm-operator/README.md
   create mode 100644 charts/smm-operator/crds/clusterfeature-crd.yaml
   create mode 100644 charts/smm-operator/crds/clusters-crd.yaml
   create mode 100644 charts/smm-operator/crds/crd-alertmanagerconfigs.yaml
   create mode 100644 charts/smm-operator/crds/crd-alertmanagers.yaml
   create mode 100644 charts/smm-operator/crds/crd-podmonitors.yaml
   create mode 100644 charts/smm-operator/crds/crd-probes.yaml
   create mode 100644 charts/smm-operator/crds/crd-prometheuses.yaml
   create mode 100644 charts/smm-operator/crds/crd-prometheusrules.yaml
   create mode 100644 charts/smm-operator/crds/crd-servicemonitors.yaml
   create mode 100644 charts/smm-operator/crds/crd-thanosrulers.yaml
   create mode 100644 charts/smm-operator/crds/crds.yaml
   create mode 100644 charts/smm-operator/crds/health.yaml
   create mode 100644 charts/smm-operator/crds/istio-operator-v1-crds.yaml
   create mode 100644 charts/smm-operator/crds/istio-operator-v2-crds.gen.yaml
   create mode 100644 charts/smm-operator/crds/istiooperator-crd.yaml
   create mode 100644 charts/smm-operator/crds/koperator-crds.yaml
   create mode 100644 charts/smm-operator/crds/metadata-crd.yaml
   create mode 100644 charts/smm-operator/crds/resourcesyncrules-crd.yaml
   create mode 100644 charts/smm-operator/crds/sre.yaml
   create mode 100644 charts/smm-operator/templates/_helpers.tpl
   create mode 100644 charts/smm-operator/templates/authproxy-rbac.yaml
   create mode 100644 charts/smm-operator/templates/authproxy-service.yaml
   create mode 100644 charts/smm-operator/templates/ecr.deployment.yaml
   create mode 100644 charts/smm-operator/templates/ecr.secret.yaml
   create mode 100644 charts/smm-operator/templates/ecr.service-account.yaml
   create mode 100644 charts/smm-operator/templates/namespace.yaml
   create mode 100644 charts/smm-operator/templates/operator-psp-basic.yaml
   create mode 100644 charts/smm-operator/templates/operator-rbac.yaml
   create mode 100644 charts/smm-operator/templates/operator-service.yaml
   create mode 100644 charts/smm-operator/templates/operator-statefulset.yaml
   create mode 100644 charts/smm-operator/values.yaml
   

   git push
   

3. Apply the Application manifests.

   kubectl apply -f "apps/smm-operator/app-set.yaml"
   

4. Verify that the applications have been added to Argo CD and are healthy.

   argocd app list
   

   Expected output:

   NAME                                    CLUSTER             NAMESPACE            PROJECT  STATUS  HEALTH   SYNCPOLICY  CONDITIONS  REPO                                                         PATH                 TARGET
   argocd/smm-operator-workload-cluster-1  workload-cluster-1  smm-registry-access  default  Synced  Healthy  Auto-Prune  <none>      https://github.com/<github-user>/calisti-gitops-multi-cluster.git  charts/smm-operator  HEAD
   argocd/smm-operator-workload-cluster-2  workload-cluster-2  smm-registry-access  default  Synced  Healthy  Auto-Prune  <none>      https://github.com/<github-user>/calisti-gitops-multi-cluster.git  charts/smm-operator  HEAD
   

5. Check the smm-operator application on the Argo CD Web UI.

   

Deploy the smm-controlplane application

The following steps show you how to deploy the smm-controlplane application as an active-passive deployment. To create and active-active deployment, follow the same steps, there is an optional step that changes the active-passive deployment to active-active. For details, see Deployment models.

Deploy the smm-controlplane application using Kustomize: the active on workload-cluster-1 and the passive on workload-cluster-2. The active cluster receives every component, while the passive cluster only a few required components. This part of the repository will look like this:

└── manifests
    ├── smm-controlplane
    │   ├── base
    │   │   ├── control-plane.yaml
    │   │   ├── cert-manager-namespace.yaml
    │   │   ├── istio-system-namespace.yaml
    │   │   ├── istio-cp-v115x.yaml
    │   │   └── kustomization.yaml
    │   └── overlays
    │       ├── workload-cluster-1
    │       │   ├── control-plane.yaml
    │       │   ├── istio-cp-v115x.yaml
    │       │   └── kustomization.yaml
    │       └── workload-cluster-2
    │           ├── control-plane.yaml
    │           ├── istio-cp-v115x.yaml
    │           └── kustomization.yaml

1. Create the following namespaces files.

   cat > manifests/smm-controlplane/base/cert-manager-namespace.yaml <<EOF
   apiVersion: v1
   kind: Namespace
   metadata:
     annotations:
       argocd.argoproj.io/sync-wave: "1"
     name: cert-manager
   EOF
   

   cat > manifests/smm-controlplane/base/istio-system-namespace.yaml << EOF
   apiVersion: v1
   kind: Namespace
   metadata:
     annotations:
       argocd.argoproj.io/sync-wave: "2"
     name: istio-system
   EOF
   

2. Create the IstioControlPlane file.

   cat > manifests/smm-controlplane/base/istio-cp-v115x.yaml << EOF
   apiVersion: servicemesh.cisco.com/v1alpha1
   kind: IstioControlPlane
   metadata:
     annotations:
       argocd.argoproj.io/sync-wave: "5"
     name: cp-v115x
     namespace: istio-system
   spec:
     containerImageConfiguration:
       imagePullPolicy: Always
       imagePullSecrets:
       - name: smm-pull-secret
     distribution: cisco
     istiod:
       deployment:
         env:
         - name: ISTIO_MULTIROOT_MESH
           value: "true"
         image: registry.eticloud.io/smm/istio-pilot:v1.15.3-bzc.0
     k8sResourceOverlays:
     - groupVersionKind:
         group: apps
         kind: Deployment
         version: v1
       objectKey:
         name: istiod-cp-v115x
         namespace: istio-system
       patches:
       - path: /spec/template/spec/containers/0/args/-
         type: replace
         value: --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256
     meshConfig:
       defaultConfig:
         envoyAccessLogService:
           address: smm-als.smm-system.svc.cluster.local:50600
           tcpKeepalive:
             interval: 10s
             probes: 3
             time: 10s
           tlsSettings:
             mode: ISTIO_MUTUAL
         holdApplicationUntilProxyStarts: true
         proxyMetadata:
           ISTIO_META_ALS_ENABLED: "true"
           PROXY_CONFIG_XDS_AGENT: "true"
         tracing:
           tlsSettings:
             mode: ISTIO_MUTUAL
           zipkin:
             address: smm-zipkin.smm-system.svc.cluster.local:59411
       enableEnvoyAccessLogService: true
       enableTracing: true
     meshExpansion:
       enabled: true
       gateway:
         deployment:
           podMetadata:
             labels:
               app: istio-meshexpansion-gateway
               istio: meshexpansiongateway
         service:
           ports:
           - name: tcp-smm-als-tls
             port: 50600
             protocol: TCP
             targetPort: 50600
           - name: tcp-smm-zipkin-tls
             port: 59411
             protocol: TCP
             targetPort: 59411
     meshID: mesh1
     mode: ACTIVE
     networkName: network1
     proxy:
       image: registry.eticloud.io/smm/istio-proxyv2:v1.15.3-bzc.0
     proxyInit:
       cni:
         daemonset:
           image: registry.eticloud.io/smm/istio-install-cni:v1.15.3-bzc.0
       image: registry.eticloud.io/smm/istio-proxyv2:v1.15.3-bzc.0
     sidecarInjector:
       deployment:
         image: registry.eticloud.io/smm/istio-sidecar-injector:v1.15.3-bzc.0
     version: 1.15.3
   EOF
   

3. Create the kustomization.yaml file.

   cat > manifests/smm-controlplane/base/kustomization.yaml <<EOF
   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   metadata:
     name: cluster-secrets
   
   
   resources:
   - cert-manager-namespace.yaml
   - istio-system-namespace.yaml
   - istio-cp-v115x.yaml
   - control-plane.yaml
   EOF
   

4. Create the manifests/smm-controlplane/base/control-plane.yaml file. You don’t need to set the CLUSTER-NAME here, you will set it with the overlays customization.

   cat > manifests/smm-controlplane/base/control-plane.yaml <<EOF
   apiVersion: smm.cisco.com/v1alpha1
   kind: ControlPlane
   metadata:
     annotations:
       argocd.argoproj.io/sync-wave: "10"
     name: smm
   spec:
     certManager:
       namespace: cert-manager
     clusterName: CLUSTER-NAME
     clusterRegistry:
       enabled: true
       namespace: cluster-registry
     log: {}
     meshManager:
       enabled: true
       istio:
         enabled: true
         istioCRRef:
           name: cp-v115x
           namespace: istio-system
         operators:
           namespace: smm-system
       namespace: smm-system
     nodeExporter:
       enabled: true
       namespace: smm-system
       psp:
         enabled: false
       rbac:
         enabled: true
     oneEye: {}
     registryAccess:
       enabled: true
       imagePullSecretsController: {}
       namespace: smm-registry-access
       pullSecrets:
       - name: smm-registry.eticloud.io-pull-secret
         namespace: smm-registry-access
     repositoryOverride:
       host: registry.eticloud.io
       prefix: smm
     role: active
     smm:
       als:
         enabled: true
         log: {}
       application:
         enabled: true
         log: {}
       auth:
         mode: impersonation
       certManager:
         enabled: true
       enabled: true
       federationGateway:
         enabled: true
         name: smm
         service:
           enabled: true
           name: smm-federation-gateway
           port: 80
       federationGatewayOperator:
         enabled: true
       impersonation:
         enabled: true
       istio:
         revision: cp-v115x.istio-system
       leo:
         enabled: true
         log: {}
       log: {}
       namespace: smm-system
       prometheus:
         enabled: true
         replicas: 1
       prometheusOperator: {}
       releaseName: smm
       role: active
       sdm:
         enabled: false
       sre:
         enabled: true
       useIstioResources: true
   EOF
   

5. Create the kustomization.yaml file for workload-cluster-1.

   cat > manifests/smm-controlplane/overlays/workload-cluster-1/kustomization.yaml <<EOF
   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   
   bases:
     - ../../base
   
   patches:
     - istio-cp-v115x.yaml
     - control-plane.yaml
   EOF
   

6. Set the clusterName by overriding some settings coming from the base configuration. Create the following files.

   cat > manifests/smm-controlplane/overlays/workload-cluster-1/control-plane.yaml <<EOF
   apiVersion: smm.cisco.com/v1alpha1
   kind: ControlPlane
   metadata:
     name: smm
   spec:
     clusterName: workload-cluster-1
     certManager:
       enabled: true
     smm:
       exposeDashboard:
         meshGateway:
           enabled: true
       auth:
         forceUnsecureCookies: true
         mode: anonymous
   EOF
   

   cat > manifests/smm-controlplane/overlays/workload-cluster-1/istio-cp-v115x.yaml <<EOF
   apiVersion: servicemesh.cisco.com/v1alpha1
   kind: IstioControlPlane
   metadata:
     annotations:
       argocd.argoproj.io/sync-wave: "5"
     name: cp-v115x
     namespace: istio-system
   spec:
     meshID: mesh1
     mode: ACTIVE
     networkName: network1
   EOF
   

7. Create the kustomization.yaml file for workload-cluster-2.

   cat > manifests/smm-controlplane/overlays/workload-cluster-2/kustomization.yaml <<EOF
   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   
   bases:
     - ../../base
   
   patches:
     - istio-cp-v115x.yaml
     - control-plane.yaml
   EOF
   

8. Create the following files for workload-cluster-2. This sets the clusterName, and also overrides some settings of the base configuration.

   cat > manifests/smm-controlplane/overlays/workload-cluster-2/control-plane.yaml <<EOF
   apiVersion: smm.cisco.com/v1alpha1
   kind: ControlPlane
   metadata:
     name: smm
   spec:
     clusterName: workload-cluster-2
     role: passive
     smm:
       als:
         enabled: true
         log: {}
       application:
         enabled: false
         log: {}
       auth:
         mode: impersonation
       certManager:
         enabled: false
       enabled: true
       federationGateway:
         enabled: false
         name: smm
         service:
           enabled: true
           name: smm-federation-gateway
           port: 80
       federationGatewayOperator:
         enabled: true
       grafana:
         enabled: false
       impersonation:
         enabled: true
       istio:
         revision: cp-v115x.istio-system
       kubestatemetrics:
         enabled: true
       leo:
         enabled: false
         log: {}
       log: {}
       namespace: smm-system
       prometheus:
         enabled: true
         replicas: 1
         retentionTime: 8h
       prometheusOperator: {}
       releaseName: smm
       role: passive
       sdm:
         enabled: false
       sre:
         enabled: false
       tracing:
         enabled: true
       useIstioResources: false
       web:
         enabled: false
   EOF
   

   cat > manifests/smm-controlplane/overlays/workload-cluster-2/istio-cp-v115x.yaml <<EOF
   apiVersion: servicemesh.cisco.com/v1alpha1
   kind: IstioControlPlane
   metadata:
     annotations:
       argocd.argoproj.io/sync-wave: "5"
     name: cp-v115x
     namespace: istio-system
   spec:
     meshID: mesh1
     mode: PASSIVE
     networkName: workload-cluster-2
   EOF
   

9. (Optional) If you want to change your active-passive deployment to active-active, complete this step. Otherwise, continue with the Commit deployment step.

   1. Run the following commands to modify the control planes of workload-cluster-2.

      cat > manifests/smm-controlplane/overlays/workload-cluster-2/control-plane.yaml <<EOF
      apiVersion: smm.cisco.com/v1alpha1
      kind: ControlPlane
      metadata:
        name: smm
      spec:
        clusterName: workload-cluster-2
        role: active
      EOF
      

      cat > manifests/smm-controlplane/overlays/workload-cluster-2/istio-cp-v115x.yaml <<EOF
      apiVersion: servicemesh.cisco.com/v1alpha1
      kind: IstioControlPlane
      metadata:
        annotations:
          argocd.argoproj.io/sync-wave: "5"
        name: cp-v115x
        namespace: istio-system
      spec:
        meshID: mesh1
        mode: ACTIVE
        networkName: network1
      EOF
      

   2. Create the smm-controlplane’s Argo CD ApplicationSet CR.

      cat > apps/smm-controlplane/app-set.yaml <<EOF
      apiVersion: argoproj.io/v1alpha1
      kind: ApplicationSet
      metadata:
        name: smm-cp-appset
        namespace: argocd
      spec:
        generators:
        - list:
            elements:
            - cluster: "${WORKLOAD_CLUSTER_1_CONTEXT}"
              path: "${WORKLOAD_CLUSTER_1_CONTEXT}"
            - cluster: "${WORKLOAD_CLUSTER_2_CONTEXT}"
              path: "${WORKLOAD_CLUSTER_2_CONTEXT}"
        template:
          metadata: 
            name: 'smm-cp-{{cluster}}'
            namespace: argocd
          spec:
            project: default
            source:
              repoURL: https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git
              targetRevision: HEAD
              path: manifests/smm-controlplane/overlays/{{path}}
            destination:
              name: '{{cluster}}'
            syncPolicy:
              automated:
                prune: true
                selfHeal: true
              retry:
                limit: 5
                backoff:
                  duration: 5s
                  maxDuration: 3m0s
                  factor: 2
              syncOptions:
                - Validate=false
                - PruneLast=true
                - CreateNamespace=true
                - Replace=true
      EOF
      

10. Commit and push the calisti-gitops repository.

    git add apps/smm-controlplane manifests
    

    git commit -m "add smm-controlplane app"
    

    git push
    

11. Apply the Application manifests.

    kubectl apply -f "apps/smm-controlplane/app-set.yaml"
    

12. Verify that the applications have been added to Argo CD and are healthy.

    argocd app list
    

13. To create trust between workload-cluster-1 and workload-cluster-2, you must exchange the Secret CRs of the clusters. The cluster registry controller helps to form a group of Kubernetes clusters and synchronize any resources across those clusters arbitrarily.

    Create and run the following bash script.

    cat > export-secrets.sh <<EOF
    set -e
    
    kubectl --context workload-cluster-1 get cluster workload-cluster-1 -o yaml | kubectl --context workload-cluster-2 apply -f -
    kubectl --context workload-cluster-1 -n cluster-registry get secrets workload-cluster-1 -o yaml | kubectl --context workload-cluster-2 apply -f -
    
    kubectl --context workload-cluster-2 get cluster workload-cluster-2 -o yaml | kubectl --context workload-cluster-1 apply -f -
    kubectl --context workload-cluster-2 -n cluster-registry get secrets workload-cluster-2 -o yaml | kubectl --context workload-cluster-1 apply -f -
    
    echo "Exporting cluster and secrets CRs successfully."
    EOF
    

    chmod +x export-secrets.sh
    ./export-secrets.sh
    

    Expected output:

    cluster.clusterregistry.k8s.cisco.com/workload-cluster-1 created
    secret/workload-cluster-1 created
    cluster.clusterregistry.k8s.cisco.com/workload-cluster-2 created
    secret/workload-cluster-2 created
    Exporting cluster and secrets CRs successfully.
    

14. Check that all pods are healthy and running in the smm-system namespace on workload-cluster-1 and workload-cluster-2. Note that it takes some time while the ControlPlane operator reconciles the resources.

    For workload-cluster-1:

    kubectl get pods -n smm-system --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_1_CONTEXT}"
    

    Expected output:

    NAME                                               READY   STATUS    RESTARTS        AGE
    istio-operator-v115x-7d77fc549f-fxmtd              2/2     Running   0               8m15s
    mesh-manager-0                                     2/2     Running   0          19m
    prometheus-node-exporter-9xnmj                     1/1     Running   0          17m
    prometheus-node-exporter-bf7g5                     1/1     Running   0          17m
    prometheus-node-exporter-cl69q                     1/1     Running   0          17m
    prometheus-smm-prometheus-0                        4/4     Running   0          18m
    smm-7f4d5d4fff-4dlcp                               2/2     Running   0          18m
    smm-7f4d5d4fff-59k7g                               2/2     Running   0          18m
    smm-als-7cc4bfb998-wjsr6                           2/2     Running   0          18m
    smm-authentication-569484f748-fj5zk                2/2     Running   0          18m
    smm-federation-gateway-6964fb956f-pb5pv            2/2     Running   0          18m
    smm-federation-gateway-operator-6664774695-9tmzj   2/2     Running   0          18m
    smm-grafana-59c54f67f4-9snc5                       3/3     Running   0          18m
    smm-health-75bf4f49c5-z9tqg                        2/2     Running   0          18m
    smm-health-api-7767d4f46-744wn                     2/2     Running   0          18m
    smm-ingressgateway-6ffdfc6d79-jttjz                1/1     Running   0          11m
    smm-ingressgateway-external-8c9bb9445-kjt8h        1/1     Running   0          11m
    smm-kubestatemetrics-86c6f96789-lp576              2/2     Running   0          18m
    smm-leo-67cd7d49b5-gmcvf                           2/2     Running   0          18m
    smm-prometheus-operator-ffbfb8b67-fwj6g            3/3     Running   0          18m
    smm-sre-alert-exporter-6654968479-fthk6            2/2     Running   0          18m
    smm-sre-api-86c9fb7cd7-mq7cm                       2/2     Running   0          18m
    smm-sre-controller-6889685f9-hxxh5                 2/2     Running   0          18m
    smm-tracing-5886d59dd-v8nb8                        2/2     Running   0          18m
    smm-vm-integration-5b89c4f7c9-wz4bt                2/2     Running   0          18m
    smm-web-d5b49c7f6-jgz7b                            3/3     Running   0          18m
    

    For workload-cluster-2:

    kubectl get pods -n smm-system --kubeconfig "${WORKLOAD_CLUSTER_2_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_2_CONTEXT}"
    

    Expected output:

    NAME                                       READY   STATUS    RESTARTS        AGE
    istio-operator-v115x-7d77fc549f-s5wnz      2/2     Running   0               9m5s
    mesh-manager-0                            2/2     Running   0             21m
    prometheus-node-exporter-fzdn4            1/1     Running   0             5m18s
    prometheus-node-exporter-rkbcl            1/1     Running   0             5m18s
    prometheus-node-exporter-x2mwp            1/1     Running   0             5m18s
    prometheus-smm-prometheus-0               3/3     Running   0             5m20s
    smm-ingressgateway-5db7859d45-6d6ns       1/1     Running   0             12m
    smm-kubestatemetrics-86c6f96789-j64q2     2/2     Running   0             19m
    smm-prometheus-operator-ffbfb8b67-zwqn2   3/3     Running   1 (11m ago)   19m
    

15. Check the applications on the Argo CD Web UI.

    

At this point, you have successfully installed smm-operator and workload-cluster-1 and workload-cluster-2. You can open the Service Mesh Manager dashboard to check them, or deploy an application.

Deploy an application

If you want to deploy want to deploy an application into the service mesh, complete the following steps. The examples use the Service Mesh Manager demo application.

The file structure for the demo application looks like this:

.
├── README.md
├── apps
│   ├── demo-app
│   │   └── app-set.yaml
│   └── ...
...manifests
   └── demo-app
        ├── base
        │   ├── demo-app-namespace.yaml
        │   ├── demo-app.yaml
        │   └── kustomization.yaml
        └── overlays
            ├── workload-cluster-1
            │   ├── demo-app.yaml
            │   └── kustomization.yaml
            └── workload-cluster-2
                ├── demo-app.yaml
                └── kustomization.yaml
...

1. Create the application manifest files.

   cat > manifests/demo-app/base/kustomization.yaml <<EOF
   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   metadata:
     name: demo-app
   
   resources:
   - demo-app-namespace.yaml
   - demo-app.yaml
   EOF
   

   cat > manifests/demo-app/base/demo-app-namespace.yaml <<EOF
   apiVersion: v1
   kind: Namespace
   metadata:
     labels:
       app.kubernetes.io/instance: smm-demo
       app.kubernetes.io/name: smm-demo
       app.kubernetes.io/part-of: smm-demo
       app.kubernetes.io/version: 0.1.4
       istio.io/rev: cp-v115x.istio-system
     name: smm-demo
   EOF
   

   cat > manifests/demo-app/base/demo-app.yaml <<EOF
   apiVersion: smm.cisco.com/v1alpha1
   kind: DemoApplication
   metadata:
     name: smm-demo
     namespace: smm-demo
   spec:
     autoscaling:
       enabled: true
     controlPlaneRef:
       name: smm
   EOF
   

   cat > manifests/demo-app/overlays/workload-cluster-1/kustomization.yaml <<EOF
   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   
   bases:
     - ../../base
   
   patches:
     - demo-app.yaml
   EOF
   

   cat > manifests/demo-app/overlays/workload-cluster-1/demo-app.yaml <<EOF
   apiVersion: smm.cisco.com/v1alpha1
   kind: DemoApplication
   metadata:
     name: smm-demo
     namespace: smm-demo
   spec:
     autoscaling:
       enabled: true
     controlPlaneRef:
       name: smm
     deployIstioResources: true
     deploySLOResources: true
     enabled: true
     enabledComponents:
     - frontpage
     - catalog
     - bookings
     - postgresql
     istio:
       revision: cp-v115x.istio-system
     load:
       enabled: true
       maxRPS: 30
       minRPS: 10
       swingPeriod: 1380000000000
     replicas: 1
     resources:
       limits:
         cpu: "2"
         memory: 192Mi
       requests:
         cpu: 40m
         memory: 64Mi
   EOF
   

   cat > manifests/demo-app/overlays/workload-cluster-2/kustomization.yaml <<EOF
   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   
   bases:
     - ../../base
   
   patches:
     - demo-app.yaml
   EOF
   

   cat > manifests/demo-app/overlays/workload-cluster-2/demo-app.yaml <<EOF
   apiVersion: smm.cisco.com/v1alpha1
   kind: DemoApplication
   metadata:
     name: smm-demo
     namespace: smm-demo
   spec:
     autoscaling:
       enabled: true
     controlPlaneRef:
       name: smm
     deployIstioResources: false
     deploySLOResources: false
     enabled: true
     enabledComponents:
     - movies
     - payments
     - notifications
     - analytics
     - database
     - mysql
     istio:
       revision: cp-v115x.istio-system
     replicas: 1
     resources:
       limits:
         cpu: "2"
         memory: 192Mi
       requests:
         cpu: 40m
         memory: 64Mi
   EOF
   

2. Create the Demo application ApplicationSet.

   cat > apps/demo-app/app-set.yaml <<EOF
   apiVersion: argoproj.io/v1alpha1
   kind: ApplicationSet
   metadata:
     name: demo-app-appset
     namespace: argocd
   spec:
     generators:
     - list:
         elements:
         - cluster: "${WORKLOAD_CLUSTER_1_CONTEXT}"
           path: "${WORKLOAD_CLUSTER_1_CONTEXT}"
         - cluster: "${WORKLOAD_CLUSTER_2_CONTEXT}"
           path: "${WORKLOAD_CLUSTER_2_CONTEXT}"
     template:
       metadata: 
         name: 'demo-app-{{cluster}}'
         namespace: argocd
       spec:
         project: default
         source:
           repoURL: https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git
           targetRevision: HEAD
           path: manifests/demo-app/overlays/{{path}}
         destination:
           name: '{{cluster}}'
         syncPolicy:
           automated:
             prune: true
             selfHeal: true
           retry:
             limit: 5
             backoff:
               duration: 5s
               maxDuration: 3m0s
               factor: 2
           syncOptions:
             - Validate=false
             - PruneLast=true
             - CreateNamespace=true
             - Replace=true
   EOF
   

3. Commit and push the calisti-gitops repository.

   git add apps/demo-app manifests
   git commit -m "add demo application"
   git push origin
   

4. Deploy the demo application on the clusters.

   kubectl apply -f apps/demo-app/app-set.yaml
   

5. Wait until all the pods in the smm-demo namespace are up and running.

   kubectl get pods -n smm-demo --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_1_CONTEXT}"
   

   Expected output:

   NAME                           READY   STATUS    RESTARTS   AGE
   bombardier-5f59948978-zx99c    2/2     Running   0          3m21s
   bookings-v1-68dd865855-fdcxk   2/2     Running   0          3m21s
   catalog-v1-6d564bbcb8-qmhbx    2/2     Running   0          3m21s
   frontpage-v1-b4686759b-fhfmv   2/2     Running   0          3m21s
   postgresql-7cf55cd596-grs46    2/2     Running   0          3m21s
   

   kubectl get pods -n smm-demo --kubeconfig "${WORKLOAD_CLUSTER_2_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_2_CONTEXT}"
   

   Expected output:

   NAME                                READY   STATUS    RESTARTS   AGE
   analytics-v1-799d668f84-p4nkk       2/2     Running   0          3m58s
   database-v1-6896cd4b59-9xxgg        2/2     Running   0          3m58s
   movies-v1-9594fff5f-8hv9l           2/2     Running   0          3m58s
   movies-v2-5559c5567c-2279n          2/2     Running   0          3m58s
   movies-v3-649b99d977-nkdxc          2/2     Running   0          3m58s
   mysql-669466cc8d-bs4s9              2/2     Running   0          3m58s
   notifications-v1-79bc79c89b-4bbss   2/2     Running   0          3m58s
   payments-v1-547884bfdf-dg2dm        2/2     Running   0          3m58s
   

6. Check the applications on the Argo CD web UI.

   

7. Open the Service Mesh Manager web interface, select MENU > TOPOLOGY, then select the smm-demo namespace.