Install SMM - GitOps - multi-cluster

This guide details how to set up a multi-cluster Service Mesh Manager scenario in a GitOps environment for Service Mesh Manager using Argo CD. The same principles can be used for other tools as well.

CAUTION:

Do not push the secrets directly into the git repository, especially when it is a public repository. Argo CD provides solutions to keep secrets safe.

Architecture

Service Mesh Manager supports multiple mesh topologies, so you can use the one that best fits your use cases. In multi-cluster configurations it provides automatic locality load-balancing.

The high level architecture for Argo CD with a multi-cluster Service Mesh Manager setup consists of the following components:

  • A git repository that stores the various charts and manifests,
  • a management cluster that runs the Argo CD server, and
  • the Service Mesh Manager clusters managed by Argo CD.

Multi-cluster GitOps architecture Multi-cluster GitOps architecture

Deployment models

When deploying Service Mesh Manager in a multi-cluster scenario you can deploy Service Mesh Manager in an active-passive model. For details on Service Mesh Manager clusters and their relationship to Istio clusters, see Istio clusters and SMM clusters.

Prerequisites

  • A free registration for the Service Mesh Manager download page
  • A Kubernetes cluster to deploy Argo CD on (called management-cluster in the examples).
  • Two Kubernetes clusters to deploy Service Mesh Manager on (called workload-cluster-1 and workload-cluster-2 in the examples).

CAUTION:

Supported providers and Kubernetes versions

The cluster must run a Kubernetes version that Service Mesh Manager supports: Kubernetes 1.21, 1.22, 1.23, 1.24.

Service Mesh Manager is tested and known to work on the following Kubernetes providers:

  • Amazon Elastic Kubernetes Service (Amazon EKS)
  • Google Kubernetes Engine (GKE)
  • Azure Kubernetes Service (AKS)
  • On-premises installation of stock Kubernetes with load balancer support (and optionally PVCs for persistence)

Resource requirements

Make sure that your Kubernetes cluster has sufficient resources. The default installation (with Service Mesh Manager and demo application) requires the following amount of resources on the cluster:

Only Service Mesh Manager Service Mesh Manager and Streaming Data Manager
CPU - 12 vCPU in total
- 4 vCPU available for allocation per worker node (If you are testing on a cluster at a cloud provider, use nodes that have at least 4 CPUs, for example, c5.xlarge on AWS.)		 - 24 vCPU in total
- 4 vCPU available for allocation per worker node (If you are testing on a cluster at a cloud provider, use nodes that have at least 4 CPUs, for example, c5.xlarge on AWS.)
Memory - 16 GB in total
- 2 GB available for allocation per worker node		 - 36 GB in total
- 2 GB available for allocation per worker node
Storage 12 GB of ephemeral storage on the Kubernetes worker nodes (for Traces and Metrics) 12 GB of ephemeral storage on the Kubernetes worker nodes (for Traces and Metrics)

These minimum requirements need to be available for allocation within your cluster, in addition to the requirements of any other loads running in your cluster (for example, DaemonSets and Kubernetes node-agents). If Kubernetes cannot allocate sufficient resources to Service Mesh Manager, some pods will remain in Pending state, and Service Mesh Manager will not function properly.

Enabling additional features, such as High Availability increases this value.

The default installation, when enough headroom is available in the cluster, should be able to support at least 150 running Pods with the same amount of Services. For setting up Service Mesh Manager for bigger workloads, see scaling Service Mesh Manager.

Install Argo CD

Complete the following steps to install Argo CD on the management cluster.

Set up the environment

  1. Set the KUBECONFIG location and context name for the management-cluster cluster.

    MANAGEMENT_CLUSTER_KUBECONFIG=management_cluster_kubeconfig.yaml
MANAGEMENT_CLUSTER_CONTEXT=management-cluster
kubectl config --kubeconfig "${MANAGEMENT_CLUSTER_KUBECONFIG}" get-contexts "${MANAGEMENT_CLUSTER_CONTEXT}"

    Expected output:

    CURRENT   NAME                 CLUSTER              AUTHINFO   NAMESPACE
*         management-cluster   management-cluster

  2. Set the KUBECONFIG location and context name for the workload-cluster-1 cluster.

    WORKLOAD_CLUSTER_1_KUBECONFIG=workload_cluster_1_kubeconfig.yaml
WORKLOAD_CLUSTER_1_CONTEXT=workload-cluster-1
kubectl config --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" get-contexts "${WORKLOAD_CLUSTER_1_CONTEXT}"

    Expected output:

    CURRENT   NAME                 CLUSTER              AUTHINFO                                          NAMESPACE
*         workload-cluster-1   workload-cluster-1

    Repeat this step for any additional workload clusters you want to use.

  3. Make sure the management-cluster Kubernetes context is the current context.

    kubectl config --kubeconfig "${MANAGEMENT_CLUSTER_KUBECONFIG}" use-context "${MANAGEMENT_CLUSTER_CONTEXT}"

    Expected output:

    Switched to context "management-cluster".

Install Argo CD Server

  1. Install the Argo CD Server. Run the following commands.

    kubectl create namespace argocd

    Expected output:

    namespace/argocd created

    kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

    customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/applicationsets.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/appprojects.argoproj.io created
serviceaccount/argocd-application-controller created
serviceaccount/argocd-applicationset-controller created
serviceaccount/argocd-dex-server created
serviceaccount/argocd-notifications-controller created
serviceaccount/argocd-redis created
serviceaccount/argocd-repo-server created
serviceaccount/argocd-server created
role.rbac.authorization.k8s.io/argocd-application-controller created
role.rbac.authorization.k8s.io/argocd-applicationset-controller created
role.rbac.authorization.k8s.io/argocd-dex-server created
role.rbac.authorization.k8s.io/argocd-notifications-controller created
role.rbac.authorization.k8s.io/argocd-server created
clusterrole.rbac.authorization.k8s.io/argocd-application-controller created
clusterrole.rbac.authorization.k8s.io/argocd-server created
rolebinding.rbac.authorization.k8s.io/argocd-application-controller created
rolebinding.rbac.authorization.k8s.io/argocd-applicationset-controller created
rolebinding.rbac.authorization.k8s.io/argocd-dex-server created
rolebinding.rbac.authorization.k8s.io/argocd-notifications-controller created
rolebinding.rbac.authorization.k8s.io/argocd-redis created
rolebinding.rbac.authorization.k8s.io/argocd-server created
clusterrolebinding.rbac.authorization.k8s.io/argocd-application-controller created
clusterrolebinding.rbac.authorization.k8s.io/argocd-server created
configmap/argocd-cm created
configmap/argocd-cmd-params-cm created
configmap/argocd-gpg-keys-cm created
configmap/argocd-notifications-cm created
configmap/argocd-rbac-cm created
configmap/argocd-ssh-known-hosts-cm created
configmap/argocd-tls-certs-cm created
secret/argocd-notifications-secret created
secret/argocd-secret created
service/argocd-applicationset-controller created
service/argocd-dex-server created
service/argocd-metrics created
service/argocd-notifications-controller-metrics created
service/argocd-redis created
service/argocd-repo-server created
service/argocd-server created
service/argocd-server-metrics created
deployment.apps/argocd-applicationset-controller created
deployment.apps/argocd-dex-server created
deployment.apps/argocd-notifications-controller created
deployment.apps/argocd-redis created
deployment.apps/argocd-repo-server created
deployment.apps/argocd-server created
statefulset.apps/argocd-application-controller created
networkpolicy.networking.k8s.io/argocd-application-controller-network-policy created
networkpolicy.networking.k8s.io/argocd-applicationset-controller-network-policy created
networkpolicy.networking.k8s.io/argocd-dex-server-network-policy created
networkpolicy.networking.k8s.io/argocd-notifications-controller-network-policy created
networkpolicy.networking.k8s.io/argocd-redis-network-policy created
networkpolicy.networking.k8s.io/argocd-repo-server-network-policy created
networkpolicy.networking.k8s.io/argocd-server-network-policy created

  2. Wait until the installation is complete, then check that the Argo CD pods are up and running.

    kubectl get pods -n argocd

    The output should be similar to:

    NAME                                                    READY   STATUS    RESTARTS   AGE
pod/argocd-application-controller-0                     1/1     Running   0          7h59m
pod/argocd-applicationset-controller-78b8b554f9-pgwbl   1/1     Running   0          7h59m
pod/argocd-dex-server-6bbc85c688-8p7zf                  1/1     Running   0          16h
pod/argocd-notifications-controller-75847756c5-dbbm5    1/1     Running   0          16h
pod/argocd-redis-f4cdbff57-wcpxh                        1/1     Running   0          7h59m
pod/argocd-repo-server-d5c7f7ffb-c8962                  1/1     Running   0          7h59m
pod/argocd-server-76497676b-pnvf4                       1/1     Running   0          7h59m

  3. For the Argo CD UI, set the argocd-server service type to LoadBalancer.

    kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

    Expected output:

    service/argocd-server patched

  4. Patch the App of Apps health check in Argo CD configuration to ignore diffs of controller/operator managed fields. For details about this patch, see the Argo CD documentation sections Resource Health and Diffing Customization.

    Apply the new Argo CD health check configurations:

    kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  # App of app health check
  resource.customizations.health.argoproj.io_Application: |
    hs = {}
    hs.status = "Progressing"
    hs.message = ""
    if obj.status ~= nil then
      if obj.status.health ~= nil then
        hs.status = obj.status.health.status
        if obj.status.health.message ~= nil then
          hs.message = obj.status.health.message
        end
      end
    end
    return hs    
  # Ignoring RBAC changes made by AggregateRoles
  resource.compareoptions: |
    # disables status field diffing in specified resource types
    ignoreAggregatedRoles: true

    # disables status field diffing in specified resource types
    # 'crd' - CustomResourceDefinition-s (default)
    # 'all' - all resources
    # 'none' - disabled
    ignoreResourceStatusField: all    
EOF

    Expected output:

    configmap/argocd-cm configured

  5. Get the initial password for the admin user.

    kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

    Expected output:

    argocd-admin-password

  6. Check the external-ip-or-hostname address of the argocd-server service.

    kubectl get service -n argocd argocd-server

    The output should be similar to:

    NAME                                      TYPE           CLUSTER-IP      EXTERNAL-IP               PORT(S)                      AGE
argocd-server                             LoadBalancer   10.108.14.130   external-ip-or-hostname   80:31306/TCP,443:30063/TCP   7d13h

  7. Open the https://external-ip-or-hostname URL and log in to the Argo CD server using the password received in the previous step.

    # Exactly one of hostname or IP will be available and used for the remote URL.
open https://$(kubectl get service -n argocd argocd-server -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{.status.loadBalancer.ingress[0].ip}')

Install Argo CD CLI

  1. Install Argo CD CLI on your computer. For details, see the Argo CD documentation.

  2. Log in with the CLI:

    # Exactly one of hostname or IP will be available and used for the remote URL.
argocd login $(kubectl get service -n argocd argocd-server -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{.status.loadBalancer.ingress[0].ip}') --insecure --username admin --password $(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)

    Expected output:

    'admin:login' logged in successfully

For more details about Argo CD installation, see the Argo CD getting started guide.

Register clusters

  1. Register the clusters that will run Service Mesh Manager in Argo CD. In this example, register workload-cluster-1 and workload-cluster-2 using one of the following methods.

    • Register the cluster from the command line by running:

      argocd cluster add --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" "${WORKLOAD_CLUSTER_1_CONTEXT}"

      Expected output:

      WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `workload-cluster-1` with full cluster level privileges. Do you want to continue [y/N]? y
INFO[0005] ServiceAccount "argocd-manager" created in namespace "kube-system"
INFO[0005] ClusterRole "argocd-manager-role" created
INFO[0005] ClusterRoleBinding "argocd-manager-role-binding" created
INFO[0011] Created bearer token secret for ServiceAccount "argocd-manager"
Cluster 'https://workload-cluster-1-ip-or-hostname' added

      argocd cluster add --kubeconfig "${WORKLOAD_CLUSTER_2_KUBECONFIG}" "${WORKLOAD_CLUSTER_2_CONTEXT}"

      Expected output:

      WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `workload-cluster-2` with full cluster level privileges. Do you want to continue [y/N]? y
INFO[0005] ServiceAccount "argocd-manager" created in namespace "kube-system"
INFO[0005] ClusterRole "argocd-manager-role" created
INFO[0005] ClusterRoleBinding "argocd-manager-role-binding" created
INFO[0011] Created bearer token secret for ServiceAccount "argocd-manager"
Cluster 'https://workload-cluster-2-ip-or-hostname' added

    • Alternatively, you can register clusters declaratively as Kubernetes secrets. Modify the following command for your environment and apply it. For details, see the Argo CD documentation.

      WORKLOAD_CLUSTER_1_IP="https://workload-cluster-1-IP" ARGOCD_BEARER_TOKEN="authentication-token" ARGOCD_CA_B64="base64 encoded certificate" ; kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: workload-cluster-1-secret
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: workload-cluster-1
  server: "${WORKLOAD_CLUSTER_1_IP}"
  config: |
    {
      "bearerToken": "${ARGOCD_BEARER_TOKEN}",
      "tlsClientConfig": {
        "insecure": false,
        "caData": "${ARGOCD_CA_B64}"
      }
    }    
EOF

      WORKLOAD_CLUSTER_2_IP="https://workload-cluster-2-IP" ARGOCD_BEARER_TOKEN="authentication-token" ARGOCD_CA_B64="base64 encoded certificate" ; kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: workload-cluster-2-secret
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: workload-cluster-2
  server: "${WORKLOAD_CLUSTER_2_IP}"
  config: |
    {
      "bearerToken": "${ARGOCD_BEARER_TOKEN}",
      "tlsClientConfig": {
        "insecure": false,
        "caData": "${ARGOCD_CA_B64}"
      }
    }    
EOF

  2. Make sure that the cluster is registered in Argo CD by running the following command:

    argocd cluster list

    The output should be similar to:

    SERVER                                      NAME                VERSION  STATUS   MESSAGE                                                  PROJECT
https://kubernetes.default.svc              in-cluster                   Unknown  Cluster has no applications and is not being monitored.
https://workload-cluster-1-ip-or-hostname   workload-cluster-1           Unknown  Cluster has no applications and is not being monitored.
https://workload-cluster-2-ip-or-hostname   workload-cluster-2           Unknown  Cluster has no applications and is not being monitored.

Prepare Git repository

  1. Create an empty repository called smm-gitops on GitHub (or another provider that Argo CD supports) and initialize it with a README.md file so that you can clone the repository. Because Service Mesh Manager credentials will be stored in this repository, make it a private repository.

    GITHUB_ID="github-id"
GITHUB_REPOSITORY_NAME="calisti-gitops"

  2. Obtain a personal access token to the repository (on GitHub, see Creating a personal access token), that has the following permissions:

    • admin:org_hook
    • admin:repo_hook
    • read:org
    • read:public_key
    • repo

  3. Log in with your personal access token with git.

    export GH_TOKEN="github-personal-access-token" # Note: this environment variable needs to be exported so the `git` binary is going to use it automatically for authentication.

  4. Clone the repository into your local workspace, for example:

    git clone "https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git"

    Expected output:

    Cloning into 'calisti-gitops'...
remote: Enumerating objects: 144, done.
remote: Counting objects: 100% (144/144), done.
remote: Compressing objects: 100% (93/93), done.
remote: Total 144 (delta 53), reused 135 (delta 47), pack-reused 0
Receiving objects: 100% (144/144), 320.08 KiB | 746.00 KiB/s, done.
Resolving deltas: 100% (53/53), done.

  5. Add the repository to Argo CD by running the following command. Alternatively, you can add it on Argo CD Web UI.

    argocd repo add "https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git" --name "${GITHUB_REPOSITORY_NAME}" --username "${GITHUB_ID}" --password "${GH_TOKEN}"

    Expected output:

    Repository 'https://github.com/github-id/calisti-gitops.git' added

  6. Verify that the repository is connected by running:

    argocd repo list

    In the output, Status should be Successful:

    TYPE  NAME            REPO                                             INSECURE  OCI    LFS    CREDS  STATUS      MESSAGE  PROJECT
git   calisti-gitops  https://github.com/github-id/calisti-gitops.git  false     false  false  true   Successful

  7. Change into the directory of the cloned repository (for example, smm-gitops) and create the following directories.

    cd "${GITHUB_REPOSITORY_NAME}"

    mkdir -p apps/smm-controlplane apps/smm-operator apps/demo-app charts manifests/smm-controlplane/base manifests/smm-controlplane/overlays/workload-cluster-1 manifests/smm-controlplane/overlays/workload-cluster-2 manifests/demo-app/base manifests/demo-app/overlays/workload-cluster-1 manifests/demo-app/overlays/workload-cluster-2

    The final structure of the repository will look like this:

    .
├── README.md
├── apps
│   ├── smm-controlplane
│   │   └── app-set.yaml
│   ├── smm-operator
│   │   └── app-set.yaml
│   └── demo-app
│       └── app-set.yaml
├── charts
│   └── smm-operator
│       ├── Chart.yaml
│       └── ...
├── export-secrets.sh
└── manifests
   ├── smm-controlplane
   │   ├── base
   │   │   ├── control-plane.yaml
   │   │   ├── cert-manager-namespace.yaml
   │   │   ├── istio-system-namespace.yaml
   │   │   ├── istio-cp-v115x.yaml
   │   │   └── kustomization.yaml
   │   └── overlays
   │       ├── workload-cluster-1
   │       │   ├── control-plane.yaml
   │       │   ├── istio-cp-v115x.yaml
   │       │   └── kustomization.yaml
   │       └── workload-cluster-2
   │           ├── control-plane.yaml
   │           ├── istio-cp-v115x.yaml
   │           └── kustomization.yaml
   └── demo-app
       ├── base
       │   ├── demo-app-namespace.yaml
       │   ├── demo-app.yaml
       │   └── kustomization.yaml
       └── overlays
           ├── workload-cluster-1
           │   ├── demo-app.yaml
           │   └── kustomization.yaml
           └── workload-cluster-2
               ├── demo-app.yaml
               └── kustomization.yaml
    • The apps folder contains the Argo CD Application of the smm-operator, the smm-controlplane, and the demo-app.
    • The charts folder contains the Helm chart of the smm-operator.
    • The manifests/demo-app folder contains the manifest files of the demo application that represents your business application.
    • The manifests/smm-controlplane folder contains the manifest files of the SMM ControlPlane.

Prepare the helm charts

  1. You need an active Service Mesh Manager registration to download the Service Mesh Manager charts and images. You can sign up for free, or obtain Enterprise credentials on the official Cisco Service Mesh Manager page. After registration, you can obtain your username and password from the Download Center. Set them as environment variables.

    CALISTI_USERNAME="<your-calisti-username>"

    CALISTI_PASSWORD="<your-calisti-password>"

  2. Download the smm-operator chart from registry.eticloud.io into the charts directory of your Service Mesh Manager GitOps repository and extract it. Run the following commands:

    export HELM_EXPERIMENTAL_OCI=1 # Needed prior to Helm version 3.8.0

echo "${CALISTI_PASSWORD}" | helm registry login registry.eticloud.io -u "${CALISTI_USERNAME}" --password-stdin

    Expected output:

    Login Succeeded

    helm pull oci://registry.eticloud.io/smm-charts/smm-operator --destination ./charts/ --untar --version 1.11.0

    Expected output:

    Pulled: registry.eticloud.io/smm-charts/smm-operator:latest-stable-version
Digest: sha256:someshadigest

Deploy Service Mesh Manager

Deploy the smm-operator application set

Complete the following steps to deploy the smm-operator chart using Argo CD.

  1. Create the smm-operator’s Argo CD ApplicationSet CR. Argo CD ApplicationSet is perfect for deploying the same application on to different clusters. You can use list generators with cluster data in the ApplicationSet.

    Before running the following command, edit it if needed:

    • If you are not using a GitHub repository, set the repoURL field to your repository.
    • Set the value of the PUBLIC_API_SERVER_ENDPOINT_ADDRESS variable to the public API endpoint of your cluster. Some managed Kubernetes solutions of public cloud providers have different API Server endpoints for internal and public access.
    PUBLIC_API_SERVER_ENDPOINT_ADDRESS_1="" PUBLIC_API_SERVER_ENDPOINT_ADDRESS_2="" ;
cat > "apps/smm-operator/app-set.yaml" <<EOF
# apps/smm-operator/app-set.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: smm-operator-appset
  namespace: argocd
spec:
  generators:
  - list:
      elements:
      - cluster: "${WORKLOAD_CLUSTER_1_CONTEXT}"
        apiServerEndpointAddress: "${PUBLIC_API_SERVER_ENDPOINT_ADDRESS_1}"
      - cluster: "${WORKLOAD_CLUSTER_2_CONTEXT}"
        apiServerEndpointAddress: "${PUBLIC_API_SERVER_ENDPOINT_ADDRESS_2}"
  template:
    metadata: 
      name: 'smm-operator-{{cluster}}'
      namespace: argocd
    spec:
      project: default
      source:
        repoURL: https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git
        targetRevision: HEAD
        path: charts/smm-operator
        helm:
          parameters:
          - name: "global.ecr.enabled"
            value: 'false'
          - name: "global.basicAuth.username"
            value: "${CALISTI_USERNAME}"
          - name: "global.basicAuth.password"
            value: "${CALISTI_PASSWORD}"
          - name: "apiServerEndpointAddress"
            value: '{{apiServerEndpointAddress}}'
      destination:
        namespace: smm-registry-access
        name: '{{cluster}}'
      ignoreDifferences:
      - kind: ValidatingWebhookConfiguration
        group: admissionregistration.k8s.io
        jsonPointers:
        - /webhooks
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        retry:
          limit: 5
          backoff:
            duration: 5s
            maxDuration: 3m0s
            factor: 2
        syncOptions:
          - Validate=false
          - PruneLast=true
          - CreateNamespace=true
          - Replace=true
EOF

  2. Commit and push the calisti-gitops repository.

    git add apps/smm-operator charts/smm-operator

    git commit -m "add smm-operator app"

    [main d4f6809] add smm-operator app
35 files changed, 80310 insertions(+)
create mode 100644 apps/smm-operator/app-set.yaml
create mode 100644 charts/smm-operator/.helmignore
create mode 100644 charts/smm-operator/Chart.yaml
create mode 100644 charts/smm-operator/README.md
create mode 100644 charts/smm-operator/crds/clusterfeature-crd.yaml
create mode 100644 charts/smm-operator/crds/clusters-crd.yaml
create mode 100644 charts/smm-operator/crds/crd-alertmanagerconfigs.yaml
create mode 100644 charts/smm-operator/crds/crd-alertmanagers.yaml
create mode 100644 charts/smm-operator/crds/crd-podmonitors.yaml
create mode 100644 charts/smm-operator/crds/crd-probes.yaml
create mode 100644 charts/smm-operator/crds/crd-prometheuses.yaml
create mode 100644 charts/smm-operator/crds/crd-prometheusrules.yaml
create mode 100644 charts/smm-operator/crds/crd-servicemonitors.yaml
create mode 100644 charts/smm-operator/crds/crd-thanosrulers.yaml
create mode 100644 charts/smm-operator/crds/crds.yaml
create mode 100644 charts/smm-operator/crds/health.yaml
create mode 100644 charts/smm-operator/crds/istio-operator-v1-crds.yaml
create mode 100644 charts/smm-operator/crds/istio-operator-v2-crds.gen.yaml
create mode 100644 charts/smm-operator/crds/istiooperator-crd.yaml
create mode 100644 charts/smm-operator/crds/koperator-crds.yaml
create mode 100644 charts/smm-operator/crds/metadata-crd.yaml
create mode 100644 charts/smm-operator/crds/resourcesyncrules-crd.yaml
create mode 100644 charts/smm-operator/crds/sre.yaml
create mode 100644 charts/smm-operator/templates/_helpers.tpl
create mode 100644 charts/smm-operator/templates/authproxy-rbac.yaml
create mode 100644 charts/smm-operator/templates/authproxy-service.yaml
create mode 100644 charts/smm-operator/templates/ecr.deployment.yaml
create mode 100644 charts/smm-operator/templates/ecr.secret.yaml
create mode 100644 charts/smm-operator/templates/ecr.service-account.yaml
create mode 100644 charts/smm-operator/templates/namespace.yaml
create mode 100644 charts/smm-operator/templates/operator-psp-basic.yaml
create mode 100644 charts/smm-operator/templates/operator-rbac.yaml
create mode 100644 charts/smm-operator/templates/operator-service.yaml
create mode 100644 charts/smm-operator/templates/operator-statefulset.yaml
create mode 100644 charts/smm-operator/values.yaml

    git push

  3. Apply the Application manifests.

    kubectl apply -f "apps/smm-operator/app-set.yaml"

  4. Verify that the applications have been added to Argo CD and are healthy.

    argocd app list

    Expected output:

    NAME                                    CLUSTER             NAMESPACE            PROJECT  STATUS  HEALTH   SYNCPOLICY  CONDITIONS  REPO                                                         PATH                 TARGET
argocd/smm-operator-workload-cluster-1  workload-cluster-1  smm-registry-access  default  Synced  Healthy  Auto-Prune  <none>      https://github.com/<github-user>/calisti-gitops-multi-cluster.git  charts/smm-operator  HEAD
argocd/smm-operator-workload-cluster-2  workload-cluster-2  smm-registry-access  default  Synced  Healthy  Auto-Prune  <none>      https://github.com/<github-user>/calisti-gitops-multi-cluster.git  charts/smm-operator  HEAD

  5. Check the smm-operator application on the Argo CD Web UI.

    SMM Operator SMM Operator

Deploy the smm-controlplane application

The following steps show you how to deploy the smm-controlplane application as an active-passive deployment. To create and active-active deployment, follow the same steps, there is an optional step that changes the active-passive deployment to active-active. For details, see Deployment models.

Deploy the smm-controlplane application using Kustomize: the active on workload-cluster-1 and the passive on workload-cluster-2. The active cluster receives every component, while the passive cluster only a few required components. This part of the repository will look like this:

└── manifests
    ├── smm-controlplane
    │   ├── base
    │   │   ├── control-plane.yaml
    │   │   ├── cert-manager-namespace.yaml
    │   │   ├── istio-system-namespace.yaml
    │   │   ├── istio-cp-v115x.yaml
    │   │   └── kustomization.yaml
    │   └── overlays
    │       ├── workload-cluster-1
    │       │   ├── control-plane.yaml
    │       │   ├── istio-cp-v115x.yaml
    │       │   └── kustomization.yaml
    │       └── workload-cluster-2
    │           ├── control-plane.yaml
    │           ├── istio-cp-v115x.yaml
    │           └── kustomization.yaml

  1. Create the following namespaces files.

    cat > manifests/smm-controlplane/base/cert-manager-namespace.yaml <<EOF
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"
  name: cert-manager
EOF

    cat > manifests/smm-controlplane/base/istio-system-namespace.yaml << EOF
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "2"
  name: istio-system
EOF

  2. Create the IstioControlPlane file.

    cat > manifests/smm-controlplane/base/istio-cp-v115x.yaml << EOF
apiVersion: servicemesh.cisco.com/v1alpha1
kind: IstioControlPlane
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "5"
  name: cp-v115x
  namespace: istio-system
spec:
  containerImageConfiguration:
    imagePullPolicy: Always
    imagePullSecrets:
    - name: smm-pull-secret
  distribution: cisco
  istiod:
    deployment:
      env:
      - name: ISTIO_MULTIROOT_MESH
        value: "true"
      image: registry.eticloud.io/smm/istio-pilot:v1.15.3-bzc.0
  k8sResourceOverlays:
  - groupVersionKind:
      group: apps
      kind: Deployment
      version: v1
    objectKey:
      name: istiod-cp-v115x
      namespace: istio-system
    patches:
    - path: /spec/template/spec/containers/0/args/-
      type: replace
      value: --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256
  meshConfig:
    defaultConfig:
      envoyAccessLogService:
        address: smm-als.smm-system.svc.cluster.local:50600
        tcpKeepalive:
          interval: 10s
          probes: 3
          time: 10s
        tlsSettings:
          mode: ISTIO_MUTUAL
      holdApplicationUntilProxyStarts: true
      proxyMetadata:
        ISTIO_META_ALS_ENABLED: "true"
        PROXY_CONFIG_XDS_AGENT: "true"
      tracing:
        tlsSettings:
          mode: ISTIO_MUTUAL
        zipkin:
          address: smm-zipkin.smm-system.svc.cluster.local:59411
    enableEnvoyAccessLogService: true
    enableTracing: true
  meshExpansion:
    enabled: true
    gateway:
      deployment:
        podMetadata:
          labels:
            app: istio-meshexpansion-gateway
            istio: meshexpansiongateway
      service:
        ports:
        - name: tcp-smm-als-tls
          port: 50600
          protocol: TCP
          targetPort: 50600
        - name: tcp-smm-zipkin-tls
          port: 59411
          protocol: TCP
          targetPort: 59411
  meshID: mesh1
  mode: ACTIVE
  networkName: network1
  proxy:
    image: registry.eticloud.io/smm/istio-proxyv2:v1.15.3-bzc.0
  proxyInit:
    cni:
      daemonset:
        image: registry.eticloud.io/smm/istio-install-cni:v1.15.3-bzc.0
    image: registry.eticloud.io/smm/istio-proxyv2:v1.15.3-bzc.0
  sidecarInjector:
    deployment:
      image: registry.eticloud.io/smm/istio-sidecar-injector:v1.15.3-bzc.0
  version: 1.15.3
EOF

  3. Create the kustomization.yaml file.

    cat > manifests/smm-controlplane/base/kustomization.yaml <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: cluster-secrets


resources:
- cert-manager-namespace.yaml
- istio-system-namespace.yaml
- istio-cp-v115x.yaml
- control-plane.yaml
EOF

  4. Create the manifests/smm-controlplane/base/control-plane.yaml file. You don’t need to set the CLUSTER-NAME here, you will set it with the overlays customization.

    cat > manifests/smm-controlplane/base/control-plane.yaml <<EOF
apiVersion: smm.cisco.com/v1alpha1
kind: ControlPlane
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "10"
  name: smm
spec:
  certManager:
    namespace: cert-manager
  clusterName: CLUSTER-NAME
  clusterRegistry:
    enabled: true
    namespace: cluster-registry
  log: {}
  meshManager:
    enabled: true
    istio:
      enabled: true
      istioCRRef:
        name: cp-v115x
        namespace: istio-system
      operators:
        namespace: smm-system
    namespace: smm-system
  nodeExporter:
    enabled: true
    namespace: smm-system
    psp:
      enabled: false
    rbac:
      enabled: true
  oneEye: {}
  registryAccess:
    enabled: true
    imagePullSecretsController: {}
    namespace: smm-registry-access
    pullSecrets:
    - name: smm-registry.eticloud.io-pull-secret
      namespace: smm-registry-access
  repositoryOverride:
    host: registry.eticloud.io
    prefix: smm
  role: active
  smm:
    als:
      enabled: true
      log: {}
    application:
      enabled: true
      log: {}
    auth:
      mode: impersonation
    certManager:
      enabled: true
    enabled: true
    federationGateway:
      enabled: true
      name: smm
      service:
        enabled: true
        name: smm-federation-gateway
        port: 80
    federationGatewayOperator:
      enabled: true
    impersonation:
      enabled: true
    istio:
      revision: cp-v115x.istio-system
    leo:
      enabled: true
      log: {}
    log: {}
    namespace: smm-system
    prometheus:
      enabled: true
      replicas: 1
    prometheusOperator: {}
    releaseName: smm
    role: active
    sdm:
      enabled: false
    sre:
      enabled: true
    useIstioResources: true
EOF

  5. Create the kustomization.yaml file for workload-cluster-1.

    cat > manifests/smm-controlplane/overlays/workload-cluster-1/kustomization.yaml <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
  - ../../base

patches:
  - istio-cp-v115x.yaml
  - control-plane.yaml
EOF

  6. Set the clusterName by overriding some settings coming from the base configuration. Create the following files.

    cat > manifests/smm-controlplane/overlays/workload-cluster-1/control-plane.yaml <<EOF
apiVersion: smm.cisco.com/v1alpha1
kind: ControlPlane
metadata:
  name: smm
spec:
  clusterName: workload-cluster-1
  certManager:
    enabled: true
  smm:
    exposeDashboard:
      meshGateway:
        enabled: true
    auth:
      forceUnsecureCookies: true
      mode: anonymous
EOF

    cat > manifests/smm-controlplane/overlays/workload-cluster-1/istio-cp-v115x.yaml <<EOF
apiVersion: servicemesh.cisco.com/v1alpha1
kind: IstioControlPlane
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "5"
  name: cp-v115x
  namespace: istio-system
spec:
  meshID: mesh1
  mode: ACTIVE
  networkName: network1
EOF

  7. Create the kustomization.yaml file for workload-cluster-2.

    cat > manifests/smm-controlplane/overlays/workload-cluster-2/kustomization.yaml <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
  - ../../base

patches:
  - istio-cp-v115x.yaml
  - control-plane.yaml
EOF

  8. Create the following files for workload-cluster-2. This sets the clusterName, and also overrides some settings of the base configuration.

    cat > manifests/smm-controlplane/overlays/workload-cluster-2/control-plane.yaml <<EOF
apiVersion: smm.cisco.com/v1alpha1
kind: ControlPlane
metadata:
  name: smm
spec:
  clusterName: workload-cluster-2
  role: passive
  smm:
    als:
      enabled: true
      log: {}
    application:
      enabled: false
      log: {}
    auth:
      mode: impersonation
    certManager:
      enabled: false
    enabled: true
    federationGateway:
      enabled: false
      name: smm
      service:
        enabled: true
        name: smm-federation-gateway
        port: 80
    federationGatewayOperator:
      enabled: true
    grafana:
      enabled: false
    impersonation:
      enabled: true
    istio:
      revision: cp-v115x.istio-system
    kubestatemetrics:
      enabled: true
    leo:
      enabled: false
      log: {}
    log: {}
    namespace: smm-system
    prometheus:
      enabled: true
      replicas: 1
      retentionTime: 8h
    prometheusOperator: {}
    releaseName: smm
    role: passive
    sdm:
      enabled: false
    sre:
      enabled: false
    tracing:
      enabled: true
    useIstioResources: false
    web:
      enabled: false
EOF

    cat > manifests/smm-controlplane/overlays/workload-cluster-2/istio-cp-v115x.yaml <<EOF
apiVersion: servicemesh.cisco.com/v1alpha1
kind: IstioControlPlane
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "5"
  name: cp-v115x
  namespace: istio-system
spec:
  meshID: mesh1
  mode: PASSIVE
  networkName: workload-cluster-2
EOF

  9. (Optional) If you want to change your active-passive deployment to active-active, complete this step. Otherwise, continue with the Commit deployment step.

    1. Run the following commands to modify the control planes of workload-cluster-2.

      cat > manifests/smm-controlplane/overlays/workload-cluster-2/control-plane.yaml <<EOF
apiVersion: smm.cisco.com/v1alpha1
kind: ControlPlane
metadata:
  name: smm
spec:
  clusterName: workload-cluster-2
  role: active
EOF

      cat > manifests/smm-controlplane/overlays/workload-cluster-2/istio-cp-v115x.yaml <<EOF
apiVersion: servicemesh.cisco.com/v1alpha1
kind: IstioControlPlane
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "5"
  name: cp-v115x
  namespace: istio-system
spec:
  meshID: mesh1
  mode: ACTIVE
  networkName: network1
EOF

    2. Create the smm-controlplane’s Argo CD ApplicationSet CR.

      cat > apps/smm-controlplane/app-set.yaml <<EOF
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: smm-cp-appset
  namespace: argocd
spec:
  generators:
  - list:
      elements:
      - cluster: "${WORKLOAD_CLUSTER_1_CONTEXT}"
        path: "${WORKLOAD_CLUSTER_1_CONTEXT}"
      - cluster: "${WORKLOAD_CLUSTER_2_CONTEXT}"
        path: "${WORKLOAD_CLUSTER_2_CONTEXT}"
  template:
    metadata: 
      name: 'smm-cp-{{cluster}}'
      namespace: argocd
    spec:
      project: default
      source:
        repoURL: https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git
        targetRevision: HEAD
        path: manifests/smm-controlplane/overlays/{{path}}
      destination:
        name: '{{cluster}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        retry:
          limit: 5
          backoff:
            duration: 5s
            maxDuration: 3m0s
            factor: 2
        syncOptions:
          - Validate=false
          - PruneLast=true
          - CreateNamespace=true
          - Replace=true
EOF

  10. Commit and push the calisti-gitops repository.

    git add apps/smm-controlplane manifests

    git commit -m "add smm-controlplane app"

    git push

  11. Apply the Application manifests.

    kubectl apply -f "apps/smm-controlplane/app-set.yaml"

  12. Verify that the applications have been added to Argo CD and are healthy.

    argocd app list

  13. To create trust between workload-cluster-1 and workload-cluster-2, you must exchange the Secret CRs of the clusters. The cluster registry controller helps to form a group of Kubernetes clusters and synchronize any resources across those clusters arbitrarily.

    Create and run the following bash script.

    cat > export-secrets.sh <<EOF
set -e

kubectl --context workload-cluster-1 get cluster workload-cluster-1 -o yaml | kubectl --context workload-cluster-2 apply -f -
kubectl --context workload-cluster-1 -n cluster-registry get secrets workload-cluster-1 -o yaml | kubectl --context workload-cluster-2 apply -f -

kubectl --context workload-cluster-2 get cluster workload-cluster-2 -o yaml | kubectl --context workload-cluster-1 apply -f -
kubectl --context workload-cluster-2 -n cluster-registry get secrets workload-cluster-2 -o yaml | kubectl --context workload-cluster-1 apply -f -

echo "Exporting cluster and secrets CRs successfully."
EOF

    chmod +x export-secrets.sh
./export-secrets.sh

    Expected output:

    cluster.clusterregistry.k8s.cisco.com/workload-cluster-1 created
secret/workload-cluster-1 created
cluster.clusterregistry.k8s.cisco.com/workload-cluster-2 created
secret/workload-cluster-2 created
Exporting cluster and secrets CRs successfully.

  14. Check that all pods are healthy and running in the smm-system namespace on workload-cluster-1 and workload-cluster-2. Note that it takes some time while the ControlPlane operator reconciles the resources.

    For workload-cluster-1:

    kubectl get pods -n smm-system --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_1_CONTEXT}"

    Expected output:

    NAME                                               READY   STATUS    RESTARTS        AGE
istio-operator-v115x-7d77fc549f-fxmtd              2/2     Running   0               8m15s
mesh-manager-0                                     2/2     Running   0          19m
prometheus-node-exporter-9xnmj                     1/1     Running   0          17m
prometheus-node-exporter-bf7g5                     1/1     Running   0          17m
prometheus-node-exporter-cl69q                     1/1     Running   0          17m
prometheus-smm-prometheus-0                        4/4     Running   0          18m
smm-7f4d5d4fff-4dlcp                               2/2     Running   0          18m
smm-7f4d5d4fff-59k7g                               2/2     Running   0          18m
smm-als-7cc4bfb998-wjsr6                           2/2     Running   0          18m
smm-authentication-569484f748-fj5zk                2/2     Running   0          18m
smm-federation-gateway-6964fb956f-pb5pv            2/2     Running   0          18m
smm-federation-gateway-operator-6664774695-9tmzj   2/2     Running   0          18m
smm-grafana-59c54f67f4-9snc5                       3/3     Running   0          18m
smm-health-75bf4f49c5-z9tqg                        2/2     Running   0          18m
smm-health-api-7767d4f46-744wn                     2/2     Running   0          18m
smm-ingressgateway-6ffdfc6d79-jttjz                1/1     Running   0          11m
smm-ingressgateway-external-8c9bb9445-kjt8h        1/1     Running   0          11m
smm-kubestatemetrics-86c6f96789-lp576              2/2     Running   0          18m
smm-leo-67cd7d49b5-gmcvf                           2/2     Running   0          18m
smm-prometheus-operator-ffbfb8b67-fwj6g            3/3     Running   0          18m
smm-sre-alert-exporter-6654968479-fthk6            2/2     Running   0          18m
smm-sre-api-86c9fb7cd7-mq7cm                       2/2     Running   0          18m
smm-sre-controller-6889685f9-hxxh5                 2/2     Running   0          18m
smm-tracing-5886d59dd-v8nb8                        2/2     Running   0          18m
smm-vm-integration-5b89c4f7c9-wz4bt                2/2     Running   0          18m
smm-web-d5b49c7f6-jgz7b                            3/3     Running   0          18m

    For workload-cluster-2:

    kubectl get pods -n smm-system --kubeconfig "${WORKLOAD_CLUSTER_2_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_2_CONTEXT}"

    Expected output:

    NAME                                       READY   STATUS    RESTARTS        AGE
istio-operator-v115x-7d77fc549f-s5wnz      2/2     Running   0               9m5s
mesh-manager-0                            2/2     Running   0             21m
prometheus-node-exporter-fzdn4            1/1     Running   0             5m18s
prometheus-node-exporter-rkbcl            1/1     Running   0             5m18s
prometheus-node-exporter-x2mwp            1/1     Running   0             5m18s
prometheus-smm-prometheus-0               3/3     Running   0             5m20s
smm-ingressgateway-5db7859d45-6d6ns       1/1     Running   0             12m
smm-kubestatemetrics-86c6f96789-j64q2     2/2     Running   0             19m
smm-prometheus-operator-ffbfb8b67-zwqn2   3/3     Running   1 (11m ago)   19m

  15. Check the applications on the Argo CD Web UI.

    Argo CD Web UI Argo CD Web UI

At this point, you have successfully installed smm-operator and workload-cluster-1 and workload-cluster-2. You can open the Service Mesh Manager dashboard to check them, or deploy an application.

Deploy an application

If you want to deploy want to deploy an application into the service mesh, complete the following steps. The examples use the Service Mesh Manager demo application.

The file structure for the demo application looks like this:

.
├── README.md
├── apps
│   ├── demo-app
│   │   └── app-set.yaml
│   └── ...
...manifests
   └── demo-app
        ├── base
        │   ├── demo-app-namespace.yaml
        │   ├── demo-app.yaml
        │   └── kustomization.yaml
        └── overlays
            ├── workload-cluster-1
            │   ├── demo-app.yaml
            │   └── kustomization.yaml
            └── workload-cluster-2
                ├── demo-app.yaml
                └── kustomization.yaml
...

  1. Create the application manifest files.

    cat > manifests/demo-app/base/kustomization.yaml <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: demo-app

resources:
- demo-app-namespace.yaml
- demo-app.yaml
EOF

    cat > manifests/demo-app/base/demo-app-namespace.yaml <<EOF
apiVersion: v1
kind: Namespace
metadata:
  labels:
    app.kubernetes.io/instance: smm-demo
    app.kubernetes.io/name: smm-demo
    app.kubernetes.io/part-of: smm-demo
    app.kubernetes.io/version: 0.1.4
    istio.io/rev: cp-v115x.istio-system
  name: smm-demo
EOF

    cat > manifests/demo-app/base/demo-app.yaml <<EOF
apiVersion: smm.cisco.com/v1alpha1
kind: DemoApplication
metadata:
  name: smm-demo
  namespace: smm-demo
spec:
  autoscaling:
    enabled: true
  controlPlaneRef:
    name: smm
EOF

    cat > manifests/demo-app/overlays/workload-cluster-1/kustomization.yaml <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
  - ../../base

patches:
  - demo-app.yaml
EOF

    cat > manifests/demo-app/overlays/workload-cluster-1/demo-app.yaml <<EOF
apiVersion: smm.cisco.com/v1alpha1
kind: DemoApplication
metadata:
  name: smm-demo
  namespace: smm-demo
spec:
  autoscaling:
    enabled: true
  controlPlaneRef:
    name: smm
  deployIstioResources: true
  deploySLOResources: true
  enabled: true
  enabledComponents:
  - frontpage
  - catalog
  - bookings
  - postgresql
  istio:
    revision: cp-v115x.istio-system
  load:
    enabled: true
    maxRPS: 30
    minRPS: 10
    swingPeriod: 1380000000000
  replicas: 1
  resources:
    limits:
      cpu: "2"
      memory: 192Mi
    requests:
      cpu: 40m
      memory: 64Mi
EOF

    cat > manifests/demo-app/overlays/workload-cluster-2/kustomization.yaml <<EOF
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

bases:
  - ../../base

patches:
  - demo-app.yaml
EOF

    cat > manifests/demo-app/overlays/workload-cluster-2/demo-app.yaml <<EOF
apiVersion: smm.cisco.com/v1alpha1
kind: DemoApplication
metadata:
  name: smm-demo
  namespace: smm-demo
spec:
  autoscaling:
    enabled: true
  controlPlaneRef:
    name: smm
  deployIstioResources: false
  deploySLOResources: false
  enabled: true
  enabledComponents:
  - movies
  - payments
  - notifications
  - analytics
  - database
  - mysql
  istio:
    revision: cp-v115x.istio-system
  replicas: 1
  resources:
    limits:
      cpu: "2"
      memory: 192Mi
    requests:
      cpu: 40m
      memory: 64Mi
EOF

  2. Create the Demo application ApplicationSet.

    cat > apps/demo-app/app-set.yaml <<EOF
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: demo-app-appset
  namespace: argocd
spec:
  generators:
  - list:
      elements:
      - cluster: "${WORKLOAD_CLUSTER_1_CONTEXT}"
        path: "${WORKLOAD_CLUSTER_1_CONTEXT}"
      - cluster: "${WORKLOAD_CLUSTER_2_CONTEXT}"
        path: "${WORKLOAD_CLUSTER_2_CONTEXT}"
  template:
    metadata: 
      name: 'demo-app-{{cluster}}'
      namespace: argocd
    spec:
      project: default
      source:
        repoURL: https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git
        targetRevision: HEAD
        path: manifests/demo-app/overlays/{{path}}
      destination:
        name: '{{cluster}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        retry:
          limit: 5
          backoff:
            duration: 5s
            maxDuration: 3m0s
            factor: 2
        syncOptions:
          - Validate=false
          - PruneLast=true
          - CreateNamespace=true
          - Replace=true
EOF

  3. Commit and push the calisti-gitops repository.

    git add apps/demo-app manifests
git commit -m "add demo application"
git push origin

  4. Deploy the demo application on the clusters.

    kubectl apply -f apps/demo-app/app-set.yaml

  5. Wait until all the pods in the smm-demo namespace are up and running.

    kubectl get pods -n smm-demo --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_1_CONTEXT}"

    Expected output:

    NAME                           READY   STATUS    RESTARTS   AGE
bombardier-5f59948978-zx99c    2/2     Running   0          3m21s
bookings-v1-68dd865855-fdcxk   2/2     Running   0          3m21s
catalog-v1-6d564bbcb8-qmhbx    2/2     Running   0          3m21s
frontpage-v1-b4686759b-fhfmv   2/2     Running   0          3m21s
postgresql-7cf55cd596-grs46    2/2     Running   0          3m21s

    kubectl get pods -n smm-demo --kubeconfig "${WORKLOAD_CLUSTER_2_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_2_CONTEXT}"

    Expected output:

    NAME                                READY   STATUS    RESTARTS   AGE
analytics-v1-799d668f84-p4nkk       2/2     Running   0          3m58s
database-v1-6896cd4b59-9xxgg        2/2     Running   0          3m58s
movies-v1-9594fff5f-8hv9l           2/2     Running   0          3m58s
movies-v2-5559c5567c-2279n          2/2     Running   0          3m58s
movies-v3-649b99d977-nkdxc          2/2     Running   0          3m58s
mysql-669466cc8d-bs4s9              2/2     Running   0          3m58s
notifications-v1-79bc79c89b-4bbss   2/2     Running   0          3m58s
payments-v1-547884bfdf-dg2dm        2/2     Running   0          3m58s

  6. Check the applications on the Argo CD web UI.

    Argo CD Web UI Argo CD Web UI

  7. Open the Service Mesh Manager web interface, select MENU > TOPOLOGY, then select the smm-demo namespace.

    Demo Application Topology Demo Application Topology

Access the Service Mesh Manager dashboard

  1. You can access the Service Mesh Manager dashboard via the smm-ingressgateway-external LoadBalancer external-ip-or-hostname address. Run the following command to retrieve the IP address:

    kubectl get services -n smm-system smm-ingressgateway-external --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_1_CONTEXT}"

    Expected output:

    NAME                          TYPE           CLUSTER-IP   EXTERNAL-IP                PORT(S)        AGE
smm-ingressgateway-external   LoadBalancer   10.0.0.199   external-ip-or-hostname    80:32505/TCP   2m28s

  2. Open the Service Mesh Manager dashboard using one of the following methods:

    • Open the http://<external-ip-or-hostname> URL in your browser.

    • Run the following command to open the dashboard with your default browser:

      # Exactly one of hostname or IP will be available and used for the remote URL.
open http://$(kubectl get services -n smm-system smm-ingressgateway-external -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{.status.loadBalancer.ingress[0].ip}' --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_1_CONTEXT}")

    • If you have installed the Service Mesh Manager CLI on your machine, run the following command to open the Service Mesh Manager Dashboard in the default browser.

      smm dashboard --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_1_CONTEXT}"

      Expected output:

      ✓ validate-kubeconfig ❯ checking cluster reachability...
✓ opening Service Mesh Manager at http://127.0.0.1:50500

  3. Check the deployments on the dashboard, for example, on the MENU > Overview, MENU > MESH, and MENU > TOPOLOGY pages.

Service Mesh Manager Overview Service Mesh Manager Overview

Service Mesh Manager Mesh Service Mesh Manager Mesh

Service Mesh Manager Topology Service Mesh Manager Topology