Install SMM - GitOps - multi-cluster

This guide details how to set up a multi-cluster Service Mesh Manager scenario in a GitOps environment for Service Mesh Manager using Argo CD. The same principles can be used for other tools as well.

CAUTION:

Do not push the secrets directly into the git repository, especially when it is a public repository. Argo CD provides solutions to keep secrets safe.

Architecture

Service Mesh Manager supports multiple mesh topologies, so you can use the one that best fits your use cases. In multi-cluster configurations it provides automatic locality load-balancing.

The high level architecture for Argo CD with a multi-cluster Service Mesh Manager setup consists of the following components:

  • A git repository that stores the various charts and manifests,
  • a management cluster that runs the Argo CD server, and
  • the Service Mesh Manager clusters managed by Argo CD.

Multi-cluster GitOps architecture Multi-cluster GitOps architecture

Deployment models

When deploying Service Mesh Manager in a multi-cluster scenario you can deploy Service Mesh Manager in an active-passive model. For details on Service Mesh Manager clusters and their relationship to Istio clusters, see Istio clusters and SMM clusters.

Prerequisites

  • A free registration for the Service Mesh Manager download page
  • A Kubernetes cluster to deploy Argo CD on (called management-cluster in the examples).
  • Two Kubernetes clusters to deploy Service Mesh Manager on (called workload-cluster-1 and workload-cluster-2 in the examples).

CAUTION:

Supported providers and Kubernetes versions

The cluster must run a Kubernetes version that Service Mesh Manager supports: Kubernetes 1.21, 1.22, 1.23, 1.24.

Service Mesh Manager is tested and known to work on the following Kubernetes providers:

  • Amazon Elastic Kubernetes Service (Amazon EKS)
  • Google Kubernetes Engine (GKE)
  • Azure Kubernetes Service (AKS)
  • Red Hat OpenShift 4.11
  • On-premises installation of stock Kubernetes with load balancer support (and optionally PVCs for persistence)

Calisti resource requirements

Make sure that your Kubernetes or OpenShift cluster has sufficient resources to install Calisti. The following table shows the number of resources needed on the cluster:

Resource Required
CPU - 32 vCPU in total
- 4 vCPU available for allocation per worker node (If you are testing on a cluster at a cloud provider, use nodes that have at least 4 CPUs, for example, c5.xlarge on AWS.)
Memory - 64 GiB in total
- 4 GiB available for allocation per worker node for the Kubernetes cluster (8 GiB in case of the OpenShift cluster)
Storage 12 GB of ephemeral storage on the Kubernetes worker nodes (for Traces and Metrics)

These minimum requirements need to be available for allocation within your cluster, in addition to the requirements of any other loads running in your cluster (for example, DaemonSets and Kubernetes node-agents). If Kubernetes cannot allocate sufficient resources to Service Mesh Manager, some pods will remain in Pending state, and Service Mesh Manager will not function properly.

Enabling additional features, such as High Availability increases this value.

The default installation, when enough headroom is available in the cluster, should be able to support at least 150 running Pods with the same amount of Services. For setting up Service Mesh Manager for bigger workloads, see scaling Service Mesh Manager.

Install Argo CD

Complete the following steps to install Argo CD on the management cluster.

Set up the environment

  1. Set the KUBECONFIG location and context name for the management-cluster cluster.

    MANAGEMENT_CLUSTER_KUBECONFIG=management_cluster_kubeconfig.yaml
    MANAGEMENT_CLUSTER_CONTEXT=management-cluster
    kubectl config --kubeconfig "${MANAGEMENT_CLUSTER_KUBECONFIG}" get-contexts "${MANAGEMENT_CLUSTER_CONTEXT}"
    

    Expected output:

    CURRENT   NAME                 CLUSTER              AUTHINFO   NAMESPACE
    *         management-cluster   management-cluster
    
  2. Set the KUBECONFIG location and context name for the workload-cluster-1 cluster.

    WORKLOAD_CLUSTER_1_KUBECONFIG=workload_cluster_1_kubeconfig.yaml
    WORKLOAD_CLUSTER_1_CONTEXT=workload-cluster-1
    kubectl config --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" get-contexts "${WORKLOAD_CLUSTER_1_CONTEXT}"
    

    Expected output:

    CURRENT   NAME                 CLUSTER              AUTHINFO                                          NAMESPACE
    *         workload-cluster-1   workload-cluster-1
    

    Repeat this step for any additional workload clusters you want to use.

  3. Add the cluster configurations to KUBECONFIG. Include any additional workload clusters you want to use.

    KUBECONFIG=$KUBECONFIG:$MANAGEMENT_CLUSTER_KUBECONFIG:$WORKLOAD_CLUSTER_1_KUBECONFIG
    
  4. Make sure the management-cluster Kubernetes context is the current context.

    kubectl config use-context "${MANAGEMENT_CLUSTER_CONTEXT}"
    

    Expected output:

    Switched to context "management-cluster".
    

Install Argo CD Server

  1. Create the argocd namespace.

    kubectl create namespace argocd
    

    Expected output:

    namespace/argocd created
    
  2. On OpenShift: Run the following command to grant the service accounts access to the argocd namespace.

    oc adm policy add-scc-to-group privileged system:serviceaccounts:argocd
    

    Expected output:

    clusterrole.rbac.authorization.k8s.io/system:openshift:scc:privileged added: "system:serviceaccounts:argocd"
    
  3. Deploy Argo CD.

    kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
    
    customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io created
    customresourcedefinition.apiextensions.k8s.io/applicationsets.argoproj.io created
    customresourcedefinition.apiextensions.k8s.io/appprojects.argoproj.io created
    serviceaccount/argocd-application-controller created
    serviceaccount/argocd-applicationset-controller created
    serviceaccount/argocd-dex-server created
    serviceaccount/argocd-notifications-controller created
    serviceaccount/argocd-redis created
    serviceaccount/argocd-repo-server created
    serviceaccount/argocd-server created
    role.rbac.authorization.k8s.io/argocd-application-controller created
    role.rbac.authorization.k8s.io/argocd-applicationset-controller created
    role.rbac.authorization.k8s.io/argocd-dex-server created
    role.rbac.authorization.k8s.io/argocd-notifications-controller created
    role.rbac.authorization.k8s.io/argocd-server created
    clusterrole.rbac.authorization.k8s.io/argocd-application-controller created
    clusterrole.rbac.authorization.k8s.io/argocd-server created
    rolebinding.rbac.authorization.k8s.io/argocd-application-controller created
    rolebinding.rbac.authorization.k8s.io/argocd-applicationset-controller created
    rolebinding.rbac.authorization.k8s.io/argocd-dex-server created
    rolebinding.rbac.authorization.k8s.io/argocd-notifications-controller created
    rolebinding.rbac.authorization.k8s.io/argocd-redis created
    rolebinding.rbac.authorization.k8s.io/argocd-server created
    clusterrolebinding.rbac.authorization.k8s.io/argocd-application-controller created
    clusterrolebinding.rbac.authorization.k8s.io/argocd-server created
    configmap/argocd-cm created
    configmap/argocd-cmd-params-cm created
    configmap/argocd-gpg-keys-cm created
    configmap/argocd-notifications-cm created
    configmap/argocd-rbac-cm created
    configmap/argocd-ssh-known-hosts-cm created
    configmap/argocd-tls-certs-cm created
    secret/argocd-notifications-secret created
    secret/argocd-secret created
    service/argocd-applicationset-controller created
    service/argocd-dex-server created
    service/argocd-metrics created
    service/argocd-notifications-controller-metrics created
    service/argocd-redis created
    service/argocd-repo-server created
    service/argocd-server created
    service/argocd-server-metrics created
    deployment.apps/argocd-applicationset-controller created
    deployment.apps/argocd-dex-server created
    deployment.apps/argocd-notifications-controller created
    deployment.apps/argocd-redis created
    deployment.apps/argocd-repo-server created
    deployment.apps/argocd-server created
    statefulset.apps/argocd-application-controller created
    networkpolicy.networking.k8s.io/argocd-application-controller-network-policy created
    networkpolicy.networking.k8s.io/argocd-applicationset-controller-network-policy created
    networkpolicy.networking.k8s.io/argocd-dex-server-network-policy created
    networkpolicy.networking.k8s.io/argocd-notifications-controller-network-policy created
    networkpolicy.networking.k8s.io/argocd-redis-network-policy created
    networkpolicy.networking.k8s.io/argocd-repo-server-network-policy created
    networkpolicy.networking.k8s.io/argocd-server-network-policy created
    
  4. Wait until the installation is complete, then check that the Argo CD pods are up and running.

    kubectl get pods -n argocd
    

    The output should be similar to:

    NAME                                                    READY   STATUS    RESTARTS   AGE
    pod/argocd-application-controller-0                     1/1     Running   0          7h59m
    pod/argocd-applicationset-controller-78b8b554f9-pgwbl   1/1     Running   0          7h59m
    pod/argocd-dex-server-6bbc85c688-8p7zf                  1/1     Running   0          16h
    pod/argocd-notifications-controller-75847756c5-dbbm5    1/1     Running   0          16h
    pod/argocd-redis-f4cdbff57-wcpxh                        1/1     Running   0          7h59m
    pod/argocd-repo-server-d5c7f7ffb-c8962                  1/1     Running   0          7h59m
    pod/argocd-server-76497676b-pnvf4                       1/1     Running   0          7h59m
    
  5. For the Argo CD UI, set the argocd-server service type to LoadBalancer.

    kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'
    

    Expected output:

    service/argocd-server patched
    
  6. Patch the App of Apps health check in Argo CD configuration to ignore diffs of controller/operator managed fields. For details about this patch, see the Argo CD documentation sections Resource Health and Diffing Customization.

    Apply the new Argo CD health check configurations:

    kubectl apply -f - <<EOF
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: argocd-cm
      namespace: argocd
      labels:
        app.kubernetes.io/name: argocd-cm
        app.kubernetes.io/part-of: argocd
    data:
      # App of app health check
      resource.customizations.health.argoproj.io_Application: |
        hs = {}
        hs.status = "Progressing"
        hs.message = ""
        if obj.status ~= nil then
          if obj.status.health ~= nil then
            hs.status = obj.status.health.status
            if obj.status.health.message ~= nil then
              hs.message = obj.status.health.message
            end
          end
        end
        return hs
      # Ignoring RBAC changes made by AggregateRoles
      resource.compareoptions: |
        # disables status field diffing in specified resource types
        ignoreAggregatedRoles: true
    
        # disables status field diffing in specified resource types
        # 'crd' - CustomResourceDefinition-s (default)
        # 'all' - all resources
        # 'none' - disabled
        ignoreResourceStatusField: all
    EOF
    

    Expected output:

    configmap/argocd-cm configured
    
  7. Get the initial password for the admin user.

    kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
    

    Expected output:

    argocd-admin-password
    
  8. Check the external-ip-or-hostname address of the argocd-server service.

    kubectl get service -n argocd argocd-server
    

    The output should be similar to:

    NAME                                      TYPE           CLUSTER-IP      EXTERNAL-IP               PORT(S)                      AGE
    argocd-server                             LoadBalancer   10.108.14.130   external-ip-or-hostname   80:31306/TCP,443:30063/TCP   7d13h
    
  9. Open the https://external-ip-or-hostname URL and log in to the Argo CD server using the password received in the previous step.

    # Exactly one of hostname or IP will be available and used for the remote URL.
    open https://$(kubectl get service -n argocd argocd-server -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{.status.loadBalancer.ingress[0].ip}')
    

Install Argo CD CLI

  1. Install Argo CD CLI on your computer. For details, see the Argo CD documentation.

  2. Log in with the CLI:

    # Exactly one of hostname or IP will be available and used for the remote URL.
    argocd login $(kubectl get service -n argocd argocd-server -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{.status.loadBalancer.ingress[0].ip}') --insecure --username admin --password $(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)
    

    Expected output:

    'admin:login' logged in successfully
    

For more details about Argo CD installation, see the Argo CD getting started guide.

Register clusters

  1. Register the clusters that will run Service Mesh Manager in Argo CD. In this example, register workload-cluster-1 and workload-cluster-2 using one of the following methods.

    • Register the cluster from the command line by running:

      argocd cluster add --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" "${WORKLOAD_CLUSTER_1_CONTEXT}"
      

      Expected output:

      WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `workload-cluster-1` with full cluster level privileges. Do you want to continue [y/N]? y
      INFO[0005] ServiceAccount "argocd-manager" created in namespace "kube-system"
      INFO[0005] ClusterRole "argocd-manager-role" created
      INFO[0005] ClusterRoleBinding "argocd-manager-role-binding" created
      INFO[0011] Created bearer token secret for ServiceAccount "argocd-manager"
      Cluster 'https://workload-cluster-1-ip-or-hostname' added
      
      argocd cluster add --kubeconfig "${WORKLOAD_CLUSTER_2_KUBECONFIG}" "${WORKLOAD_CLUSTER_2_CONTEXT}"
      

      Expected output:

      WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `workload-cluster-2` with full cluster level privileges. Do you want to continue [y/N]? y
      INFO[0005] ServiceAccount "argocd-manager" created in namespace "kube-system"
      INFO[0005] ClusterRole "argocd-manager-role" created
      INFO[0005] ClusterRoleBinding "argocd-manager-role-binding" created
      INFO[0011] Created bearer token secret for ServiceAccount "argocd-manager"
      Cluster 'https://workload-cluster-2-ip-or-hostname' added
      
    • Alternatively, you can register clusters declaratively as Kubernetes secrets. Modify the following command for your environment and apply it. For details, see the Argo CD documentation.

      WORKLOAD_CLUSTER_1_IP="https://workload-cluster-1-IP" ARGOCD_BEARER_TOKEN="authentication-token" ARGOCD_CA_B64="base64 encoded certificate" ; kubectl apply -f - <<EOF
      apiVersion: v1
      kind: Secret
      metadata:
        name: workload-cluster-1-secret
        labels:
          argocd.argoproj.io/secret-type: cluster
      type: Opaque
      stringData:
        name: workload-cluster-1
        server: "${WORKLOAD_CLUSTER_1_IP}"
        config: |
          {
            "bearerToken": "${ARGOCD_BEARER_TOKEN}",
            "tlsClientConfig": {
              "insecure": false,
              "caData": "${ARGOCD_CA_B64}"
            }
          }
      EOF
      
      WORKLOAD_CLUSTER_2_IP="https://workload-cluster-2-IP" ARGOCD_BEARER_TOKEN="authentication-token" ARGOCD_CA_B64="base64 encoded certificate" ; kubectl apply -f - <<EOF
      apiVersion: v1
      kind: Secret
      metadata:
        name: workload-cluster-2-secret
        labels:
          argocd.argoproj.io/secret-type: cluster
      type: Opaque
      stringData:
        name: workload-cluster-2
        server: "${WORKLOAD_CLUSTER_2_IP}"
        config: |
          {
            "bearerToken": "${ARGOCD_BEARER_TOKEN}",
            "tlsClientConfig": {
              "insecure": false,
              "caData": "${ARGOCD_CA_B64}"
            }
          }
      EOF
      
  2. Make sure that the cluster is registered in Argo CD by running the following command:

    argocd cluster list
    

    The output should be similar to:

    SERVER                                      NAME                VERSION  STATUS   MESSAGE                                                  PROJECT
    https://kubernetes.default.svc              in-cluster                   Unknown  Cluster has no applications and is not being monitored.
    https://workload-cluster-1-ip-or-hostname   workload-cluster-1           Unknown  Cluster has no applications and is not being monitored.
    https://workload-cluster-2-ip-or-hostname   workload-cluster-2           Unknown  Cluster has no applications and is not being monitored.
    

Prepare Git repository

  1. Create an empty repository called calisti-gitops on GitHub (or another provider that Argo CD supports) and initialize it with a README.md file so that you can clone the repository. Because Service Mesh Manager credentials will be stored in this repository, make it a private repository.

    GITHUB_ID="github-id"
    GITHUB_REPOSITORY_NAME="calisti-gitops"
    
  2. Obtain a personal access token to the repository (on GitHub, see Creating a personal access token), that has the following permissions:

    • admin:org_hook
    • admin:repo_hook
    • read:org
    • read:public_key
    • repo
  3. Log in with your personal access token with git.

    export GH_TOKEN="github-personal-access-token" # Note: this environment variable needs to be exported so the `git` binary is going to use it automatically for authentication.
    
  4. Clone the repository into your local workspace, for example:

    git clone "https://${GITHUB_ID}:${GH_TOKEN}@github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git"
    

    Expected output:

    Cloning into 'calisti-gitops'...
    remote: Enumerating objects: 144, done.
    remote: Counting objects: 100% (144/144), done.
    remote: Compressing objects: 100% (93/93), done.
    remote: Total 144 (delta 53), reused 135 (delta 47), pack-reused 0
    Receiving objects: 100% (144/144), 320.08 KiB | 746.00 KiB/s, done.
    Resolving deltas: 100% (53/53), done.
    
  5. Add the repository to Argo CD by running the following command. Alternatively, you can add it on Argo CD Web UI.

    argocd repo add "https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git" --name "${GITHUB_REPOSITORY_NAME}" --username "${GITHUB_ID}" --password "${GH_TOKEN}"
    

    Expected output:

    Repository 'https://github.com/github-id/calisti-gitops.git' added
    
  6. Verify that the repository is connected by running:

    argocd repo list
    

    In the output, Status should be Successful:

    TYPE  NAME            REPO                                             INSECURE  OCI    LFS    CREDS  STATUS      MESSAGE  PROJECT
    git   calisti-gitops  https://github.com/github-id/calisti-gitops.git  false     false  false  true   Successful
    
  7. Change into the directory of the cloned repository (for example, calisti-gitops) and create the following directories.

    cd "${GITHUB_REPOSITORY_NAME}"
    
    mkdir -p apps/smm-controlplane apps/smm-operator apps/demo-app charts manifests/smm-controlplane/base manifests/smm-controlplane/overlays/workload-cluster-1 manifests/smm-controlplane/overlays/workload-cluster-2 manifests/demo-app/base manifests/demo-app/overlays/workload-cluster-1 manifests/demo-app/overlays/workload-cluster-2
    

    The final structure of the repository will look like this:

    .
    ├── README.md
    ├── apps
    │   ├── smm-controlplane
    │   │   └── app-set.yaml
    │   ├── smm-operator
    │   │   └── app-set.yaml
    │   └── demo-app
    │       └── app-set.yaml
    ├── charts
    │   └── smm-operator
    │       ├── Chart.yaml
    │       └── ...
    ├── export-secrets.sh
    └── manifests
       ├── smm-controlplane
       │   ├── base
       │   │   ├── control-plane.yaml
       │   │   ├── cert-manager-namespace.yaml
       │   │   ├── istio-system-namespace.yaml
       │   │   ├── istio-cp-v115x.yaml
       │   │   └── kustomization.yaml
       │   └── overlays
       │       ├── workload-cluster-1
       │       │   ├── control-plane.yaml
       │       │   ├── istio-cp-v115x.yaml
       │       │   └── kustomization.yaml
       │       └── workload-cluster-2
       │           ├── control-plane.yaml
       │           ├── istio-cp-v115x.yaml
       │           └── kustomization.yaml
       └── demo-app
           ├── base
           │   ├── demo-app-namespace.yaml
           │   ├── demo-app.yaml
           │   └── kustomization.yaml
           └── overlays
               ├── workload-cluster-1
               │   ├── demo-app.yaml
               │   └── kustomization.yaml
               └── workload-cluster-2
                   ├── demo-app.yaml
                   └── kustomization.yaml
    
    • The apps folder contains the Argo CD Application of the smm-operator, the smm-controlplane, and the demo-app.
    • The charts folder contains the Helm chart of the smm-operator.
    • The manifests/demo-app folder contains the manifest files of the demo application that represents your business application.
    • The manifests/smm-controlplane folder contains the manifest files of the SMM ControlPlane.

Prepare the helm charts

  1. You need an active Service Mesh Manager registration to download the Service Mesh Manager charts and images. You can sign up for free, or obtain Enterprise credentials on the official Cisco Service Mesh Manager page. After registration, you can obtain your username and password from the Download Center. Set them as environment variables.

    CALISTI_USERNAME="<your-calisti-username>"
    
    CALISTI_PASSWORD="<your-calisti-password>"
    
  2. Download the smm-operator chart from registry.eticloud.io into the charts directory of your Service Mesh Manager GitOps repository and extract it. Run the following commands:

    export HELM_EXPERIMENTAL_OCI=1 # Needed prior to Helm version 3.8.0
    
    echo "${CALISTI_PASSWORD}" | helm registry login registry.eticloud.io -u "${CALISTI_USERNAME}" --password-stdin
    

    Expected output:

    Login Succeeded
    
    helm pull oci://registry.eticloud.io/smm-charts/smm-operator --destination ./charts/ --untar --version 1.12.1
    

    Expected output:

    Pulled: registry.eticloud.io/smm-charts/smm-operator:latest-stable-version
    Digest: sha256:someshadigest
    

Deploy Service Mesh Manager

Deploy the smm-operator application set

Complete the following steps to deploy the smm-operator chart using Argo CD.

  1. Create the smm-operator’s Argo CD ApplicationSet CR. Argo CD ApplicationSet is perfect for deploying the same application on to different clusters. You can use list generators with cluster data in the ApplicationSet.

    Before running the following command, edit it if needed:

    • If you are not using a GitHub repository, set the repoURL field to your repository.
    • For multi-cluster setups, the Kubernetes API server address of one cluster must be reachable from other clusters. The API server addresses are private for certain clusters (for example, OpenShift) and not reachable by default from other clusters. In such case, use the PUBLIC_API_SERVER_ENDPOINT_ADDRESS variable to provide an address that’s reachable from the other clusters. This can be a public address, or one that’s routable from the other clusters.

    PUBLIC_API_SERVER_ENDPOINT_ADDRESS_1="" PUBLIC_API_SERVER_ENDPOINT_ADDRESS_2="" ;
    cat > "apps/smm-operator/app-set.yaml" <<EOF
    # apps/smm-operator/app-set.yaml
    apiVersion: argoproj.io/v1alpha1
    kind: ApplicationSet
    metadata:
      name: smm-operator-appset
      namespace: argocd
    spec:
      generators:
      - list:
          elements:
          - cluster: "${WORKLOAD_CLUSTER_1_CONTEXT}"
            apiServerEndpointAddress: "${PUBLIC_API_SERVER_ENDPOINT_ADDRESS_1}"
          - cluster: "${WORKLOAD_CLUSTER_2_CONTEXT}"
            apiServerEndpointAddress: "${PUBLIC_API_SERVER_ENDPOINT_ADDRESS_2}"
      template:
        metadata: 
          name: 'smm-operator-{{cluster}}'
          namespace: argocd
        spec:
          project: default
          source:
            repoURL: https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git
            targetRevision: HEAD
            path: charts/smm-operator
            helm:
              parameters:
              - name: "global.ecr.enabled"
                value: 'false'
              - name: "global.basicAuth.username"
                value: "${CALISTI_USERNAME}"
              - name: "global.basicAuth.password"
                value: "${CALISTI_PASSWORD}"
              - name: "apiServerEndpointAddress"
                value: '{{apiServerEndpointAddress}}'
          destination:
            namespace: smm-registry-access
            name: '{{cluster}}'
          ignoreDifferences:
          - kind: ValidatingWebhookConfiguration
            group: admissionregistration.k8s.io
            jsonPointers:
            - /webhooks
          syncPolicy:
            automated:
              prune: true
              selfHeal: true
            retry:
              limit: 5
              backoff:
                duration: 5s
                maxDuration: 3m0s
                factor: 2
            syncOptions:
              - Validate=false
              - PruneLast=true
              - CreateNamespace=true
              - Replace=true
    EOF
    
  2. Commit and push the calisti-gitops repository.

    git add apps/smm-operator charts/smm-operator
    
    git commit -m "add smm-operator app"
    
    [main d4f6809] add smm-operator app
    35 files changed, 80310 insertions(+)
    create mode 100644 apps/smm-operator/app-set.yaml
    create mode 100644 charts/smm-operator/.helmignore
    create mode 100644 charts/smm-operator/Chart.yaml
    create mode 100644 charts/smm-operator/README.md
    create mode 100644 charts/smm-operator/crds/clusterfeature-crd.yaml
    create mode 100644 charts/smm-operator/crds/clusters-crd.yaml
    create mode 100644 charts/smm-operator/crds/crd-alertmanagerconfigs.yaml
    create mode 100644 charts/smm-operator/crds/crd-alertmanagers.yaml
    create mode 100644 charts/smm-operator/crds/crd-podmonitors.yaml
    create mode 100644 charts/smm-operator/crds/crd-probes.yaml
    create mode 100644 charts/smm-operator/crds/crd-prometheuses.yaml
    create mode 100644 charts/smm-operator/crds/crd-prometheusrules.yaml
    create mode 100644 charts/smm-operator/crds/crd-servicemonitors.yaml
    create mode 100644 charts/smm-operator/crds/crd-thanosrulers.yaml
    create mode 100644 charts/smm-operator/crds/crds.yaml
    create mode 100644 charts/smm-operator/crds/health.yaml
    create mode 100644 charts/smm-operator/crds/istio-operator-v1-crds.yaml
    create mode 100644 charts/smm-operator/crds/istio-operator-v2-crds.gen.yaml
    create mode 100644 charts/smm-operator/crds/istiooperator-crd.yaml
    create mode 100644 charts/smm-operator/crds/koperator-crds.yaml
    create mode 100644 charts/smm-operator/crds/metadata-crd.yaml
    create mode 100644 charts/smm-operator/crds/resourcesyncrules-crd.yaml
    create mode 100644 charts/smm-operator/crds/sre.yaml
    create mode 100644 charts/smm-operator/templates/_helpers.tpl
    create mode 100644 charts/smm-operator/templates/authproxy-rbac.yaml
    create mode 100644 charts/smm-operator/templates/authproxy-service.yaml
    create mode 100644 charts/smm-operator/templates/ecr.deployment.yaml
    create mode 100644 charts/smm-operator/templates/ecr.secret.yaml
    create mode 100644 charts/smm-operator/templates/ecr.service-account.yaml
    create mode 100644 charts/smm-operator/templates/namespace.yaml
    create mode 100644 charts/smm-operator/templates/operator-psp-basic.yaml
    create mode 100644 charts/smm-operator/templates/operator-rbac.yaml
    create mode 100644 charts/smm-operator/templates/operator-service.yaml
    create mode 100644 charts/smm-operator/templates/operator-statefulset.yaml
    create mode 100644 charts/smm-operator/values.yaml
    
    git push
    
  3. Apply the Application manifests.

    kubectl apply -f "apps/smm-operator/app-set.yaml"
    
  4. Verify that the applications have been added to Argo CD and are healthy.

    argocd app list
    

    Expected output:

    NAME                                    CLUSTER             NAMESPACE            PROJECT  STATUS  HEALTH   SYNCPOLICY  CONDITIONS  REPO                                                         PATH                 TARGET
    argocd/smm-operator-workload-cluster-1  workload-cluster-1  smm-registry-access  default  Synced  Healthy  Auto-Prune  <none>      https://github.com/<github-user>/calisti-gitops-multi-cluster.git  charts/smm-operator  HEAD
    argocd/smm-operator-workload-cluster-2  workload-cluster-2  smm-registry-access  default  Synced  Healthy  Auto-Prune  <none>      https://github.com/<github-user>/calisti-gitops-multi-cluster.git  charts/smm-operator  HEAD
    
  5. Check the smm-operator application on the Argo CD Web UI.

    SMM Operator SMM Operator

Deploy the smm-controlplane application

The following steps show you how to deploy the smm-controlplane application as an active-passive deployment. To create and active-active deployment, follow the same steps, there is an optional step that changes the active-passive deployment to active-active. For details, see Deployment models.

Deploy the smm-controlplane application using Kustomize: the active on workload-cluster-1 and the passive on workload-cluster-2. The active cluster receives every component, while the passive cluster only a few required components. This part of the repository will look like this:

└── manifests
    ├── smm-controlplane
    │   ├── base
    │   │   ├── control-plane.yaml
    │   │   ├── cert-manager-namespace.yaml
    │   │   ├── istio-system-namespace.yaml
    │   │   ├── istio-cp-v115x.yaml
    │   │   └── kustomization.yaml
    │   └── overlays
    │       ├── workload-cluster-1
    │       │   ├── control-plane.yaml
    │       │   ├── istio-cp-v115x.yaml
    │       │   └── kustomization.yaml
    │       └── workload-cluster-2
    │           ├── control-plane.yaml
    │           ├── istio-cp-v115x.yaml
    │           └── kustomization.yaml
  1. Create the following namespaces files.

    cat > manifests/smm-controlplane/base/cert-manager-namespace.yaml <<EOF
    apiVersion: v1
    kind: Namespace
    metadata:
      annotations:
        argocd.argoproj.io/sync-wave: "1"
      name: cert-manager
    EOF
    
    cat > manifests/smm-controlplane/base/istio-system-namespace.yaml << EOF
    apiVersion: v1
    kind: Namespace
    metadata:
      annotations:
        argocd.argoproj.io/sync-wave: "2"
      name: istio-system
    EOF
    
  2. Create the IstioControlPlane file.

    cat > manifests/smm-controlplane/base/istio-cp-v115x.yaml << EOF
    apiVersion: servicemesh.cisco.com/v1alpha1
    kind: IstioControlPlane
    metadata:
      annotations:
        argocd.argoproj.io/sync-wave: "5"
      name: cp-v115x
      namespace: istio-system
    spec:
      containerImageConfiguration:
        imagePullPolicy: Always
        imagePullSecrets:
        - name: smm-pull-secret
      distribution: cisco
      istiod:
        deployment:
          env:
          - name: ISTIO_MULTIROOT_MESH
            value: "true"
          image: registry.eticloud.io/smm/istio-pilot:v1.15.3-bzc.0
      k8sResourceOverlays:
      - groupVersionKind:
          group: apps
          kind: Deployment
          version: v1
        objectKey:
          name: istiod-cp-v115x
          namespace: istio-system
        patches:
        - path: /spec/template/spec/containers/0/args/-
          type: replace
          value: --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256
      meshConfig:
        defaultConfig:
          envoyAccessLogService:
            address: smm-als.smm-system.svc.cluster.local:50600
            tcpKeepalive:
              interval: 10s
              probes: 3
              time: 10s
            tlsSettings:
              mode: ISTIO_MUTUAL
          holdApplicationUntilProxyStarts: true
          proxyMetadata:
            ISTIO_META_ALS_ENABLED: "true"
            PROXY_CONFIG_XDS_AGENT: "true"
          tracing:
            tlsSettings:
              mode: ISTIO_MUTUAL
            zipkin:
              address: smm-zipkin.smm-system.svc.cluster.local:59411
        enableEnvoyAccessLogService: true
        enableTracing: true
      meshExpansion:
        enabled: true
        gateway:
          deployment:
            podMetadata:
              labels:
                app: istio-meshexpansion-gateway
                istio: meshexpansiongateway
          service:
            ports:
            - name: tcp-smm-als-tls
              port: 50600
              protocol: TCP
              targetPort: 50600
            - name: tcp-smm-zipkin-tls
              port: 59411
              protocol: TCP
              targetPort: 59411
      meshID: mesh1
      mode: ACTIVE
      networkName: network1
      proxy:
        image: registry.eticloud.io/smm/istio-proxyv2:v1.15.3-bzc.1
      proxyInit:
        cni:
          daemonset:
            image: registry.eticloud.io/smm/istio-install-cni:v1.15.3-bzc.1
            # Uncomment all following lines for OpenShift installations
            # securityContext:
            #   privileged: true
          # enabled: true
          # binDir: /var/lib/cni/bin
          # chained: false
          # confDir: /etc/cni/multus/net.d
          # confFileName: istio-cni-cp-v115x-istio-system.conf
        image: registry.eticloud.io/smm/istio-proxyv2:v1.15.3-bzc.1
      sidecarInjector:
        deployment:
          image: registry.eticloud.io/smm/istio-sidecar-injector:v1.15.3-bzc.1
      version: 1.15.3
    EOF
    
  3. Create the kustomization.yaml file.

    cat > manifests/smm-controlplane/base/kustomization.yaml <<EOF
    apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    metadata:
      name: cluster-secrets
    
    
    resources:
    - cert-manager-namespace.yaml
    - istio-system-namespace.yaml
    - istio-cp-v115x.yaml
    - control-plane.yaml
    EOF
    
  4. Create the manifests/smm-controlplane/base/control-plane.yaml file. You don’t need to set the CLUSTER-NAME here, you will set it with the overlays customization.

    cat > manifests/smm-controlplane/base/control-plane.yaml <<EOF
    apiVersion: smm.cisco.com/v1alpha1
    kind: ControlPlane
    metadata:
      annotations:
        argocd.argoproj.io/sync-wave: "10"
      name: smm
    spec:
      certManager:
        namespace: cert-manager
      clusterName: CLUSTER-NAME
      clusterRegistry:
        enabled: true
        namespace: cluster-registry
      log: {}
      meshManager:
        enabled: true
        istio:
          enabled: true
          istioCRRef:
            name: cp-v115x
            namespace: istio-system
          operators:
            namespace: smm-system
        namespace: smm-system
      nodeExporter:
        enabled: true
        namespace: smm-system
        psp:
          enabled: false
        rbac:
          enabled: true
      oneEye: {}
      registryAccess:
        enabled: true
        imagePullSecretsController: {}
        namespace: smm-registry-access
        pullSecrets:
        - name: smm-registry.eticloud.io-pull-secret
          namespace: smm-registry-access
      repositoryOverride:
        host: registry.eticloud.io
        prefix: smm
      role: active
      smm:
        als:
          enabled: true
          log: {}
        application:
          enabled: true
          log: {}
        auth:
          mode: impersonation
        certManager:
          enabled: true
        enabled: true
        federationGateway:
          enabled: true
          name: smm
          service:
            enabled: true
            name: smm-federation-gateway
            port: 80
        federationGatewayOperator:
          enabled: true
        impersonation:
          enabled: true
        istio:
          revision: cp-v115x.istio-system
        leo:
          enabled: true
          log: {}
        log: {}
        namespace: smm-system
        prometheus:
          enabled: true
          replicas: 1
        prometheusOperator: {}
        releaseName: smm
        role: active
        sdm:
          enabled: false
        sre:
          enabled: true
        useIstioResources: true
    EOF
    
  5. Create the kustomization.yaml file for workload-cluster-1.

    cat > manifests/smm-controlplane/overlays/workload-cluster-1/kustomization.yaml <<EOF
    apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    
    bases:
      - ../../base
    
    patchesStrategicMerge:
      - istio-cp-v115x.yaml
      - control-plane.yaml
    EOF
    
  6. Set the clusterName by overriding some settings coming from the base configuration. Create the following files.

    cat > manifests/smm-controlplane/overlays/workload-cluster-1/control-plane.yaml <<EOF
    apiVersion: smm.cisco.com/v1alpha1
    kind: ControlPlane
    metadata:
      name: smm
    spec:
      clusterName: workload-cluster-1
      certManager:
        enabled: true
      smm:
        exposeDashboard:
          meshGateway:
            enabled: true
        auth:
          forceUnsecureCookies: true
          mode: anonymous
    EOF
    
    cat > manifests/smm-controlplane/overlays/workload-cluster-1/istio-cp-v115x.yaml <<EOF
    apiVersion: servicemesh.cisco.com/v1alpha1
    kind: IstioControlPlane
    metadata:
      annotations:
        argocd.argoproj.io/sync-wave: "5"
      name: cp-v115x
      namespace: istio-system
    spec:
      meshID: mesh1
      mode: ACTIVE
      networkName: network1
    EOF
    
  7. Create the kustomization.yaml file for workload-cluster-2.

    cat > manifests/smm-controlplane/overlays/workload-cluster-2/kustomization.yaml <<EOF
    apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    
    bases:
      - ../../base
    
    patchesStrategicMerge:
      - istio-cp-v115x.yaml
      - control-plane.yaml
    EOF
    
  8. Create the following files for workload-cluster-2. This sets the clusterName, and also overrides some settings of the base configuration.

    cat > manifests/smm-controlplane/overlays/workload-cluster-2/control-plane.yaml <<EOF
    apiVersion: smm.cisco.com/v1alpha1
    kind: ControlPlane
    metadata:
      name: smm
    spec:
      clusterName: workload-cluster-2
      role: passive
      smm:
        als:
          enabled: true
          log: {}
        application:
          enabled: false
          log: {}
        auth:
          mode: impersonation
        certManager:
          enabled: false
        enabled: true
        federationGateway:
          enabled: false
          name: smm
          service:
            enabled: true
            name: smm-federation-gateway
            port: 80
        federationGatewayOperator:
          enabled: true
        grafana:
          enabled: false
        impersonation:
          enabled: true
        istio:
          revision: cp-v115x.istio-system
        kubestatemetrics:
          enabled: true
        leo:
          enabled: false
          log: {}
        log: {}
        namespace: smm-system
        prometheus:
          enabled: true
          replicas: 1
          retentionTime: 8h
        prometheusOperator: {}
        releaseName: smm
        role: passive
        sdm:
          enabled: false
        sre:
          enabled: false
        tracing:
          enabled: true
        useIstioResources: false
        web:
          enabled: false
    EOF
    
    cat > manifests/smm-controlplane/overlays/workload-cluster-2/istio-cp-v115x.yaml <<EOF
    apiVersion: servicemesh.cisco.com/v1alpha1
    kind: IstioControlPlane
    metadata:
      annotations:
        argocd.argoproj.io/sync-wave: "5"
      name: cp-v115x
      namespace: istio-system
    spec:
      meshID: mesh1
      mode: PASSIVE
      networkName: workload-cluster-2
    EOF
    
  9. (Optional) If you want to change your active-passive deployment to active-active, complete this step. Otherwise, continue with the Commit deployment step. Run the following commands to modify the control planes of workload-cluster-2.

    cat > manifests/smm-controlplane/overlays/workload-cluster-2/control-plane.yaml <<EOF
    apiVersion: smm.cisco.com/v1alpha1
    kind: ControlPlane
    metadata:
      name: smm
    spec:
      clusterName: workload-cluster-2
      role: active
    EOF
    
    cat > manifests/smm-controlplane/overlays/workload-cluster-2/istio-cp-v115x.yaml <<EOF
    apiVersion: servicemesh.cisco.com/v1alpha1
    kind: IstioControlPlane
    metadata:
      annotations:
        argocd.argoproj.io/sync-wave: "5"
      name: cp-v115x
      namespace: istio-system
    spec:
      meshID: mesh1
      mode: ACTIVE
      networkName: network1
    EOF
    
  10. Create the smm-controlplane’s Argo CD ApplicationSet CR.

    cat > apps/smm-controlplane/app-set.yaml <<EOF
    apiVersion: argoproj.io/v1alpha1
    kind: ApplicationSet
    metadata:
      name: smm-cp-appset
      namespace: argocd
    spec:
      generators:
      - list:
          elements:
          - cluster: "${WORKLOAD_CLUSTER_1_CONTEXT}"
            path: "${WORKLOAD_CLUSTER_1_CONTEXT}"
          - cluster: "${WORKLOAD_CLUSTER_2_CONTEXT}"
            path: "${WORKLOAD_CLUSTER_2_CONTEXT}"
      template:
        metadata: 
          name: 'smm-cp-{{cluster}}'
          namespace: argocd
        spec:
          project: default
          source:
            repoURL: https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git
            targetRevision: HEAD
            path: manifests/smm-controlplane/overlays/{{path}}
          destination:
            name: '{{cluster}}'
          syncPolicy:
            automated:
              prune: true
              selfHeal: true
            retry:
              limit: 5
              backoff:
                duration: 5s
                maxDuration: 3m0s
                factor: 2
            syncOptions:
              - Validate=false
              - PruneLast=true
              - CreateNamespace=true
              - Replace=true
    EOF
    
  11. Commit and push the calisti-gitops repository.

    git add apps/smm-controlplane manifests
    
    git commit -m "add smm-controlplane app"
    
    git push
    
  12. Apply the Application manifests.

    kubectl apply -f "apps/smm-controlplane/app-set.yaml"
    
  13. Verify that the applications have been added to Argo CD and are healthy.

    argocd app list
    
  14. To create trust between workload-cluster-1 and workload-cluster-2, you must exchange the Secret CRs of the clusters. The cluster registry controller helps to form a group of Kubernetes clusters and synchronize any resources across those clusters arbitrarily.

    Create the following bash script and run it on workload-cluster-1.

    cat > export-secrets.sh <<EOF
    set -e
    
    kubectl --context workload-cluster-1 get cluster workload-cluster-1 -o yaml | kubectl --context workload-cluster-2 apply -f -
    kubectl --context workload-cluster-1 -n cluster-registry get secrets workload-cluster-1 -o yaml | kubectl --context workload-cluster-2 apply -f -
    
    kubectl --context workload-cluster-2 get cluster workload-cluster-2 -o yaml | kubectl --context workload-cluster-1 apply -f -
    kubectl --context workload-cluster-2 -n cluster-registry get secrets workload-cluster-2 -o yaml | kubectl --context workload-cluster-1 apply -f -
    
    echo "Exporting cluster and secrets CRs successfully."
    EOF
    
    chmod +x export-secrets.sh
    ./export-secrets.sh
    

    Expected output:

    cluster.clusterregistry.k8s.cisco.com/workload-cluster-1 created
    secret/workload-cluster-1 created
    cluster.clusterregistry.k8s.cisco.com/workload-cluster-2 created
    secret/workload-cluster-2 created
    Exporting cluster and secrets CRs successfully.
    
  15. Check that all pods are healthy and running in the smm-system namespace on workload-cluster-1 and workload-cluster-2. Note that it takes some time while the ControlPlane operator reconciles the resources.

    For workload-cluster-1:

    kubectl get pods -n smm-system --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_1_CONTEXT}"
    

    Expected output:

    NAME                                               READY   STATUS    RESTARTS        AGE
    istio-operator-v115x-7d77fc549f-fxmtd              2/2     Running   0               8m15s
    mesh-manager-0                                     2/2     Running   0          19m
    prometheus-node-exporter-9xnmj                     1/1     Running   0          17m
    prometheus-node-exporter-bf7g5                     1/1     Running   0          17m
    prometheus-node-exporter-cl69q                     1/1     Running   0          17m
    prometheus-smm-prometheus-0                        4/4     Running   0          18m
    smm-7f4d5d4fff-4dlcp                               2/2     Running   0          18m
    smm-7f4d5d4fff-59k7g                               2/2     Running   0          18m
    smm-als-7cc4bfb998-wjsr6                           2/2     Running   0          18m
    smm-authentication-569484f748-fj5zk                2/2     Running   0          18m
    smm-federation-gateway-6964fb956f-pb5pv            2/2     Running   0          18m
    smm-federation-gateway-operator-6664774695-9tmzj   2/2     Running   0          18m
    smm-grafana-59c54f67f4-9snc5                       3/3     Running   0          18m
    smm-health-75bf4f49c5-z9tqg                        2/2     Running   0          18m
    smm-health-api-7767d4f46-744wn                     2/2     Running   0          18m
    smm-ingressgateway-6ffdfc6d79-jttjz                1/1     Running   0          11m
    smm-ingressgateway-external-8c9bb9445-kjt8h        1/1     Running   0          11m
    smm-kubestatemetrics-86c6f96789-lp576              2/2     Running   0          18m
    smm-leo-67cd7d49b5-gmcvf                           2/2     Running   0          18m
    smm-prometheus-operator-ffbfb8b67-fwj6g            3/3     Running   0          18m
    smm-sre-alert-exporter-6654968479-fthk6            2/2     Running   0          18m
    smm-sre-api-86c9fb7cd7-mq7cm                       2/2     Running   0          18m
    smm-sre-controller-6889685f9-hxxh5                 2/2     Running   0          18m
    smm-tracing-5886d59dd-v8nb8                        2/2     Running   0          18m
    smm-vm-integration-5b89c4f7c9-wz4bt                2/2     Running   0          18m
    smm-web-d5b49c7f6-jgz7b                            3/3     Running   0          18m
    

    For workload-cluster-2:

    kubectl get pods -n smm-system --kubeconfig "${WORKLOAD_CLUSTER_2_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_2_CONTEXT}"
    

    Expected output:

    NAME                                       READY   STATUS    RESTARTS        AGE
    istio-operator-v115x-7d77fc549f-s5wnz      2/2     Running   0               9m5s
    mesh-manager-0                            2/2     Running   0             21m
    prometheus-node-exporter-fzdn4            1/1     Running   0             5m18s
    prometheus-node-exporter-rkbcl            1/1     Running   0             5m18s
    prometheus-node-exporter-x2mwp            1/1     Running   0             5m18s
    prometheus-smm-prometheus-0               3/3     Running   0             5m20s
    smm-ingressgateway-5db7859d45-6d6ns       1/1     Running   0             12m
    smm-kubestatemetrics-86c6f96789-j64q2     2/2     Running   0             19m
    smm-prometheus-operator-ffbfb8b67-zwqn2   3/3     Running   1 (11m ago)   19m
    
  16. Check the applications on the Argo CD Web UI.

    Argo CD Web UI Argo CD Web UI

At this point, you have successfully installed smm-operator and workload-cluster-1 and workload-cluster-2. You can open the Service Mesh Manager dashboard to check them, or deploy an application.

Deploy an application

If you want to deploy want to deploy an application into the service mesh, complete the following steps. The examples use the Service Mesh Manager demo application.

The file structure for the demo application looks like this:

.
├── README.md
├── apps
│   ├── demo-app
│   │   └── app-set.yaml
│   └── ...
...manifests
   └── demo-app
        ├── base
        │   ├── demo-app-namespace.yaml
        │   ├── demo-app.yaml
        │   └── kustomization.yaml
        └── overlays
            ├── workload-cluster-1
            │   ├── demo-app.yaml
            │   └── kustomization.yaml
            └── workload-cluster-2
                ├── demo-app.yaml
                └── kustomization.yaml
...
  1. Create the application manifest files.

    cat > manifests/demo-app/base/kustomization.yaml <<EOF
    apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    metadata:
      name: demo-app
    
    resources:
    - demo-app-namespace.yaml
    - demo-app.yaml
    EOF
    
    cat > manifests/demo-app/base/demo-app-namespace.yaml <<EOF
    apiVersion: v1
    kind: Namespace
    metadata:
      labels:
        app.kubernetes.io/instance: smm-demo
        app.kubernetes.io/name: smm-demo
        app.kubernetes.io/part-of: smm-demo
        app.kubernetes.io/version: 0.1.4
        istio.io/rev: cp-v115x.istio-system
      name: smm-demo
    EOF
    
    cat > manifests/demo-app/base/demo-app.yaml <<EOF
    apiVersion: smm.cisco.com/v1alpha1
    kind: DemoApplication
    metadata:
      name: smm-demo
      namespace: smm-demo
    spec:
      autoscaling:
        enabled: true
      controlPlaneRef:
        name: smm
    EOF
    
    cat > manifests/demo-app/overlays/workload-cluster-1/kustomization.yaml <<EOF
    apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    
    bases:
      - ../../base
    
    patchesStrategicMerge:
      - demo-app.yaml
    EOF
    
    cat > manifests/demo-app/overlays/workload-cluster-1/demo-app.yaml <<EOF
    apiVersion: smm.cisco.com/v1alpha1
    kind: DemoApplication
    metadata:
      name: smm-demo
      namespace: smm-demo
    spec:
      autoscaling:
        enabled: true
      controlPlaneRef:
        name: smm
      deployIstioResources: true
      deploySLOResources: true
      enabled: true
      enabledComponents:
      - frontpage
      - catalog
      - bookings
      - postgresql
      istio:
        revision: cp-v115x.istio-system
      load:
        enabled: true
        maxRPS: 30
        minRPS: 10
        swingPeriod: 1380000000000
      replicas: 1
      resources:
        limits:
          cpu: "2"
          memory: 192Mi
        requests:
          cpu: 40m
          memory: 64Mi
    EOF
    
    cat > manifests/demo-app/overlays/workload-cluster-2/kustomization.yaml <<EOF
    apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    
    bases:
      - ../../base
    
    patchesStrategicMerge:
      - demo-app.yaml
    EOF
    
    cat > manifests/demo-app/overlays/workload-cluster-2/demo-app.yaml <<EOF
    apiVersion: smm.cisco.com/v1alpha1
    kind: DemoApplication
    metadata:
      name: smm-demo
      namespace: smm-demo
    spec:
      autoscaling:
        enabled: true
      controlPlaneRef:
        name: smm
      deployIstioResources: false
      deploySLOResources: false
      enabled: true
      enabledComponents:
      - movies
      - payments
      - notifications
      - analytics
      - database
      - mysql
      istio:
        revision: cp-v115x.istio-system
      replicas: 1
      resources:
        limits:
          cpu: "2"
          memory: 192Mi
        requests:
          cpu: 40m
          memory: 64Mi
    EOF
    
  2. Create the Demo application ApplicationSet.

    cat > apps/demo-app/app-set.yaml <<EOF
    apiVersion: argoproj.io/v1alpha1
    kind: ApplicationSet
    metadata:
      name: demo-app-appset
      namespace: argocd
    spec:
      generators:
      - list:
          elements:
          - cluster: "${WORKLOAD_CLUSTER_1_CONTEXT}"
            path: "${WORKLOAD_CLUSTER_1_CONTEXT}"
          - cluster: "${WORKLOAD_CLUSTER_2_CONTEXT}"
            path: "${WORKLOAD_CLUSTER_2_CONTEXT}"
      template:
        metadata: 
          name: 'demo-app-{{cluster}}'
          namespace: argocd
        spec:
          project: default
          source:
            repoURL: https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git
            targetRevision: HEAD
            path: manifests/demo-app/overlays/{{path}}
          destination:
            name: '{{cluster}}'
          syncPolicy:
            automated:
              prune: true
              selfHeal: true
            retry:
              limit: 5
              backoff:
                duration: 5s
                maxDuration: 3m0s
                factor: 2
            syncOptions:
              - Validate=false
              - PruneLast=true
              - CreateNamespace=true
              - Replace=true
    EOF
    
  3. Commit and push the calisti-gitops repository.

    git add apps/demo-app manifests
    git commit -m "add demo application"
    git push origin
    
  4. Deploy the demo application on the clusters.

    kubectl apply -f apps/demo-app/app-set.yaml
    
  5. Wait until all the pods in the smm-demo namespace are up and running.

    kubectl get pods -n smm-demo --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_1_CONTEXT}"
    

    Expected output:

    NAME                           READY   STATUS    RESTARTS   AGE
    bombardier-5f59948978-zx99c    2/2     Running   0          3m21s
    bookings-v1-68dd865855-fdcxk   2/2     Running   0          3m21s
    catalog-v1-6d564bbcb8-qmhbx    2/2     Running   0          3m21s
    frontpage-v1-b4686759b-fhfmv   2/2     Running   0          3m21s
    postgresql-7cf55cd596-grs46    2/2     Running   0          3m21s
    
    kubectl get pods -n smm-demo --kubeconfig "${WORKLOAD_CLUSTER_2_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_2_CONTEXT}"
    

    Expected output:

    NAME                                READY   STATUS    RESTARTS   AGE
    analytics-v1-799d668f84-p4nkk       2/2     Running   0          3m58s
    database-v1-6896cd4b59-9xxgg        2/2     Running   0          3m58s
    movies-v1-9594fff5f-8hv9l           2/2     Running   0          3m58s
    movies-v2-5559c5567c-2279n          2/2     Running   0          3m58s
    movies-v3-649b99d977-nkdxc          2/2     Running   0          3m58s
    mysql-669466cc8d-bs4s9              2/2     Running   0          3m58s
    notifications-v1-79bc79c89b-4bbss   2/2     Running   0          3m58s
    payments-v1-547884bfdf-dg2dm        2/2     Running   0          3m58s
    
  6. Check the applications on the Argo CD web UI.

    Argo CD Web UI Argo CD Web UI

  7. Open the Service Mesh Manager web interface, select MENU > TOPOLOGY, then select the smm-demo namespace.

    Demo Application Topology Demo Application Topology

Access the Service Mesh Manager dashboard

  1. You can access the Service Mesh Manager dashboard via the smm-ingressgateway-external LoadBalancer external-ip-or-hostname address. Run the following command to retrieve the IP address:

    kubectl get services -n smm-system smm-ingressgateway-external --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_1_CONTEXT}"
    

    Expected output:

    NAME                          TYPE           CLUSTER-IP   EXTERNAL-IP                PORT(S)        AGE
    smm-ingressgateway-external   LoadBalancer   10.0.0.199   external-ip-or-hostname    80:32505/TCP   2m28s
    
  2. Open the Service Mesh Manager dashboard using one of the following methods:

    • Open the http://<external-ip-or-hostname> URL in your browser.

    • Run the following command to open the dashboard with your default browser:

      # Exactly one of hostname or IP will be available and used for the remote URL.
      open http://$(kubectl get services -n smm-system smm-ingressgateway-external -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{.status.loadBalancer.ingress[0].ip}' --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_1_CONTEXT}")
      
    • If you have installed the Service Mesh Manager CLI on your machine, run the following command to open the Service Mesh Manager Dashboard in the default browser.

      smm dashboard --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_1_CONTEXT}"
      

      Expected output:

      ✓ validate-kubeconfig ❯ checking cluster reachability...
      ✓ opening Service Mesh Manager at http://127.0.0.1:50500
      
  3. Check the deployments on the dashboard, for example, on the MENU > Overview, MENU > MESH, and MENU > TOPOLOGY pages.

Service Mesh Manager Overview Service Mesh Manager Overview

Service Mesh Manager Mesh Service Mesh Manager Mesh

Service Mesh Manager Topology Service Mesh Manager Topology