Install SMM - GitOps - multi-cluster
This guide details how to set up a multi-cluster Service Mesh Manager scenario in a GitOps environment for Service Mesh Manager using Argo CD. The same principles can be used for other tools as well.
CAUTION:
Do not push the secrets directly into the git repository, especially when it is a public repository. Argo CD provides solutions to keep secrets safe.Architecture
Service Mesh Manager supports multiple mesh topologies, so you can use the one that best fits your use cases. In multi-cluster configurations it provides automatic locality load-balancing.
The high level architecture for Argo CD with a multi-cluster Service Mesh Manager setup consists of the following components:
- A git repository that stores the various charts and manifests,
- a management cluster that runs the Argo CD server, and
- the Service Mesh Manager clusters managed by Argo CD.
Deployment models
When deploying Service Mesh Manager in a multi-cluster scenario you can deploy Service Mesh Manager in an active-passive model. For details on Service Mesh Manager clusters and their relationship to Istio clusters, see Istio clusters and SMM clusters.
Prerequisites
- A free registration for the Service Mesh Manager download page
- A Kubernetes cluster to deploy Argo CD on (called
management-cluster
in the examples). - Two Kubernetes clusters to deploy Service Mesh Manager on (called
workload-cluster-1
andworkload-cluster-2
in the examples).
CAUTION:
Supported providers and Kubernetes versions
The cluster must run a Kubernetes version that Service Mesh Manager supports: Kubernetes 1.21, 1.22, 1.23, 1.24.
Service Mesh Manager is tested and known to work on the following Kubernetes providers:
- Amazon Elastic Kubernetes Service (Amazon EKS)
- Google Kubernetes Engine (GKE)
- Azure Kubernetes Service (AKS)
- Red Hat OpenShift 4.11
- On-premises installation of stock Kubernetes with load balancer support (and optionally PVCs for persistence)
Calisti resource requirements
Make sure that your Kubernetes or OpenShift cluster has sufficient resources to install Calisti. The following table shows the number of resources needed on the cluster:
Resource | Required |
---|---|
CPU | - 32 vCPU in total - 4 vCPU available for allocation per worker node (If you are testing on a cluster at a cloud provider, use nodes that have at least 4 CPUs, for example, c5.xlarge on AWS.) |
Memory | - 64 GiB in total - 4 GiB available for allocation per worker node for the Kubernetes cluster (8 GiB in case of the OpenShift cluster) |
Storage | 12 GB of ephemeral storage on the Kubernetes worker nodes (for Traces and Metrics) |
These minimum requirements need to be available for allocation within your cluster, in addition to the requirements of any other loads running in your cluster (for example, DaemonSets and Kubernetes node-agents). If Kubernetes cannot allocate sufficient resources to Service Mesh Manager, some pods will remain in Pending state, and Service Mesh Manager will not function properly.
Enabling additional features, such as High Availability increases this value.
The default installation, when enough headroom is available in the cluster, should be able to support at least 150 running Pods
with the same amount of Services
. For setting up Service Mesh Manager for bigger workloads, see scaling Service Mesh Manager.
Install Argo CD
Complete the following steps to install Argo CD on the management cluster.
Set up the environment
-
Set the KUBECONFIG location and context name for the
management-cluster
cluster.MANAGEMENT_CLUSTER_KUBECONFIG=management_cluster_kubeconfig.yaml MANAGEMENT_CLUSTER_CONTEXT=management-cluster kubectl config --kubeconfig "${MANAGEMENT_CLUSTER_KUBECONFIG}" get-contexts "${MANAGEMENT_CLUSTER_CONTEXT}"
Expected output:
CURRENT NAME CLUSTER AUTHINFO NAMESPACE * management-cluster management-cluster
-
Set the KUBECONFIG location and context name for the
workload-cluster-1
cluster.WORKLOAD_CLUSTER_1_KUBECONFIG=workload_cluster_1_kubeconfig.yaml WORKLOAD_CLUSTER_1_CONTEXT=workload-cluster-1 kubectl config --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" get-contexts "${WORKLOAD_CLUSTER_1_CONTEXT}"
Expected output:
CURRENT NAME CLUSTER AUTHINFO NAMESPACE * workload-cluster-1 workload-cluster-1
Repeat this step for any additional workload clusters you want to use.
-
Add the cluster configurations to KUBECONFIG. Include any additional workload clusters you want to use.
KUBECONFIG=$KUBECONFIG:$MANAGEMENT_CLUSTER_KUBECONFIG:$WORKLOAD_CLUSTER_1_KUBECONFIG
-
Make sure the
management-cluster
Kubernetes context is the current context.kubectl config use-context "${MANAGEMENT_CLUSTER_CONTEXT}"
Expected output:
Switched to context "management-cluster".
Install Argo CD Server
-
Create the
argocd
namespace.kubectl create namespace argocd
Expected output:
namespace/argocd created
-
On OpenShift: Run the following command to grant the service accounts access to the
argocd
namespace.oc adm policy add-scc-to-group privileged system:serviceaccounts:argocd
Expected output:
clusterrole.rbac.authorization.k8s.io/system:openshift:scc:privileged added: "system:serviceaccounts:argocd"
-
Deploy Argo CD.
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
-
Wait until the installation is complete, then check that the Argo CD pods are up and running.
kubectl get pods -n argocd
The output should be similar to:
NAME READY STATUS RESTARTS AGE pod/argocd-application-controller-0 1/1 Running 0 7h59m pod/argocd-applicationset-controller-78b8b554f9-pgwbl 1/1 Running 0 7h59m pod/argocd-dex-server-6bbc85c688-8p7zf 1/1 Running 0 16h pod/argocd-notifications-controller-75847756c5-dbbm5 1/1 Running 0 16h pod/argocd-redis-f4cdbff57-wcpxh 1/1 Running 0 7h59m pod/argocd-repo-server-d5c7f7ffb-c8962 1/1 Running 0 7h59m pod/argocd-server-76497676b-pnvf4 1/1 Running 0 7h59m
-
For the Argo CD UI, set the
argocd-server service
type toLoadBalancer
.kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'
Expected output:
service/argocd-server patched
-
Patch the App of Apps health check in Argo CD configuration to ignore diffs of controller/operator managed fields. For details about this patch, see the Argo CD documentation sections Resource Health and Diffing Customization.
Apply the new Argo CD health check configurations:
kubectl apply -f - <<EOF apiVersion: v1 kind: ConfigMap metadata: name: argocd-cm namespace: argocd labels: app.kubernetes.io/name: argocd-cm app.kubernetes.io/part-of: argocd data: # App of app health check resource.customizations.health.argoproj.io_Application: | hs = {} hs.status = "Progressing" hs.message = "" if obj.status ~= nil then if obj.status.health ~= nil then hs.status = obj.status.health.status if obj.status.health.message ~= nil then hs.message = obj.status.health.message end end end return hs # Ignoring RBAC changes made by AggregateRoles resource.compareoptions: | # disables status field diffing in specified resource types ignoreAggregatedRoles: true # disables status field diffing in specified resource types # 'crd' - CustomResourceDefinition-s (default) # 'all' - all resources # 'none' - disabled ignoreResourceStatusField: all EOF
Expected output:
configmap/argocd-cm configured
-
Get the initial password for the
admin
user.kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo
Expected output:
argocd-admin-password
-
Check the
external-ip-or-hostname
address of theargocd-server
service.kubectl get service -n argocd argocd-server
The output should be similar to:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE argocd-server LoadBalancer 10.108.14.130 external-ip-or-hostname 80:31306/TCP,443:30063/TCP 7d13h
-
Open the
https://external-ip-or-hostname
URL and log in to the Argo CD server using the password received in the previous step.# Exactly one of hostname or IP will be available and used for the remote URL. open https://$(kubectl get service -n argocd argocd-server -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{.status.loadBalancer.ingress[0].ip}')
Install Argo CD CLI
-
Install Argo CD CLI on your computer. For details, see the Argo CD documentation.
-
Log in with the CLI:
# Exactly one of hostname or IP will be available and used for the remote URL. argocd login $(kubectl get service -n argocd argocd-server -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{.status.loadBalancer.ingress[0].ip}') --insecure --username admin --password $(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d)
Expected output:
'admin:login' logged in successfully
For more details about Argo CD installation, see the Argo CD getting started guide.
Register clusters
-
Register the clusters that will run Service Mesh Manager in Argo CD. In this example, register
workload-cluster-1
andworkload-cluster-2
using one of the following methods.-
Register the cluster from the command line by running:
argocd cluster add --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" "${WORKLOAD_CLUSTER_1_CONTEXT}"
Expected output:
WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `workload-cluster-1` with full cluster level privileges. Do you want to continue [y/N]? y INFO[0005] ServiceAccount "argocd-manager" created in namespace "kube-system" INFO[0005] ClusterRole "argocd-manager-role" created INFO[0005] ClusterRoleBinding "argocd-manager-role-binding" created INFO[0011] Created bearer token secret for ServiceAccount "argocd-manager" Cluster 'https://workload-cluster-1-ip-or-hostname' added
argocd cluster add --kubeconfig "${WORKLOAD_CLUSTER_2_KUBECONFIG}" "${WORKLOAD_CLUSTER_2_CONTEXT}"
Expected output:
WARNING: This will create a service account `argocd-manager` on the cluster referenced by context `workload-cluster-2` with full cluster level privileges. Do you want to continue [y/N]? y INFO[0005] ServiceAccount "argocd-manager" created in namespace "kube-system" INFO[0005] ClusterRole "argocd-manager-role" created INFO[0005] ClusterRoleBinding "argocd-manager-role-binding" created INFO[0011] Created bearer token secret for ServiceAccount "argocd-manager" Cluster 'https://workload-cluster-2-ip-or-hostname' added
-
Alternatively, you can register clusters declaratively as Kubernetes secrets. Modify the following command for your environment and apply it. For details, see the Argo CD documentation.
WORKLOAD_CLUSTER_1_IP="https://workload-cluster-1-IP" ARGOCD_BEARER_TOKEN="authentication-token" ARGOCD_CA_B64="base64 encoded certificate" ; kubectl apply -f - <<EOF apiVersion: v1 kind: Secret metadata: name: workload-cluster-1-secret labels: argocd.argoproj.io/secret-type: cluster type: Opaque stringData: name: workload-cluster-1 server: "${WORKLOAD_CLUSTER_1_IP}" config: | { "bearerToken": "${ARGOCD_BEARER_TOKEN}", "tlsClientConfig": { "insecure": false, "caData": "${ARGOCD_CA_B64}" } } EOF
WORKLOAD_CLUSTER_2_IP="https://workload-cluster-2-IP" ARGOCD_BEARER_TOKEN="authentication-token" ARGOCD_CA_B64="base64 encoded certificate" ; kubectl apply -f - <<EOF apiVersion: v1 kind: Secret metadata: name: workload-cluster-2-secret labels: argocd.argoproj.io/secret-type: cluster type: Opaque stringData: name: workload-cluster-2 server: "${WORKLOAD_CLUSTER_2_IP}" config: | { "bearerToken": "${ARGOCD_BEARER_TOKEN}", "tlsClientConfig": { "insecure": false, "caData": "${ARGOCD_CA_B64}" } } EOF
-
-
Make sure that the cluster is registered in Argo CD by running the following command:
argocd cluster list
The output should be similar to:
SERVER NAME VERSION STATUS MESSAGE PROJECT https://kubernetes.default.svc in-cluster Unknown Cluster has no applications and is not being monitored. https://workload-cluster-1-ip-or-hostname workload-cluster-1 Unknown Cluster has no applications and is not being monitored. https://workload-cluster-2-ip-or-hostname workload-cluster-2 Unknown Cluster has no applications and is not being monitored.
Prepare Git repository
-
Create an empty repository called
calisti-gitops
on GitHub (or another provider that Argo CD supports) and initialize it with a README.md file so that you can clone the repository. Because Service Mesh Manager credentials will be stored in this repository, make it a private repository.GITHUB_ID="github-id" GITHUB_REPOSITORY_NAME="calisti-gitops"
-
Obtain a personal access token to the repository (on GitHub, see Creating a personal access token), that has the following permissions:
- admin:org_hook
- admin:repo_hook
- read:org
- read:public_key
- repo
-
Log in with your personal access token with
git
.export GH_TOKEN="github-personal-access-token" # Note: this environment variable needs to be exported so the `git` binary is going to use it automatically for authentication.
-
Clone the repository into your local workspace, for example:
git clone "https://${GITHUB_ID}:${GH_TOKEN}@github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git"
Expected output:
Cloning into 'calisti-gitops'... remote: Enumerating objects: 144, done. remote: Counting objects: 100% (144/144), done. remote: Compressing objects: 100% (93/93), done. remote: Total 144 (delta 53), reused 135 (delta 47), pack-reused 0 Receiving objects: 100% (144/144), 320.08 KiB | 746.00 KiB/s, done. Resolving deltas: 100% (53/53), done.
-
Add the repository to Argo CD by running the following command. Alternatively, you can add it on Argo CD Web UI.
argocd repo add "https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git" --name "${GITHUB_REPOSITORY_NAME}" --username "${GITHUB_ID}" --password "${GH_TOKEN}"
Expected output:
Repository 'https://github.com/github-id/calisti-gitops.git' added
-
Verify that the repository is connected by running:
argocd repo list
In the output, Status should be Successful:
TYPE NAME REPO INSECURE OCI LFS CREDS STATUS MESSAGE PROJECT git calisti-gitops https://github.com/github-id/calisti-gitops.git false false false true Successful
-
Change into the directory of the cloned repository (for example,
calisti-gitops
) and create the following directories.cd "${GITHUB_REPOSITORY_NAME}"
mkdir -p apps/smm-controlplane apps/smm-operator apps/demo-app charts manifests/smm-controlplane/base manifests/smm-controlplane/overlays/workload-cluster-1 manifests/smm-controlplane/overlays/workload-cluster-2 manifests/demo-app/base manifests/demo-app/overlays/workload-cluster-1 manifests/demo-app/overlays/workload-cluster-2
The final structure of the repository will look like this:
. ├── README.md ├── apps │ ├── smm-controlplane │ │ └── app-set.yaml │ ├── smm-operator │ │ └── app-set.yaml │ └── demo-app │ └── app-set.yaml ├── charts │ └── smm-operator │ ├── Chart.yaml │ └── ... ├── export-secrets.sh └── manifests ├── smm-controlplane │ ├── base │ │ ├── control-plane.yaml │ │ ├── cert-manager-namespace.yaml │ │ ├── istio-system-namespace.yaml │ │ ├── istio-cp-v115x.yaml │ │ └── kustomization.yaml │ └── overlays │ ├── workload-cluster-1 │ │ ├── control-plane.yaml │ │ ├── istio-cp-v115x.yaml │ │ └── kustomization.yaml │ └── workload-cluster-2 │ ├── control-plane.yaml │ ├── istio-cp-v115x.yaml │ └── kustomization.yaml └── demo-app ├── base │ ├── demo-app-namespace.yaml │ ├── demo-app.yaml │ └── kustomization.yaml └── overlays ├── workload-cluster-1 │ ├── demo-app.yaml │ └── kustomization.yaml └── workload-cluster-2 ├── demo-app.yaml └── kustomization.yaml
- The
apps
folder contains the Argo CD Application of thesmm-operator
, thesmm-controlplane
, and thedemo-app
. - The
charts
folder contains the Helm chart of thesmm-operator
. - The
manifests/demo-app
folder contains the manifest files of the demo application that represents your business application. - The
manifests/smm-controlplane
folder contains the manifest files of the SMM ControlPlane.
- The
Prepare the helm charts
-
You need an active Service Mesh Manager registration to download the Service Mesh Manager charts and images. You can sign up for free, or obtain Enterprise credentials on the official Cisco Service Mesh Manager page. After registration, you can obtain your username and password from the Download Center. Set them as environment variables.
CALISTI_USERNAME="<your-calisti-username>"
CALISTI_PASSWORD="<your-calisti-password>"
-
Download the
smm-operator
chart fromregistry.eticloud.io
into thecharts
directory of your Service Mesh Manager GitOps repository and extract it. Run the following commands:export HELM_EXPERIMENTAL_OCI=1 # Needed prior to Helm version 3.8.0 echo "${CALISTI_PASSWORD}" | helm registry login registry.eticloud.io -u "${CALISTI_USERNAME}" --password-stdin
Expected output:
Login Succeeded
helm pull oci://registry.eticloud.io/smm-charts/smm-operator --destination ./charts/ --untar --version 1.12.1
Expected output:
Pulled: registry.eticloud.io/smm-charts/smm-operator:latest-stable-version Digest: sha256:someshadigest
Deploy Service Mesh Manager
Deploy the smm-operator application set
Complete the following steps to deploy the smm-operator chart using Argo CD.
-
Create the
smm-operator
’s Argo CDApplicationSet
CR. Argo CDApplicationSet
is perfect for deploying the same application on to different clusters. You can use list generators with cluster data in theApplicationSet
.Before running the following command, edit it if needed:
- If you are not using a GitHub repository, set the
repoURL
field to your repository. -
For multi-cluster setups, the Kubernetes API server address of one cluster must be reachable from other clusters. The API server addresses are private for certain clusters (for example, OpenShift) and not reachable by default from other clusters. In such case, use the
PUBLIC_API_SERVER_ENDPOINT_ADDRESS
variable to provide an address that’s reachable from the other clusters. This can be a public address, or one that’s routable from the other clusters.
PUBLIC_API_SERVER_ENDPOINT_ADDRESS_1="" PUBLIC_API_SERVER_ENDPOINT_ADDRESS_2="" ; cat > "apps/smm-operator/app-set.yaml" <<EOF # apps/smm-operator/app-set.yaml apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: smm-operator-appset namespace: argocd spec: generators: - list: elements: - cluster: "${WORKLOAD_CLUSTER_1_CONTEXT}" apiServerEndpointAddress: "${PUBLIC_API_SERVER_ENDPOINT_ADDRESS_1}" - cluster: "${WORKLOAD_CLUSTER_2_CONTEXT}" apiServerEndpointAddress: "${PUBLIC_API_SERVER_ENDPOINT_ADDRESS_2}" template: metadata: name: 'smm-operator-{{cluster}}' namespace: argocd spec: project: default source: repoURL: https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git targetRevision: HEAD path: charts/smm-operator helm: parameters: - name: "global.ecr.enabled" value: 'false' - name: "global.basicAuth.username" value: "${CALISTI_USERNAME}" - name: "global.basicAuth.password" value: "${CALISTI_PASSWORD}" - name: "apiServerEndpointAddress" value: '{{apiServerEndpointAddress}}' destination: namespace: smm-registry-access name: '{{cluster}}' ignoreDifferences: - kind: ValidatingWebhookConfiguration group: admissionregistration.k8s.io jsonPointers: - /webhooks syncPolicy: automated: prune: true selfHeal: true retry: limit: 5 backoff: duration: 5s maxDuration: 3m0s factor: 2 syncOptions: - Validate=false - PruneLast=true - CreateNamespace=true - Replace=true EOF
- If you are not using a GitHub repository, set the
-
Commit and push the
calisti-gitops
repository.git add apps/smm-operator charts/smm-operator
git commit -m "add smm-operator app"
git push
-
Apply the Application manifests.
kubectl apply -f "apps/smm-operator/app-set.yaml"
-
Verify that the applications have been added to Argo CD and are healthy.
argocd app list
Expected output:
NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS REPO PATH TARGET argocd/smm-operator-workload-cluster-1 workload-cluster-1 smm-registry-access default Synced Healthy Auto-Prune <none> https://github.com/<github-user>/calisti-gitops-multi-cluster.git charts/smm-operator HEAD argocd/smm-operator-workload-cluster-2 workload-cluster-2 smm-registry-access default Synced Healthy Auto-Prune <none> https://github.com/<github-user>/calisti-gitops-multi-cluster.git charts/smm-operator HEAD
-
Check the
smm-operator
application on the Argo CD Web UI.
Deploy the smm-controlplane application
The following steps show you how to deploy the smm-controlplane
application as an active-passive deployment. To create and active-active deployment, follow the same steps, there is an optional step that changes the active-passive deployment to active-active. For details, see Deployment models.
Deploy the smm-controlplane
application using Kustomize: the active
on workload-cluster-1
and the passive
on workload-cluster-2
. The active cluster receives every component, while the passive cluster only a few required components. This part of the repository will look like this:
└── manifests
├── smm-controlplane
│ ├── base
│ │ ├── control-plane.yaml
│ │ ├── cert-manager-namespace.yaml
│ │ ├── istio-system-namespace.yaml
│ │ ├── istio-cp-v115x.yaml
│ │ └── kustomization.yaml
│ └── overlays
│ ├── workload-cluster-1
│ │ ├── control-plane.yaml
│ │ ├── istio-cp-v115x.yaml
│ │ └── kustomization.yaml
│ └── workload-cluster-2
│ ├── control-plane.yaml
│ ├── istio-cp-v115x.yaml
│ └── kustomization.yaml
-
Create the following namespaces files.
cat > manifests/smm-controlplane/base/cert-manager-namespace.yaml <<EOF apiVersion: v1 kind: Namespace metadata: annotations: argocd.argoproj.io/sync-wave: "1" name: cert-manager EOF
cat > manifests/smm-controlplane/base/istio-system-namespace.yaml << EOF apiVersion: v1 kind: Namespace metadata: annotations: argocd.argoproj.io/sync-wave: "2" name: istio-system EOF
-
Create the IstioControlPlane file.
cat > manifests/smm-controlplane/base/istio-cp-v115x.yaml << EOF apiVersion: servicemesh.cisco.com/v1alpha1 kind: IstioControlPlane metadata: annotations: argocd.argoproj.io/sync-wave: "5" name: cp-v115x namespace: istio-system spec: containerImageConfiguration: imagePullPolicy: Always imagePullSecrets: - name: smm-pull-secret distribution: cisco istiod: deployment: env: - name: ISTIO_MULTIROOT_MESH value: "true" image: registry.eticloud.io/smm/istio-pilot:v1.15.3-bzc.0 k8sResourceOverlays: - groupVersionKind: group: apps kind: Deployment version: v1 objectKey: name: istiod-cp-v115x namespace: istio-system patches: - path: /spec/template/spec/containers/0/args/- type: replace value: --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256 meshConfig: defaultConfig: envoyAccessLogService: address: smm-als.smm-system.svc.cluster.local:50600 tcpKeepalive: interval: 10s probes: 3 time: 10s tlsSettings: mode: ISTIO_MUTUAL holdApplicationUntilProxyStarts: true proxyMetadata: ISTIO_META_ALS_ENABLED: "true" PROXY_CONFIG_XDS_AGENT: "true" tracing: tlsSettings: mode: ISTIO_MUTUAL zipkin: address: smm-zipkin.smm-system.svc.cluster.local:59411 enableEnvoyAccessLogService: true enableTracing: true meshExpansion: enabled: true gateway: deployment: podMetadata: labels: app: istio-meshexpansion-gateway istio: meshexpansiongateway service: ports: - name: tcp-smm-als-tls port: 50600 protocol: TCP targetPort: 50600 - name: tcp-smm-zipkin-tls port: 59411 protocol: TCP targetPort: 59411 meshID: mesh1 mode: ACTIVE networkName: network1 proxy: image: registry.eticloud.io/smm/istio-proxyv2:v1.15.3-bzc.1 proxyInit: cni: daemonset: image: registry.eticloud.io/smm/istio-install-cni:v1.15.3-bzc.1 # Uncomment all following lines for OpenShift installations # securityContext: # privileged: true # enabled: true # binDir: /var/lib/cni/bin # chained: false # confDir: /etc/cni/multus/net.d # confFileName: istio-cni-cp-v115x-istio-system.conf image: registry.eticloud.io/smm/istio-proxyv2:v1.15.3-bzc.1 sidecarInjector: deployment: image: registry.eticloud.io/smm/istio-sidecar-injector:v1.15.3-bzc.1 version: 1.15.3 EOF
-
Create the
kustomization.yaml
file.cat > manifests/smm-controlplane/base/kustomization.yaml <<EOF apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization metadata: name: cluster-secrets resources: - cert-manager-namespace.yaml - istio-system-namespace.yaml - istio-cp-v115x.yaml - control-plane.yaml EOF
-
Create the
manifests/smm-controlplane/base/control-plane.yaml
file. You don’t need to set theCLUSTER-NAME
here, you will set it with theoverlays
customization.cat > manifests/smm-controlplane/base/control-plane.yaml <<EOF apiVersion: smm.cisco.com/v1alpha1 kind: ControlPlane metadata: annotations: argocd.argoproj.io/sync-wave: "10" name: smm spec: certManager: namespace: cert-manager clusterName: CLUSTER-NAME clusterRegistry: enabled: true namespace: cluster-registry log: {} meshManager: enabled: true istio: enabled: true istioCRRef: name: cp-v115x namespace: istio-system operators: namespace: smm-system namespace: smm-system nodeExporter: enabled: true namespace: smm-system psp: enabled: false rbac: enabled: true oneEye: {} registryAccess: enabled: true imagePullSecretsController: {} namespace: smm-registry-access pullSecrets: - name: smm-registry.eticloud.io-pull-secret namespace: smm-registry-access repositoryOverride: host: registry.eticloud.io prefix: smm role: active smm: als: enabled: true log: {} application: enabled: true log: {} auth: mode: impersonation certManager: enabled: true enabled: true federationGateway: enabled: true name: smm service: enabled: true name: smm-federation-gateway port: 80 federationGatewayOperator: enabled: true impersonation: enabled: true istio: revision: cp-v115x.istio-system leo: enabled: true log: {} log: {} namespace: smm-system prometheus: enabled: true replicas: 1 prometheusOperator: {} releaseName: smm role: active sdm: enabled: false sre: enabled: true useIstioResources: true EOF
-
Create the
kustomization.yaml
file forworkload-cluster-1
.cat > manifests/smm-controlplane/overlays/workload-cluster-1/kustomization.yaml <<EOF apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization bases: - ../../base patchesStrategicMerge: - istio-cp-v115x.yaml - control-plane.yaml EOF
-
Set the
clusterName
by overriding some settings coming from thebase
configuration. Create the following files.cat > manifests/smm-controlplane/overlays/workload-cluster-1/control-plane.yaml <<EOF apiVersion: smm.cisco.com/v1alpha1 kind: ControlPlane metadata: name: smm spec: clusterName: workload-cluster-1 certManager: enabled: true smm: exposeDashboard: meshGateway: enabled: true auth: forceUnsecureCookies: true mode: anonymous EOF
cat > manifests/smm-controlplane/overlays/workload-cluster-1/istio-cp-v115x.yaml <<EOF apiVersion: servicemesh.cisco.com/v1alpha1 kind: IstioControlPlane metadata: annotations: argocd.argoproj.io/sync-wave: "5" name: cp-v115x namespace: istio-system spec: meshID: mesh1 mode: ACTIVE networkName: network1 EOF
-
Create the
kustomization.yaml
file forworkload-cluster-2
.cat > manifests/smm-controlplane/overlays/workload-cluster-2/kustomization.yaml <<EOF apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization bases: - ../../base patchesStrategicMerge: - istio-cp-v115x.yaml - control-plane.yaml EOF
-
Create the following files for
workload-cluster-2
. This sets theclusterName
, and also overrides some settings of thebase
configuration.cat > manifests/smm-controlplane/overlays/workload-cluster-2/control-plane.yaml <<EOF apiVersion: smm.cisco.com/v1alpha1 kind: ControlPlane metadata: name: smm spec: clusterName: workload-cluster-2 role: passive smm: als: enabled: true log: {} application: enabled: false log: {} auth: mode: impersonation certManager: enabled: false enabled: true federationGateway: enabled: false name: smm service: enabled: true name: smm-federation-gateway port: 80 federationGatewayOperator: enabled: true grafana: enabled: false impersonation: enabled: true istio: revision: cp-v115x.istio-system kubestatemetrics: enabled: true leo: enabled: false log: {} log: {} namespace: smm-system prometheus: enabled: true replicas: 1 retentionTime: 8h prometheusOperator: {} releaseName: smm role: passive sdm: enabled: false sre: enabled: false tracing: enabled: true useIstioResources: false web: enabled: false EOF
cat > manifests/smm-controlplane/overlays/workload-cluster-2/istio-cp-v115x.yaml <<EOF apiVersion: servicemesh.cisco.com/v1alpha1 kind: IstioControlPlane metadata: annotations: argocd.argoproj.io/sync-wave: "5" name: cp-v115x namespace: istio-system spec: meshID: mesh1 mode: PASSIVE networkName: workload-cluster-2 EOF
-
(Optional) If you want to change your active-passive deployment to active-active, complete this step. Otherwise, continue with the Commit deployment step. Run the following commands to modify the control planes of
workload-cluster-2
.cat > manifests/smm-controlplane/overlays/workload-cluster-2/control-plane.yaml <<EOF apiVersion: smm.cisco.com/v1alpha1 kind: ControlPlane metadata: name: smm spec: clusterName: workload-cluster-2 role: active EOF
cat > manifests/smm-controlplane/overlays/workload-cluster-2/istio-cp-v115x.yaml <<EOF apiVersion: servicemesh.cisco.com/v1alpha1 kind: IstioControlPlane metadata: annotations: argocd.argoproj.io/sync-wave: "5" name: cp-v115x namespace: istio-system spec: meshID: mesh1 mode: ACTIVE networkName: network1 EOF
-
Create the
smm-controlplane
’s Argo CDApplicationSet
CR.cat > apps/smm-controlplane/app-set.yaml <<EOF apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: smm-cp-appset namespace: argocd spec: generators: - list: elements: - cluster: "${WORKLOAD_CLUSTER_1_CONTEXT}" path: "${WORKLOAD_CLUSTER_1_CONTEXT}" - cluster: "${WORKLOAD_CLUSTER_2_CONTEXT}" path: "${WORKLOAD_CLUSTER_2_CONTEXT}" template: metadata: name: 'smm-cp-{{cluster}}' namespace: argocd spec: project: default source: repoURL: https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git targetRevision: HEAD path: manifests/smm-controlplane/overlays/{{path}} destination: name: '{{cluster}}' syncPolicy: automated: prune: true selfHeal: true retry: limit: 5 backoff: duration: 5s maxDuration: 3m0s factor: 2 syncOptions: - Validate=false - PruneLast=true - CreateNamespace=true - Replace=true EOF
-
Commit and push the
calisti-gitops
repository.git add apps/smm-controlplane manifests
git commit -m "add smm-controlplane app"
git push
-
Apply the Application manifests.
kubectl apply -f "apps/smm-controlplane/app-set.yaml"
-
Verify that the applications have been added to Argo CD and are healthy.
argocd app list
-
To create trust between
workload-cluster-1
andworkload-cluster-2
, you must exchange the Secret CRs of the clusters. The cluster registry controller helps to form a group of Kubernetes clusters and synchronize any resources across those clusters arbitrarily.Create the following bash script and run it on
workload-cluster-1
.cat > export-secrets.sh <<EOF set -e kubectl --context workload-cluster-1 get cluster workload-cluster-1 -o yaml | kubectl --context workload-cluster-2 apply -f - kubectl --context workload-cluster-1 -n cluster-registry get secrets workload-cluster-1 -o yaml | kubectl --context workload-cluster-2 apply -f - kubectl --context workload-cluster-2 get cluster workload-cluster-2 -o yaml | kubectl --context workload-cluster-1 apply -f - kubectl --context workload-cluster-2 -n cluster-registry get secrets workload-cluster-2 -o yaml | kubectl --context workload-cluster-1 apply -f - echo "Exporting cluster and secrets CRs successfully." EOF
chmod +x export-secrets.sh ./export-secrets.sh
Expected output:
cluster.clusterregistry.k8s.cisco.com/workload-cluster-1 created secret/workload-cluster-1 created cluster.clusterregistry.k8s.cisco.com/workload-cluster-2 created secret/workload-cluster-2 created Exporting cluster and secrets CRs successfully.
-
Check that all pods are healthy and running in the
smm-system
namespace onworkload-cluster-1
andworkload-cluster-2
. Note that it takes some time while the ControlPlane operator reconciles the resources.For
workload-cluster-1
:kubectl get pods -n smm-system --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_1_CONTEXT}"
Expected output:
NAME READY STATUS RESTARTS AGE istio-operator-v115x-7d77fc549f-fxmtd 2/2 Running 0 8m15s mesh-manager-0 2/2 Running 0 19m prometheus-node-exporter-9xnmj 1/1 Running 0 17m prometheus-node-exporter-bf7g5 1/1 Running 0 17m prometheus-node-exporter-cl69q 1/1 Running 0 17m prometheus-smm-prometheus-0 4/4 Running 0 18m smm-7f4d5d4fff-4dlcp 2/2 Running 0 18m smm-7f4d5d4fff-59k7g 2/2 Running 0 18m smm-als-7cc4bfb998-wjsr6 2/2 Running 0 18m smm-authentication-569484f748-fj5zk 2/2 Running 0 18m smm-federation-gateway-6964fb956f-pb5pv 2/2 Running 0 18m smm-federation-gateway-operator-6664774695-9tmzj 2/2 Running 0 18m smm-grafana-59c54f67f4-9snc5 3/3 Running 0 18m smm-health-75bf4f49c5-z9tqg 2/2 Running 0 18m smm-health-api-7767d4f46-744wn 2/2 Running 0 18m smm-ingressgateway-6ffdfc6d79-jttjz 1/1 Running 0 11m smm-ingressgateway-external-8c9bb9445-kjt8h 1/1 Running 0 11m smm-kubestatemetrics-86c6f96789-lp576 2/2 Running 0 18m smm-leo-67cd7d49b5-gmcvf 2/2 Running 0 18m smm-prometheus-operator-ffbfb8b67-fwj6g 3/3 Running 0 18m smm-sre-alert-exporter-6654968479-fthk6 2/2 Running 0 18m smm-sre-api-86c9fb7cd7-mq7cm 2/2 Running 0 18m smm-sre-controller-6889685f9-hxxh5 2/2 Running 0 18m smm-tracing-5886d59dd-v8nb8 2/2 Running 0 18m smm-vm-integration-5b89c4f7c9-wz4bt 2/2 Running 0 18m smm-web-d5b49c7f6-jgz7b 3/3 Running 0 18m
For
workload-cluster-2
:kubectl get pods -n smm-system --kubeconfig "${WORKLOAD_CLUSTER_2_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_2_CONTEXT}"
Expected output:
NAME READY STATUS RESTARTS AGE istio-operator-v115x-7d77fc549f-s5wnz 2/2 Running 0 9m5s mesh-manager-0 2/2 Running 0 21m prometheus-node-exporter-fzdn4 1/1 Running 0 5m18s prometheus-node-exporter-rkbcl 1/1 Running 0 5m18s prometheus-node-exporter-x2mwp 1/1 Running 0 5m18s prometheus-smm-prometheus-0 3/3 Running 0 5m20s smm-ingressgateway-5db7859d45-6d6ns 1/1 Running 0 12m smm-kubestatemetrics-86c6f96789-j64q2 2/2 Running 0 19m smm-prometheus-operator-ffbfb8b67-zwqn2 3/3 Running 1 (11m ago) 19m
-
Check the applications on the Argo CD Web UI.
At this point, you have successfully installed smm-operator
and workload-cluster-1
and workload-cluster-2
. You can open the Service Mesh Manager dashboard to check them, or deploy an application.
Deploy an application
If you want to deploy want to deploy an application into the service mesh, complete the following steps. The examples use the Service Mesh Manager demo application.
The file structure for the demo application looks like this:
.
├── README.md
├── apps
│ ├── demo-app
│ │ └── app-set.yaml
│ └── ...
...manifests
└── demo-app
├── base
│ ├── demo-app-namespace.yaml
│ ├── demo-app.yaml
│ └── kustomization.yaml
└── overlays
├── workload-cluster-1
│ ├── demo-app.yaml
│ └── kustomization.yaml
└── workload-cluster-2
├── demo-app.yaml
└── kustomization.yaml
...
-
Create the application manifest files.
cat > manifests/demo-app/base/kustomization.yaml <<EOF apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization metadata: name: demo-app resources: - demo-app-namespace.yaml - demo-app.yaml EOF
cat > manifests/demo-app/base/demo-app-namespace.yaml <<EOF apiVersion: v1 kind: Namespace metadata: labels: app.kubernetes.io/instance: smm-demo app.kubernetes.io/name: smm-demo app.kubernetes.io/part-of: smm-demo app.kubernetes.io/version: 0.1.4 istio.io/rev: cp-v115x.istio-system name: smm-demo EOF
cat > manifests/demo-app/base/demo-app.yaml <<EOF apiVersion: smm.cisco.com/v1alpha1 kind: DemoApplication metadata: name: smm-demo namespace: smm-demo spec: autoscaling: enabled: true controlPlaneRef: name: smm EOF
cat > manifests/demo-app/overlays/workload-cluster-1/kustomization.yaml <<EOF apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization bases: - ../../base patchesStrategicMerge: - demo-app.yaml EOF
cat > manifests/demo-app/overlays/workload-cluster-1/demo-app.yaml <<EOF apiVersion: smm.cisco.com/v1alpha1 kind: DemoApplication metadata: name: smm-demo namespace: smm-demo spec: autoscaling: enabled: true controlPlaneRef: name: smm deployIstioResources: true deploySLOResources: true enabled: true enabledComponents: - frontpage - catalog - bookings - postgresql istio: revision: cp-v115x.istio-system load: enabled: true maxRPS: 30 minRPS: 10 swingPeriod: 1380000000000 replicas: 1 resources: limits: cpu: "2" memory: 192Mi requests: cpu: 40m memory: 64Mi EOF
cat > manifests/demo-app/overlays/workload-cluster-2/kustomization.yaml <<EOF apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization bases: - ../../base patchesStrategicMerge: - demo-app.yaml EOF
cat > manifests/demo-app/overlays/workload-cluster-2/demo-app.yaml <<EOF apiVersion: smm.cisco.com/v1alpha1 kind: DemoApplication metadata: name: smm-demo namespace: smm-demo spec: autoscaling: enabled: true controlPlaneRef: name: smm deployIstioResources: false deploySLOResources: false enabled: true enabledComponents: - movies - payments - notifications - analytics - database - mysql istio: revision: cp-v115x.istio-system replicas: 1 resources: limits: cpu: "2" memory: 192Mi requests: cpu: 40m memory: 64Mi EOF
-
Create the Demo application
ApplicationSet
.cat > apps/demo-app/app-set.yaml <<EOF apiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: demo-app-appset namespace: argocd spec: generators: - list: elements: - cluster: "${WORKLOAD_CLUSTER_1_CONTEXT}" path: "${WORKLOAD_CLUSTER_1_CONTEXT}" - cluster: "${WORKLOAD_CLUSTER_2_CONTEXT}" path: "${WORKLOAD_CLUSTER_2_CONTEXT}" template: metadata: name: 'demo-app-{{cluster}}' namespace: argocd spec: project: default source: repoURL: https://github.com/${GITHUB_ID}/${GITHUB_REPOSITORY_NAME}.git targetRevision: HEAD path: manifests/demo-app/overlays/{{path}} destination: name: '{{cluster}}' syncPolicy: automated: prune: true selfHeal: true retry: limit: 5 backoff: duration: 5s maxDuration: 3m0s factor: 2 syncOptions: - Validate=false - PruneLast=true - CreateNamespace=true - Replace=true EOF
-
Commit and push the
calisti-gitops
repository.git add apps/demo-app manifests git commit -m "add demo application" git push origin
-
Deploy the demo application on the clusters.
kubectl apply -f apps/demo-app/app-set.yaml
-
Wait until all the pods in the
smm-demo
namespace are up and running.kubectl get pods -n smm-demo --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_1_CONTEXT}"
Expected output:
NAME READY STATUS RESTARTS AGE bombardier-5f59948978-zx99c 2/2 Running 0 3m21s bookings-v1-68dd865855-fdcxk 2/2 Running 0 3m21s catalog-v1-6d564bbcb8-qmhbx 2/2 Running 0 3m21s frontpage-v1-b4686759b-fhfmv 2/2 Running 0 3m21s postgresql-7cf55cd596-grs46 2/2 Running 0 3m21s
kubectl get pods -n smm-demo --kubeconfig "${WORKLOAD_CLUSTER_2_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_2_CONTEXT}"
Expected output:
NAME READY STATUS RESTARTS AGE analytics-v1-799d668f84-p4nkk 2/2 Running 0 3m58s database-v1-6896cd4b59-9xxgg 2/2 Running 0 3m58s movies-v1-9594fff5f-8hv9l 2/2 Running 0 3m58s movies-v2-5559c5567c-2279n 2/2 Running 0 3m58s movies-v3-649b99d977-nkdxc 2/2 Running 0 3m58s mysql-669466cc8d-bs4s9 2/2 Running 0 3m58s notifications-v1-79bc79c89b-4bbss 2/2 Running 0 3m58s payments-v1-547884bfdf-dg2dm 2/2 Running 0 3m58s
-
Check the applications on the Argo CD web UI.
-
Open the Service Mesh Manager web interface, select MENU > TOPOLOGY, then select the
smm-demo
namespace.
Access the Service Mesh Manager dashboard
-
You can access the Service Mesh Manager dashboard via the
smm-ingressgateway-external
LoadBalancerexternal-ip-or-hostname
address. Run the following command to retrieve the IP address:kubectl get services -n smm-system smm-ingressgateway-external --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_1_CONTEXT}"
Expected output:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE smm-ingressgateway-external LoadBalancer 10.0.0.199 external-ip-or-hostname 80:32505/TCP 2m28s
-
Open the Service Mesh Manager dashboard using one of the following methods:
-
Open the
http://<external-ip-or-hostname>
URL in your browser. -
Run the following command to open the dashboard with your default browser:
# Exactly one of hostname or IP will be available and used for the remote URL. open http://$(kubectl get services -n smm-system smm-ingressgateway-external -o jsonpath='{.status.loadBalancer.ingress[0].hostname}{.status.loadBalancer.ingress[0].ip}' --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_1_CONTEXT}")
-
If you have installed the Service Mesh Manager CLI on your machine, run the following command to open the Service Mesh Manager Dashboard in the default browser.
smm dashboard --kubeconfig "${WORKLOAD_CLUSTER_1_KUBECONFIG}" --context "${WORKLOAD_CLUSTER_1_CONTEXT}"
Expected output:
✓ validate-kubeconfig ❯ checking cluster reachability... ✓ opening Service Mesh Manager at http://127.0.0.1:50500
-
-
Check the deployments on the dashboard, for example, on the MENU > Overview, MENU > MESH, and MENU > TOPOLOGY pages.