Scale the Istio Control Plane
Istio has two major components:
- The proxy is responsible for handling the incoming traffic, and is injected into the Pods of every Workload.
- The Control Plane is responsible for coordinating the proxies, by sending them the latest configuration.
This section focuses on the performance characteristics of the Control Plane. For details on the proxies, see Scale the Istio proxy sidecar of a workload.
Find the Control Plane
By default, Service Mesh Manager installs the Control Plane into the
kubectl get pods -n istio-system
The output should be similar to:
NAME READY STATUS RESTARTS AGE istio-meshexpansion-gateway-cp-v115x-5d647494bb-qnn8b 1/1 Running 0 3h57m istiod-cp-v115x-749c46569-ld8fn 1/1 Running 0 3h57m
From this list the
istiod-cp-v115x-749c46569-ld8fn Pod is the Control Plane for Istio for version 1.15.x (
v115x). The other pod is used to provide cross-cluster connectivity.
In practice, these Pods are provisioned from an
IstioControlPlane resource in the same namespace:
kubectl get istiocontrolplanes -A
The output should be similar to:
NAMESPACE NAME MODE NETWORK STATUS MESH EXPANSION EXPANSION GW IPS ERROR AGE istio-system cp-v115x ACTIVE network1 Available true ["172.18.250.1"] 42h
When using the Service Mesh Manager dashboard for analyzing the resource usage, check these Workloads.
Understand scaling behavior
For the Control Plane, the CPU usage scales horizontally: if you add more pods to Kubernetes cluster,
istiod requires more CPU resources to send the configuration to all of those proxies. By adding more
istiod Pods, the number of workloads the service mesh can support increases.
On the other hand, each
istiod Pod needs to maintain the inventory of all the running Pods, Services, and Istio custom resources, thus as more of these added to the Kubernetes cluster, each Pod will need more memory. So from the memory consumption point of view, the service behaves as if it had vertical scalability.
This is quite a common situation where a service exhibits both vertical and horizontal scalability. In such cases, focus on the horizontal scalability properties, as long as the vertically scaling part (CPU or Memory) can be somewhat contained.
According to the upstream Istio documentation:
When namespace isolation is enabled, a single Istiod instance can support 1000 services, 2000 sidecars with 1 vCPU and 1.5 GB of memory. You can increase the number of Istiod instances to reduce the amount of time it takes for the configuration to reach all proxies.
When deciding on how to scale the workload, it is worth looking at the backing Kubernetes nodes. Usually the nodes are smaller ones: they have a few CPU cores (2-4) and 8-16 GB of memory. For the sake of this example, let’s say we have Kubernetes nodes with 2 vCPUs and 8GB RAM available.
In this case, giving 1 vCPU to Istio allocates 50% of the available CPU resources, while giving it 1.5 GB only reserves 18% of the available memory.
- If we want to scale vertically, based on memory, we can only double the resource utilization before we exhaust all of the available CPUs on the underlying node.
- With horizontal scaling with 1 vCPU allocated to each Istio control plane instance, we can add 8 Istios if needed before we exhaust the number of available resources on a single node, assuming that both memory and CPU utilization is scaling linearly.
Set resource limits via Control Plane
Service Mesh Manager provides two ways to change resource limits. The easiest one is to change the
ControlPlane resource by running the following commands:
cat > istio-cp-limits.yaml <<EOF spec: meshManager: istio: pilot: resources: requests: cpu: 500m memory: "1500M" limits: cpu: "1" memory: "2000M" EOF kubectl patch controlplane --type=merge --patch "$(cat istio-cp-limits.yaml )" smm
- If you are using Service Mesh Manager in Operator Mode, then the Istio deployment is updated automatically.
- If you are using the imperative mode, run the
smm operator reconcilecommand to apply the changes.
Note: By default, the Istio Control Plane has a HPA set up with the minimum of 1 and the maximum of 5 Pods.
Set resource limits via the Istio Resource
If the deployment needs more control over the Istio behavior, then the
IstioConrolPlane resource in the
istio-system namespace must be changed. Besides any settings (resources and images) defined in the
ControlPlane resource, any modifications will be preserved even if the
operator reconcile command is invoked, or if the Service Mesh Manager is deployed in Operator Mode.
For more details on this approach, see the Open Source Istio operator