Circuit Breaking

Circuit Breaking is a pattern for creating resilient microservices applications. In microservices architecture, services are deployed across multiple nodes or clusters and have different response times or failure rate. Downstream clients need to be protected from excessive slowness of upstream services. Upstream services, in turn, must be protected from being overloaded by a backlog of requests.

A circuit breaker can have three states:

  • Closed: requests succeed or fail until the number of failures reach a predetermined threshold, with no interference from the breaker. When the threshold is reached, the circuit breaker opens.
  • Open: the circuit breaker trips the requests, which means that it returns an error without attempting to execute the call
  • Half open: the failing service is given time to recover from its broken behavior. If requests continue to fail in this state, then the circuit breaker is opened again and keeps tripping requests. Otherwise, if the requests succeed in the half open state, then the circuit breaker will close and the service will be allowed to handle requests again.

Circuit breaking in Istio Circuit breaking in Istio

Service Mesh Manager is using Istio’s - and therefore Envoy’s - circuit breaking feature under the hood.

Circuit breaking using the dashboard

Set circuit breaking

To configure a circuit breaker for a service, complete the following steps.

  1. Select the service on the TOPOLOGY or the SERVICES page. Destination rule overview Destination rule overview

  2. Select the DESTINATION RULE tab, then click CREATE NEW. Destination rule Destination rule

  3. If you have already configured a circuit breaker using the destination rule for the service and want to modify it, click Edit .

  4. Configure the template and parameters of the circuit breaker:

Set outlier detection

Outlier detection controls the eviction of unhealthy services from the load-balancing pool. OutlierDetection controls the number of errors before a service is ejected from the connection pool, and using he template the minimum ejection duration and maximum ejection percentage can be set.

  • To set the outlier detection for a service, select the Outlier Detection template in the Template dropdown. Outlier detection Outlier detection
  • Modify the YAML configuration to set the parameters for the outlier detection. Outlier Outlier
  • Once the YAML configuration is modified per the service needs, to validate the correctness of the YAML resource, click ValidateValidate .
  • Verify that there are no errors. Then, to create the circuit breaking with the destination rule for the selected service, click Create.

Set connection pool

Connection pool settings sets the volume of connections that can be configured for a service. ConnectionPoolSettings controls the maximum number of requests, pending requests, retries or timeout.

  • To set the connection pool for a service, select the Connection Pool template in the Template dropdown. Connection pool Connection pool
  • Modify the YAML configuration to set the parameters for the connection pool. connection connection
  • Once the YAML configuration is modified per the service needs, to validate the correctness of the YAML resource, click ValidateValidate .
  • Verify that there are no errors. Then, to create the connection pool breaking with the destination rule for the selected service, click Create.

Monitor circuit breaking

With this configuration you’ve just set, when traffic begins to flow from two connections simultaneously, the circuit breaker starts to trip requests. In the Service Mesh Manager UI, you can see the tripped requests in the graph’s red edges on the MENU > TOPOLOGY page. Click on the service to see two live Grafana dashboards, which specifically show the circuit breaker trips and help you learn more about the errors involved.

The first dashboard details the percentage of total requests that were tripped by the circuit breaker. When there are no circuit breaker errors, and your service works as expected, this graph shows 0%. Otherwise, it shows the percentage of the requests that were tripped by the circuit breaker.

The second dashboard provides a breakdown of the trips caused by the circuit breaker by source. If no circuit breaker trips occurred, there are no spikes in this graph. Otherwise, it shows which service caused the circuit breaker to trip, when, and how many times. You can track for any malicious clients in this graph.

Circuit Breaking trip Circuit Breaking trip

These are live Grafana dashboards customized to display circuit breaker-related information.

Remove circuit breaking

To remove circuit breaking, navigate to the service, select Destination rule for circuit breaker, and click Delete .