Circuit Breaking

Circuit Breaking is a pattern for creating resilient microservices applications. In microservices architecture, services are deployed across multiple nodes or clusters and have different response times or failure rate. Downstream clients need to be protected from excessive slowness of upstream services. Upstream services, in turn, must be protected from being overloaded by a backlog of requests.

A circuit breaker can have three states:

  • Closed: requests succeed or fail until the number of failures reach a predetermined threshold, with no interference from the breaker. When the threshold is reached, the circuit breaker opens.
  • Open: the circuit breaker trips the requests, which means that it returns an error without attempting to execute the call
  • Half open: the failing service is given time to recover from its broken behavior. If requests continue to fail in this state, then the circuit breaker is opened again and keeps tripping requests. Otherwise, if the requests succeed in the half open state, then the circuit breaker will close and the service will be allowed to handle requests again.

Circuit breaking in Istio Circuit breaking in Istio

Service Mesh Manager is using Istio’s - and therefore Envoy’s - circuit breaking feature under the hood.

Circuit breaking using the UI

Set circuit breaking

To configure a circuit breaker for a service, you don’t need to create or edit a Destination Rule resource manually, just complete the following steps.

  1. Select the service on the TOPOLOGY or the SERVICES page

  2. Select CIRCUIT BREAKER, then click CONFIGURE. If you have already configured a circuit breaker for the service and want to modify it, click Edit .

  3. Configure the parameters of the circuit breaker.

    Circuit Breaking set Circuit Breaking set

Monitor circuit breaking

With this configuration you’ve just set, when traffic begins to flow from two connections simultaneously, the circuit breaker starts to trip requests. In the Service Mesh Manager UI, you can see this the graph’s red edges on the MENU > TOPOLOGY page. Click on the service to see two live Grafana dashboards which specifically show the circuit breaker trips and help you learn more about the errors involved.

The first dashboard details the percentage of total requests that were tripped by the circuit breaker. When there are no circuit breaker errors, and your service works as expected, this graph shows 0%. Otherwise, it shows the percentage of the requests that were tripped by the circuit breaker.

The second dashboard provides a breakdown of the trips caused by the circuit breaker by source. If no circuit breaker trips occurred, there are no spikes in this graph. Otherwise, it shows which service caused the circuit breaker to trip, when, and how many times. Malicious clients can be tracked by checking this graph.

Circuit Breaking trip Circuit Breaking trip

These are live Grafana dashboards customized in order to display circuit breaker-related information.

Remove circuit breaking

To remove circuit breaking, navigate to the service, then select CIRCUIT BREAKER > Delete .