Fault Injection

Fault injection is a system testing method which involves the deliberate introduction of network faults and errors into a system. It can be used to identify design or configuration weaknesses, and to ensure that the system can handle faults and recover from error conditions.

With Service Mesh Manager, you can inject failures at the application layer to test the resiliency of the services. You can configure faults to be injected into requests that match specific conditions to simulate service failures and higher latency between services. There are two types of failures:

• Delay adds a time delay before forwarding the requests, emulating various failures such as network issues, an overloaded upstream service, and so on.

• Abort aborts the HTTP request attempts and returns error codes to a downstream service, giving the impression that the upstream service is faulty.

Service Mesh Manager uses Istio - and therefore Envoy’s - fault injection feature under the hood.

Fault injection using the UI

To configure fault injection on the UI, complete the following steps.

1. Select the service on the MENU > SERVICES or the MENU > TOPOLOGY page.

   

2. Select TRAFFIC MANAGEMENT > CREATE NEW.

3. Set the destination PORT NUMBER of the service where you want to add the rule.

4. By default, the new rule matches every incoming request. Click ADD CUSTOM MATCH to select only specific traffic for the rule based on scheme, method, URI, host, port, or authority.

   

5. Scroll down, select FAULT INJECTION, and set the percentage of traffic you want to add faults to, and the amount of delay or the HTTP error code to return.

   

6. Click Apply, then check the status of the service on the MENU > TOPOLOGY page. You should see the effects of the injected faults (for example, a delay in the response times).

   

Fault injection using the Istio resources

Set fault injection

To inject fault and test the resiliency of your service, complete the following steps.

1. Select the service on the MENU > ISTIO RESOURCES.

2. Select CREATE NEW > VIRTUAL SERVICE.

3. If you have already configured a fault injection configuration using the VirtualService for the service and want to modify it, click .

4. Configure the template and parameters of the fault injection:

   • Injecting an abort fault
   • Injecting a delay fault

Injecting an abort fault

HTTP abort fault injection prematurely aborts the request and returns the amount pre-specified error code describe in httpStatus. You can specify the percentage of the requests to be aborted.

• To set the HTTP aborts, select the Fault Injection Abort Template in the Template dropdown.

  

• Modify the YAML configuration to set the parameters for the abort fault injection.

  

• Once the YAML configuration is modified per the service needs, to validate the correctness of the YAML resource, click Validate .

• Verify that there are no errors. Then, to create the abort fault injection with the virtual service rule for the selected service, click Create.

Injecting a delay fault

Delay fault injection injects latency in the request forwarding path. You can specify the percentage of the requests to be delayed and the fixedDelay to indicate the number of delays.

• To introduce network latency or overload the upstream traffic, select the Fault Injection Delay Template in the Template dropdown.

  

• Modify the YAML configuration to set the parameters for the delay fault injection.

  

• Once the YAML configuration is modified per the service needs, to validate the correctness of the YAML resource, click Validate .

• Verify that there are no errors. Then, to create the delay fault injection with the virtual service rule for the selected service, click Create.

Remove fault injection

To remove fault injection, navigate to the service, then select VIRTUAL SERVICE and click .