Chaos Engineering for Spring Boot Applications

In this article, we will discuss an out of box library “Spring Boot Chaos Monkey” to provide a Chaos Monkey for Spring Boot applications and this lib will try to check the resiliency of your running spring boot applications. This library is inspired by PRINCIPLES OF CHAOS ENGINEERING, with a focus on Spring Boot to test applications better during operation.

According to the PRINCIPLES OF CHAOS ENGINEERING–

Chaos Engineering is the discipline of experimenting on a system in order to build confidence in the system’s capability to withstand turbulent conditions in production.

Everything from getting started to advanced usage is explained in the Documentation for Chaos Monkey for Spring Boot.

Need of Chaos Engineering for Spring Boot applications

Netflix has started Chaos Engineering to check their APIs in recent years and also it contributed significantly to the growing importance of Chaos Engineering in distributed systems.

The below lines by Fire Chief Mike Burtch but quite interesting I found.

You don’t choose the moment, the moment chooses you!

You only choose how prepared you are when it does.

As application developers, we focus on developing a product that must be stable, secure and bug-free software. To achieve this, we write a lot of unit tests and integration tests so that we can capture unexpected behaviour and ensure that the patterns we test do not lead to errors. But what I observe in my career of software development after writing many units and integration tests, we might maximum achieve a code coverage from 70% to 80%, sometimes more but not 100%, we can’t be happy with this unpleasant feeling, how our application behaves in production?

Many more questions may arise to mind such as

• How our code will work on the production?
• What will happen if one service went down?
• How does the application behave with network latency?
• Will our fallback scenarios work?
• Will Service Discovery work?
• Is our Client-Side-Load-Balancing also working?

Today’s applications are adopting microservice architectures and distributed applications creation. These architectures contain many components that can’t all be fully covered with unit and integration tests.

If you want more familiar with the principles of chaos engineering, you can check out this blog post for chaos engineering.

Let’s integrate Chaos Monkey for Spring Boot applications.

Adding Chaos Monkey in Spring Boot application

In a spring boot application, just add Spring Boot Chaos Monkey on your classpath of the application and activate with the profile name, it will automatically hook into your spring boot application.

Add Chaos Monkey for Spring Boot as a dependency for your project using MAVEN.

<dependency>

    <groupId>de.codecentric</groupId>

    <artifactId>chaos-monkey-spring-boot</artifactId>

    <version>2.5.4</version>

</dependency>

Add Chaos Monkey for Spring Boot as a dependency for your project using GRADLE.

implementation 'de.codecentric:chaos-monkey-spring-boot:2.5.4'

Let start your Spring Boot Application with the chaos-monkey spring profile enabled. You can also pass some other properties to assault services with latency. Let’s see the following properties as I have defined run time as java arguments.

java -jar my-springboot-app.jar --spring.profiles.active=chaos-monkey --chaos.monkey.enabled=true --chaos.monkey.watcher.service=true --chaos.monkey.assaults.latencyActive=true

You can also specify the above properties in your application.properties or application.yml file. Let’s see the following application.properties file with the minimum required configuration for the Chaos Monkey for Spring Boot application.

File application.properties

spring.profiles.active=chaos-monkey

chaos.monkey.enabled=true

 

chaos.monkey.watcher.service=true

chaos.monkey.assaults.latencyActive=true

File application.yml

spring:

  profiles:

    active: chaos-monkey

chaos:

  monkey:

    enabled: true

    watcher:

      service: true

    assaults:

      latencyActive: true

Now you can activate watchers, which look for classes to assault. There are also runtime assaults, which attack your whole application. Let’s see the following diagram how does it work?

In the above diagram, we can see that Chaos Monkey provides watchers and assaults. In the Spring boot applications, we have controllers, repositories, and services and Spring Boot Chaos Monkey has a corresponding watcher for each resource. For example, @Controller watched by controller watcher, @Respository watched by repository watchers etc. And each watcher can generate multiple assaults for each resource.

Watcher

A watcher is a Chaos Monkey for Spring Boot component, that will scan your app for beans based on one of the conditions described in Watcher Types.

These watchers find your beans based on the following annotations:

• @Controller
• @RestController
• @Service
• @Repository
• @Component

We can also define custom watchers as well.

Assault

Assaults are the heart of Monkey’s Chaos and he makes use of them based on your configuration. There are the following types of assault provided by Chaos Monkey.

Request Assaults

These assaults attack some point of your application and are triggered by a /actuator/chaosmonkey/watchers endpoint.

• Latency Assault
• Exception Assault

Runtime Assaults

• AppKiller Assault
• Memory Assault
• CPU Assault

The above minimum configuration is enough to test your spring boot application resiliency but if you pass these properties by using the configuration files either using application.properties or application.yml file, you have to start your application on production. Starting the production application to test your resiliency is not recommended way you have to take a lot of approval from upper management, so you can avoid this restart if you use the Spring Boot Actuator and HTTP/JMX.

Spring Boot Actuator Endpoints

Chaos Monkey for Spring Boot offers you some built-in endpoints exposed via JMX or HTTP. This allows you to change the configuration at runtime.

File application.properties

management.endpoint.chaosmonkey.enabled=true

management.endpoint.chaosmonkeyjmx.enabled=true

 

# include all endpoints

management.endpoints.web.exposure.include=*

 

# include specific endpoints

management.endpoints.web.exposure.include=health,info,chaosmonkey

File application.yml

management:

  endpoint:

    chaosmonkey:

      enabled: true

    chaosmonkeyjmx:

      enabled: true

 

  endpoints:

    web:

      exposure:

        # include all endpoints

        include: "*"

        # include specific endpoints

        include:

          - health

          - info

          - chaosmonkey

Let’s start the Spring Boot application test all endpoints as below:

GET Chaos Monkey Configuration

/actuator/chaosmonkey

Response:

{
    "chaosMonkeyProperties": {
        "enabled": true,
        "togglePrefix": "chaos.monkey"
    },
    "assaultProperties": {
        "level": 1,
        "deterministic": false,
        "latencyRangeStart": 1000,
        "latencyRangeEnd": 3000,
        "latencyActive": true,
        "exceptionsActive": false,
        "exception": {
            "type": null,
            "arguments": null
        },
        "killApplicationActive": false,
        "memoryActive": false,
        "memoryMillisecondsHoldFilledMemory": 90000,
        "memoryMillisecondsWaitNextIncrease": 1000,
        "memoryFillIncrementFraction": 0.15,
        "memoryFillTargetFraction": 0.25,
        "cpuActive": false,
        "cpuMillisecondsHoldLoad": 90000,
        "cpuLoadTargetFraction": 0.9,
        "runtimeAssaultCronExpression": "OFF"
    },
    "watcherProperties": {
        "controller": false,
        "restController": false,
        "service": true,
        "repository": false,
        "component": false,
        "restTemplate": false,
        "webClient": false,
        "actuatorHealth": false,
        "beans": []
    }
}

GET Chaos Monkey Status

/actuator/chaosmonkey/status

Response:

Ready to be evil!

POST Chaos Monkey disable

/actuator/chaosmonkey/disable

Response:

{
    "status": "Chaos Monkey is disabled",
    "disabledAt": "2022-01-04T16:41:40.426+05:30"
}

POST Chaos Monkey enable

/actuator/chaosmonkey/enable

Response:

{
    "status": "Chaos Monkey is enabled",
    "enabledAt": "2022-01-04T16:44:35.909+05:30"
}

GET Watchers

/actuator/chaosmonkey/watchers

Response:

{
    "controller": false,
    "restController": false,
    "service": true,
    "repository": false,
    "component": false,
    "restTemplate": false,
    "webClient": false,
    "actuatorHealth": false,
    "beans": []
}

POST Watchers
Request to enable/disable Watchers

POST /actuator/chaosmonkey/watchers

Request Body:

{
  "controller": true,
  "restController": true,
  "service": true,
  "repository": true,
  "component": false,
  "restTemplate": false,
  "webClient": false,
  "actuatorHealth": false
}

Response:

Watcher config has changed

Check again watchers

GET Watchers

/actuator/chaosmonkey/watchers

Response:

{
    "controller": true,
    "restController": true,
    "service": true,
    "repository": true,
    "component": false,
    "restTemplate": false,
    "webClient": false,
    "actuatorHealth": false,
    "beans": []
}

GET Assaults

GET /actuator/chaosmonkey/assaults

Response:

{
    "level": 1,
    "deterministic": false,
    "latencyRangeStart": 1000,
    "latencyRangeEnd": 3000,
    "latencyActive": true,
    "exceptionsActive": false,
    "exception": {
        "type": null,
        "arguments": null
    },
    "killApplicationActive": false,
    "memoryActive": false,
    "memoryMillisecondsHoldFilledMemory": 90000,
    "memoryMillisecondsWaitNextIncrease": 1000,
    "memoryFillIncrementFraction": 0.15,
    "memoryFillTargetFraction": 0.25,
    "cpuActive": false,
    "cpuMillisecondsHoldLoad": 90000,
    "cpuLoadTargetFraction": 0.9,
    "runtimeAssaultCronExpression": "OFF"
}

POST Assaults
Request to enable Latency & Exception Assault

POST /actuator/chaosmonkey/assaults

Request Body:

{
  "level": 5,
  "latencyRangeStart": 2000,
  "latencyRangeEnd": 5000,
  "latencyActive": true,
  "exceptionsActive": true,
  "killApplicationActive": false
}

Response:

Assault config has changed

GET Assaults again

GET /actuator/chaosmonkey/assaults

Response:

{
    "level": 5,
    "deterministic": false,
    "latencyRangeStart": 2000,
    "latencyRangeEnd": 5000,
    "latencyActive": true,
    "exceptionsActive": true,
    "exception": {
        "type": null,
        "arguments": null
    },
    "killApplicationActive": false,
    "memoryActive": false,
    "memoryMillisecondsHoldFilledMemory": 90000,
    "memoryMillisecondsWaitNextIncrease": 1000,
    "memoryFillIncrementFraction": 0.15,
    "memoryFillTargetFraction": 0.25,
    "cpuActive": false,
    "cpuMillisecondsHoldLoad": 90000,
    "cpuLoadTargetFraction": 0.9,
    "runtimeAssaultCronExpression": "OFF"
}

Define specific method attacks

POST /actuator/chaosmonkey/assaults

Request Body:

{
  "level": 5,
  "latencyRangeStart": 2000,
  "latencyRangeEnd": 5000,
  "latencyActive": true,
  "exceptionsActive": true,
  "killApplicationActive": false,
  "watchedCustomServices": [
    "com.doj.myapp.controller.HelloController.sayHello",
    "com.doj.myapp.controller.HelloController.sayWelcome"
  ]
}

Response:

Assault config has changed

GET Assaults again

GET /actuator/chaosmonkey/assaults

Response:

{
    "level": 5,
    "deterministic": false,
    "latencyRangeStart": 2000,
    "latencyRangeEnd": 5000,
    "latencyActive": true,
    "exceptionsActive": true,
    "exception": {
        "type": null,
        "arguments": null
    },
    "killApplicationActive": false,
    "memoryActive": false,
    "memoryMillisecondsHoldFilledMemory": 90000,
    "memoryMillisecondsWaitNextIncrease": 1000,
    "memoryFillIncrementFraction": 0.15,
    "memoryFillTargetFraction": 0.25,
    "cpuActive": false,
    "cpuMillisecondsHoldLoad": 90000,
    "cpuLoadTargetFraction": 0.9,
    "runtimeAssaultCronExpression": "OFF",
    "watchedCustomServices": [
        "com.doj.myapp.controller.HelloController.sayHello",
        "com.doj.myapp.controller.HelloController.sayWelcome"
    ]
}