Category: spring-cloud

Scenario

spring-cloud/spring-boot application is using zuul API gateway to route client requests. Here is zuul config snippet

zuul.ribbonIsolationStrategy=THREAD
zuul.threadPool.useSeparateThreadPools=true

ribbon.eureka.enabled = false

# sevices
...

Invoking service APIs through zuul gateway resulted in HTTP 500 response that looked something like this:

{
  "timestamp": "2019-12-18T21:37:10.168+0000",
  "status": 500,
  "error": "Internal Server Error",
  "message": "REJECTED_THREAD_EXECUTION"
}

Interesting part of this response is REJECTED_THREAD_EXECUTION. None of the services were reporting this message. After looking deeper, the Zuul log showed this error

Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3b17817e rejected from java.util.concurrent.ThreadPoolExecutor@67f89c35[Running, pool size = 10, active threads = 10, queued tasks = 0, completed tasks = 2535368]
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_212]
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_212]

Solution

After realizing that exception is in the zuul log it was a matter of finding the right properties to configure services thread pools. By default, zuul was using 10 threads with the max thread count set to 10. To change this default across all services set these properties:

# defualt threadpool config
hystrix.threadpool.default.coreSize = 150
hystrix.threadpool.default.maximumSize = 150

It is also possible to configure thread pools for each service that zuul is routing to. For example:

zuul.routes.first.url=firstServiceId
zuul.routes.second.url=secondServiceId

hystrix.threadpool.firstServiceId.coreSize = 150
hystrix.threadpool.firstServiceId.maximumSize = 150

hystrix.threadpool.secondServiceId.coreSize = 50
hystrix.threadpool.secondServiceId.maximumSize = 150

Scenario

In a spring-boot application that uses spring-boot-dependencies instead of spring-boot-starter-parent and a third party library that depends on elasticsearch version that is not comparable with version 6.x, elasticsearch dependency version get mixed up creating class loading issue.

Steps to reproduce

Create a spring-boot project and include spring-boot-dependencies in the dependency management section of the maven pom.xml

<properties>
  <spring.boot.version>2.1.6.RELEASE</spring.boot.version>
  <spring.cloud.version>Greenwich.RELEASE</spring.cloud.version>
</properties>

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-dependencies</artifactId>
      <version>${spring.boot.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>

    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-dependencies</artifactId>
      <version>${spring.cloud.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

Later, in the dependencies section include a third party library that depends on elastic search version 5.6.8

<dependnecies>
  <dependency>
    <groupId>com.netflix.conductor</groupId>
    <artifactId>conductor-es5-persistence</artifactId>
    <version>2.8.1</version>
  </dependency>
</dependnecies>

Note that conductor-es5-persistence depends on elasticsearch version 5.6.8

The try to run mvn dependency:tree on your project and observe the following:

[INFO] +- com.netflix.conductor:conductor-es5-persistence:jar:2.8.1:compile
[INFO] |  +- org.elasticsearch:elasticsearch:jar:6.4.3:compile
[INFO] |  |  +- org.elasticsearch:elasticsearch-core:jar:6.4.3:compile

Where 6.4.3 is coming from?

Diagnostic

Turns out mvn dependency:tree shows the issue, but it does not say where org.elasticsearch:elasticsearch:jar:6.4.3:compile is comming from. The best command for the job is:

mvn -Dverbose=true help:effective-pom

This command outputs effective POM that can be redirected to a file mvn -Dverbose=true help:effective-pom > ~/pom.xml and analysed. The effective pom will look something like this

...
<dependency>
  <groupId>org.elasticsearch</groupId>  <!-- org.springframework.boot:spring-boot-dependencies:2.1.6.RELEASE, line 1965 -->
  <artifactId>elasticsearch</artifactId>  <!-- org.springframework.boot:spring-boot-dependencies:2.1.6.RELEASE, line 1966 -->
  <version>6.4.3</version>  <!-- org.springframework.boot:spring-boot-dependencies:2.1.6.RELEASE, line 1967 -->
</dependency>
...

Magic! now we can see exactly where the problem is:

<!-- org.springframework.boot:spring-boot-dependencies:2.1.6.RELEASE, line 1967 -->

Solution

Lets look at this import dependency:

<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-dependencies</artifactId>
    <version>${spring.boot.version}</version>
    <type>pom</type>
    <scope>import</scope>
  </dependency>

spring-boot-dependencies is imported, so there is no easy way to exclude elasticsearch dependency that it pulls into our project. A simple maven <exclude> will not work here. The only way to solve the issue is to bring elasticsearch dependencies into your project dependency management block like this:

<properties>
  <spring.boot.version>2.1.6.RELEASE</spring.boot.version>
  <spring.cloud.version>Greenwich.RELEASE</spring.cloud.version>
  
  <elasticsearch.version>5.6.8</elasticsearch.version>
</properties>

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-dependencies</artifactId>
      <version>${spring.boot.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>

    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-dependencies</artifactId>
      <version>${spring.cloud.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
    
    <!-- Elasticsearch -->
    <dependency>
      <groupId>org.elasticsearch</groupId>
      <artifactId>elasticsearch</artifactId>
      <version>${elasticsearch.version}</version>
    </dependency>

    <dependency>
      <groupId>org.elasticsearch.client</groupId>
      <artifactId>transport</artifactId>
      <version>${elasticsearch.version}</version>
    </dependency>

    <dependency>
      <groupId>org.elasticsearch.distribution.integ-test-zip</groupId>
      <artifactId>elasticsearch</artifactId>
      <version>${elasticsearch.version}</version>
      <type>zip</type>
    </dependency>

    <dependency>
      <groupId>org.elasticsearch.plugin</groupId>
      <artifactId>transport-netty4-client</artifactId>
      <version>${elasticsearch.version}</version>
    </dependency>

  </dependencies>
</dependencyManagement>

Now run mvn -Dverbose=true help:effective-pom and confirm that the problem is fixed.