Category: Bug of the day

Scenario

spring-cloud/spring-boot application is using zuul API gateway to route client requests. Here is zuul config snippet

zuul.ribbonIsolationStrategy=THREAD
zuul.threadPool.useSeparateThreadPools=true

ribbon.eureka.enabled = false

# sevices
...

Invoking service APIs through zuul gateway resulted in HTTP 500 response that looked something like this:

{
  "timestamp": "2019-12-18T21:37:10.168+0000",
  "status": 500,
  "error": "Internal Server Error",
  "message": "REJECTED_THREAD_EXECUTION"
}

Interesting part of this response is REJECTED_THREAD_EXECUTION. None of the services were reporting this message. After looking deeper, the Zuul log showed this error

Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3b17817e rejected from java.util.concurrent.ThreadPoolExecutor@67f89c35[Running, pool size = 10, active threads = 10, queued tasks = 0, completed tasks = 2535368]
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_212]
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_212]

Solution

After realizing that exception is in the zuul log it was a matter of finding the right properties to configure services thread pools. By default, zuul was using 10 threads with the max thread count set to 10. To change this default across all services set these properties:

# defualt threadpool config
hystrix.threadpool.default.coreSize = 150
hystrix.threadpool.default.maximumSize = 150

It is also possible to configure thread pools for each service that zuul is routing to. For example:

zuul.routes.first.url=firstServiceId
zuul.routes.second.url=secondServiceId

hystrix.threadpool.firstServiceId.coreSize = 150
hystrix.threadpool.firstServiceId.maximumSize = 150

hystrix.threadpool.secondServiceId.coreSize = 50
hystrix.threadpool.secondServiceId.maximumSize = 150

Scenario

Suppose you are working on multiple Spring Boot/Spring Cloud projects that are using the spring cloud config server. You setup property encryption for the spring-cloud config. The encryption key is sored in your local profile as an environment variable ENCRYPT_KEY. Then, you switch to work on another spring-boot/spring-cloud project that does not use spring config server and have spring-cloud-context as a dependency. When you run this project you get the following exception stack trace:

Caused by: java.lang.NullPointerException: null
	at org.springframework.cloud.context.encrypt.EncryptorFactory.create(EncryptorFactory.java:54)
	at org.springframework.cloud.bootstrap.encrypt.EncryptionBootstrapConfiguration$RsaEncryptionConfiguration.textEncryptor(EncryptionBootstrapConfiguration.java:83)
	at org.springframework.cloud.bootstrap.encrypt.EncryptionBootstrapConfiguration$RsaEncryptionConfiguration$$EnhancerBySpringCGLIB$$c1a972bf.CGLIB$textEncryptor$0(<generated>)
	at org.springframework.cloud.bootstrap.encrypt.EncryptionBootstrapConfiguration$RsaEncryptionConfiguration$$EnhancerBySpringCGLIB$$c1a972bf$$FastClassBySpringCGLIB$$4d33eb7b.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:363)
	at org.springframework.cloud.bootstrap.encrypt.EncryptionBootstrapConfiguration$RsaEncryptionConfiguration$$EnhancerBySpringCGLIB$$c1a972bf.textEncryptor(<generated>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
	... 31 common frames omitted

Diagnostic

This is actually a spring bug! Another way to provide the encryption key to the config server is to use -Dencrypt.key=123 system property. If you do it this way everything is working fine. The problem only arises when the environment variable ENCRYPT_KEY is used. The root cause is in:

org.springframework.cloud.bootstrap.encrypt.EncryptionBootstrapConfiguration.KeyCondition

This condition bean determines whether spring will initialize TextEncryptor class or not. When spring detects ENCRYPT_KEY environment variable the TextEncryptor class gets created. It uses KeyProperties to initialize encryptor. If -Dencrypt.key=123 is provided then KeyProperties is initialized with the key provided as a system property. If ENCRYPT_KEY environment variable is found KeyProperties are not initialized and the key member is null.

Solution

Remove ENCRYPT_KEY property from the environment prior to running spring boot application. The problem is fixed.

Scenario

Let’s say you built spring-boot executable jar and uploaded it into nexus repository. To deploy your spring-boot application you create a Dockerfile that downloads your jar using nexus rest search and download API. For example:

FROM openjdk:8-jre-alpine

MAINTAINER itzap <mailer@itzap.com>

WORKDIR /

RUN wget -O app-service.jar  \
	'https://nexus.companyUrl.com/nexus/service/rest/v1/search/assets/download?group=com.mycompany.service&name=app-service&repository=company-snapshots&sort=version&direction=des'

EXPOSE 8080

ENTRYPOINT [ "java", "-jar","app-service.jar"]

When you run docker container you see the following error message:

no main manifest attribute, in app-service.jar

Diagnostic

When you build and run jar locally everything is working fine. app-service.jar inside docker container appears to be corrupted. To diagnose the issue run the wget command from the terminal:

wget -O app-service.jar  \
	'https://nexus.companyUrl.com/nexus/service/rest/v1/search/assets/download?group=com.mycompany.service&name=app-service&repository=company-snapshots&sort=version&direction=des'

Notice the following results on the command line:

$ wget -O app-service.jar 'https://nexus.companyUrl.com/nexus/service/rest/v1/search/assets/download?group=com.mycompany.service&name=app-service&repository=company-snapshots&sort=version&direction=desc'

--2019-10-21 21:48:27--  https://nexus.companyUrl.com/nexus/service/rest/v1/search/assets/download?group=com.mycompany.service&name=app-service&repository=company-snapshots&sort=version&direction=desc

Распознаётся nexus.compnyUrl.com (nexus.companyUrl.com)… 00.00.00.00

Подключение к nexus.compnyUrl.com (nexus.compnyUrl.com)|00.00.00.00|:443... соединение установлено.

HTTP-запрос отправлен. Ожидание ответа… 302 Found

Адрес: https://nexus.compnyUrl.com/nexus/repository/company-snapshots/com/mycompany/service/app-service/0.0.1-SNAPSHOT/app-service-0.0.1-20191021.220127-1-javadoc.jar [переход]

--2019-10-21 21:48:28--  https://nexus.compnyUrl.com/nexus/repository/app-snapshots/com/mycompany/service/app-service/0.0.1-SNAPSHOT/app-service-0.0.1-20191021.220127-1-javadoc.jar

Повторное использование соединения с nexus.compnyUrl.com:443.

HTTP-запрос отправлен. Ожидание ответа… 200 OK

Длина: 247463 (242K) [application/java-archive]

Сохранение в: «app-service.jar»

Do not get all these messages in Russian fool you. This is not a Russian hacker attack. Pay attention to what is actually being downloaded. app-service-0.0.1-20191021.220127-1-javadoc.jar javadoc!!!

Solution

Solution to this issue can be found in sonatype Search API. Looking through documentation I came across this query parameter maven.classifier This issue happens when your repository contains javadoc jar alnog side your artifact. &sort=version&direction=desc parameters are also critical to insure you are downloading the latest version, but not enough to uniquely identify artifact. Final URL that works looks like this:

'https://nexus.companyUrl.com/nexus/service/rest/v1/search/assets/download?group=com.mycompany.service&name=app-service&repository=company-snapshots&sort=version&direction=desc&maven.classifier'

Scenario

In a spring-boot application that uses spring-boot-dependencies instead of spring-boot-starter-parent and a third party library that depends on elasticsearch version that is not comparable with version 6.x, elasticsearch dependency version get mixed up creating class loading issue.

Steps to reproduce

Create a spring-boot project and include spring-boot-dependencies in the dependency management section of the maven pom.xml

<properties>
  <spring.boot.version>2.1.6.RELEASE</spring.boot.version>
  <spring.cloud.version>Greenwich.RELEASE</spring.cloud.version>
</properties>

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-dependencies</artifactId>
      <version>${spring.boot.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>

    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-dependencies</artifactId>
      <version>${spring.cloud.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

Later, in the dependencies section include a third party library that depends on elastic search version 5.6.8

<dependnecies>
  <dependency>
    <groupId>com.netflix.conductor</groupId>
    <artifactId>conductor-es5-persistence</artifactId>
    <version>2.8.1</version>
  </dependency>
</dependnecies>

Note that conductor-es5-persistence depends on elasticsearch version 5.6.8

The try to run mvn dependency:tree on your project and observe the following:

[INFO] +- com.netflix.conductor:conductor-es5-persistence:jar:2.8.1:compile
[INFO] |  +- org.elasticsearch:elasticsearch:jar:6.4.3:compile
[INFO] |  |  +- org.elasticsearch:elasticsearch-core:jar:6.4.3:compile

Where 6.4.3 is coming from?

Diagnostic

Turns out mvn dependency:tree shows the issue, but it does not say where org.elasticsearch:elasticsearch:jar:6.4.3:compile is comming from. The best command for the job is:

mvn -Dverbose=true help:effective-pom

This command outputs effective POM that can be redirected to a file mvn -Dverbose=true help:effective-pom > ~/pom.xml and analysed. The effective pom will look something like this

...
<dependency>
  <groupId>org.elasticsearch</groupId>  <!-- org.springframework.boot:spring-boot-dependencies:2.1.6.RELEASE, line 1965 -->
  <artifactId>elasticsearch</artifactId>  <!-- org.springframework.boot:spring-boot-dependencies:2.1.6.RELEASE, line 1966 -->
  <version>6.4.3</version>  <!-- org.springframework.boot:spring-boot-dependencies:2.1.6.RELEASE, line 1967 -->
</dependency>
...

Magic! now we can see exactly where the problem is:

<!-- org.springframework.boot:spring-boot-dependencies:2.1.6.RELEASE, line 1967 -->

Solution

Lets look at this import dependency:

<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-dependencies</artifactId>
    <version>${spring.boot.version}</version>
    <type>pom</type>
    <scope>import</scope>
  </dependency>

spring-boot-dependencies is imported, so there is no easy way to exclude elasticsearch dependency that it pulls into our project. A simple maven <exclude> will not work here. The only way to solve the issue is to bring elasticsearch dependencies into your project dependency management block like this:

<properties>
  <spring.boot.version>2.1.6.RELEASE</spring.boot.version>
  <spring.cloud.version>Greenwich.RELEASE</spring.cloud.version>
  
  <elasticsearch.version>5.6.8</elasticsearch.version>
</properties>

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-dependencies</artifactId>
      <version>${spring.boot.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>

    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-dependencies</artifactId>
      <version>${spring.cloud.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
    
    <!-- Elasticsearch -->
    <dependency>
      <groupId>org.elasticsearch</groupId>
      <artifactId>elasticsearch</artifactId>
      <version>${elasticsearch.version}</version>
    </dependency>

    <dependency>
      <groupId>org.elasticsearch.client</groupId>
      <artifactId>transport</artifactId>
      <version>${elasticsearch.version}</version>
    </dependency>

    <dependency>
      <groupId>org.elasticsearch.distribution.integ-test-zip</groupId>
      <artifactId>elasticsearch</artifactId>
      <version>${elasticsearch.version}</version>
      <type>zip</type>
    </dependency>

    <dependency>
      <groupId>org.elasticsearch.plugin</groupId>
      <artifactId>transport-netty4-client</artifactId>
      <version>${elasticsearch.version}</version>
    </dependency>

  </dependencies>
</dependencyManagement>

Now run mvn -Dverbose=true help:effective-pom and confirm that the problem is fixed.