Scenario
spring-cloud/spring-boot application is using zuul API gateway to route client requests. Here is zuul config snippet
zuul.ribbonIsolationStrategy=THREAD zuul.threadPool.useSeparateThreadPools=true ribbon.eureka.enabled = false # sevices ...
Invoking service APIs through zuul gateway resulted in HTTP 500 response that looked something like this:
{
"timestamp": "2019-12-18T21:37:10.168+0000",
"status": 500,
"error": "Internal Server Error",
"message": "REJECTED_THREAD_EXECUTION"
}Interesting part of this response is REJECTED_THREAD_EXECUTION. None of the services were reporting this message. After looking deeper, the Zuul log showed this error
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3b17817e rejected from java.util.concurrent.ThreadPoolExecutor@67f89c35[Running, pool size = 10, active threads = 10, queued tasks = 0, completed tasks = 2535368]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_212]Solution
After realizing that exception is in the zuul log it was a matter of finding the right properties to configure services thread pools. By default, zuul was using 10 threads with the max thread count set to 10. To change this default across all services set these properties:
# defualt threadpool config hystrix.threadpool.default.coreSize = 150 hystrix.threadpool.default.maximumSize = 150
It is also possible to configure thread pools for each service that zuul is routing to. For example:
zuul.routes.first.url=firstServiceId zuul.routes.second.url=secondServiceId hystrix.threadpool.firstServiceId.coreSize = 150 hystrix.threadpool.firstServiceId.maximumSize = 150 hystrix.threadpool.secondServiceId.coreSize = 50 hystrix.threadpool.secondServiceId.maximumSize = 150