Scenario
spring-cloud/spring-boot application is using zuul API gateway to route client requests. Here is zuul config snippet
zuul.ribbonIsolationStrategy=THREAD zuul.threadPool.useSeparateThreadPools=true ribbon.eureka.enabled = false # sevices ...
Invoking service APIs through zuul gateway resulted in HTTP 500 response that looked something like this:
{
"timestamp": "2019-12-18T21:37:10.168+0000",
"status": 500,
"error": "Internal Server Error",
"message": "REJECTED_THREAD_EXECUTION"
}Interesting part of this response is REJECTED_THREAD_EXECUTION. None of the services were reporting this message. After looking deeper, the Zuul log showed this error
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3b17817e rejected from java.util.concurrent.ThreadPoolExecutor@67f89c35[Running, pool size = 10, active threads = 10, queued tasks = 0, completed tasks = 2535368]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_212]Solution
After realizing that exception is in the zuul log it was a matter of finding the right properties to configure services thread pools. By default, zuul was using 10 threads with the max thread count set to 10. To change this default across all services set these properties:
# defualt threadpool config hystrix.threadpool.default.coreSize = 150 hystrix.threadpool.default.maximumSize = 150
It is also possible to configure thread pools for each service that zuul is routing to. For example:
zuul.routes.first.url=firstServiceId zuul.routes.second.url=secondServiceId hystrix.threadpool.firstServiceId.coreSize = 150 hystrix.threadpool.firstServiceId.maximumSize = 150 hystrix.threadpool.secondServiceId.coreSize = 50 hystrix.threadpool.secondServiceId.maximumSize = 150
Awesome post! Keep up the great work! 🙂
Great content! Super high-quality! Keep it up! 🙂
Good web site! I truly love how it is simple on my eyes and the data are well written. I am wondering how I might be notified whenever a new post has been made. I have subscribed to your RSS feed which must do the trick! Have a nice day!
I am continuously searching online for posts that can help me. Thanks!
I wanted to write you a little observation to help give thanks over again with your magnificent advice you’ve shown above. It’s really unbelievably generous with you to provide easily exactly what most people could have supplied for an ebook to end up making some bucks for their own end, particularly given that you could have done it in case you wanted. Those ideas also acted to provide a great way to be certain that other individuals have a similar eagerness like my own to know very much more in regard to this condition. I’m certain there are some more pleasant situations in the future for individuals that looked at your blog post.
Good post. I am facing a few of these issues as well..