Details
-
Bug
-
Resolution: Done
-
Major
-
2.10.0-fuse-71-047
-
None
-
None
Description
Attached is a JUnit test that reproduces a deadlock in Camel.
The route definition uses a dead letter channel, two onException policies and a recipientList.
Trying to suspend the Camel route while it is processing messages concurrently results in a dead lock.
The thread that suspends the route waits for the suspend() call to return.
Name: pool-4-thread-1 State: WAITING on java.util.concurrent.CountDownLatch$Sync@439ddf4e Total blocked: 0 Total waited: 3 Stack trace: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281) java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207) org.apache.camel.component.seda.SedaConsumer.prepareShutdown(SedaConsumer.java:119) org.apache.camel.impl.DefaultShutdownStrategy.prepareShutdown(DefaultShutdownStrategy.java:385) org.apache.camel.impl.DefaultShutdownStrategy.doShutdown(DefaultShutdownStrategy.java:198) org.apache.camel.impl.DefaultShutdownStrategy.suspend(DefaultShutdownStrategy.java:126) org.apache.camel.impl.DefaultCamelContext.suspendRoute(DefaultCamelContext.java:868) - locked org.apache.camel.spring.SpringCamelContext@20d417bd org.apache.camel.test.suspend_route.SuspendRouteTest$1.call(SuspendRouteTest.java:101) org.apache.camel.test.suspend_route.SuspendRouteTest$1.call(SuspendRouteTest.java:97) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) java.util.concurrent.FutureTask.run(FutureTask.java:138) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) java.lang.Thread.run(Thread.java:680)
The call to suspend a route is handled by another thread called ShutdownTask but that thread is waiting for the consumers of the Camel route to shut down:
Name: Camel (camel-server) thread #3 - ShutdownTask State: WAITING on java.util.concurrent.CountDownLatch$Sync@439ddf4e Total blocked: 0 Total waited: 14 Stack trace: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281) java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207) org.apache.camel.component.seda.SedaConsumer.prepareShutdown(SedaConsumer.java:119) org.apache.camel.impl.DefaultShutdownStrategy.prepareShutdown(DefaultShutdownStrategy.java:385) org.apache.camel.impl.DefaultShutdownStrategy.access$100(DefaultShutdownStrategy.java:98) org.apache.camel.impl.DefaultShutdownStrategy$ShutdownTask.run(DefaultShutdownStrategy.java:546) java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) java.util.concurrent.FutureTask.run(FutureTask.java:138) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) java.lang.Thread.run(Thread.java:680)
The concurrent Camel consumer threads are stuck creating a new error handler for the recipientList as they are blocked with this stack trace
Name: Camel (camel-server) thread #4 - seda://TEST_IN State: BLOCKED on org.apache.camel.spring.SpringCamelContext@20d417bd owned by: pool-4-thread-1 Total blocked: 1 Total waited: 0 Stack trace: org.apache.camel.impl.DefaultCamelContext.getErrorHandlerExecutorService(DefaultCamelContext.java:1288) org.apache.camel.processor.RedeliveryErrorHandler.doStart(RedeliveryErrorHandler.java:1068) org.apache.camel.support.ChildServiceSupport.start(ChildServiceSupport.java:41) org.apache.camel.support.ChildServiceSupport.start(ChildServiceSupport.java:28) org.apache.camel.util.ServiceHelper.startService(ServiceHelper.java:62) org.apache.camel.util.ServiceHelper.startService(ServiceHelper.java:52) org.apache.camel.util.ServiceHelper.startServices(ServiceHelper.java:73) org.apache.camel.processor.DelegateAsyncProcessor.doStart(DelegateAsyncProcessor.java:78) org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:60) org.apache.camel.util.ServiceHelper.startService(ServiceHelper.java:62) org.apache.camel.util.ServiceHelper.startService(ServiceHelper.java:52) org.apache.camel.util.ServiceHelper.startServices(ServiceHelper.java:73) org.apache.camel.processor.DelegateAsyncProcessor.doStart(DelegateAsyncProcessor.java:78) org.apache.camel.processor.UnitOfWorkProcessor.doStart(UnitOfWorkProcessor.java:88) org.apache.camel.support.ServiceSupport.start(ServiceSupport.java:60) org.apache.camel.util.ServiceHelper.startService(ServiceHelper.java:62) org.apache.camel.util.ServiceHelper.startService(ServiceHelper.java:52) org.apache.camel.util.ServiceHelper.startServices(ServiceHelper.java:73) org.apache.camel.processor.MulticastProcessor.createErrorHandler(MulticastProcessor.java:876) org.apache.camel.processor.RecipientListProcessor.createProcessorExchangePair(RecipientListProcessor.java:199) org.apache.camel.processor.RecipientListProcessor.createProcessorExchangePairs(RecipientListProcessor.java:174) org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:206) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) org.apache.camel.processor.RecipientList.sendToRecipientList(RecipientList.java:151) org.apache.camel.processor.RecipientList.process(RecipientList.java:112) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) org.apache.camel.processor.Pipeline.process(Pipeline.java:117) org.apache.camel.processor.Pipeline.process(Pipeline.java:80) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99) org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:73) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99) org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99) org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) org.apache.camel.fabric.FabricTraceProcessor.process(FabricTraceProcessor.java:81) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:334) org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:220) org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45) org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:303) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) org.apache.camel.processor.Pipeline.process(Pipeline.java:117) org.apache.camel.processor.Pipeline.process(Pipeline.java:80) org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45) org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45) org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150) org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117) org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:48) org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99) org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90) org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:73) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73) org.apache.camel.component.seda.SedaConsumer.sendToConsumers(SedaConsumer.java:275) org.apache.camel.component.seda.SedaConsumer.doRun(SedaConsumer.java:183) org.apache.camel.component.seda.SedaConsumer.run(SedaConsumer.java:139) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) java.lang.Thread.run(Thread.java:680)
These threads are waiting for a lock that is held by the thread which invoked the suspend command.