Details
-
Bug
-
Resolution: Done
-
Major
-
JBoss A-MQ 6.2.1
-
None
Description
The AMQ had this issue since last boot-up .There are 3 Fuse containers in this environment and each is connecting to standalone AMQ in the same machine. It seems that AMQ on second node is having issue.
Found one Java-level deadlock:
=============================
"Thread-4360":
waiting to lock monitor 0x00007ff1bc027748 (object 0x00000006c13a51c8, a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor),
which is held by "ActiveMQ BrokerService.worker.1"
"ActiveMQ BrokerService.worker.1":
waiting for ownable synchronizer 0x00000006c13a6a08, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
which is held by "ActiveMQ Transport: tcp:///10.2.41.51:45396@61616"
"ActiveMQ Transport: tcp:///10.2.41.51:45396@61616":
waiting to lock monitor 0x00007ff1bc027748 (object 0x00000006c13a51c8, a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor),
which is held by "ActiveMQ BrokerService.worker.1"
Java stack information for the threads listed above:
===================================================
"Thread-4360":
at org.apache.activemq.broker.region.cursors.AbstractPendingMessageCursor.isCacheEnabled(AbstractPendingMessageCursor.java:332)
- waiting to lock <0x00000006c13a51c8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.isCacheEnabled(StoreQueueCursor.java:308)
at org.apache.activemq.broker.jmx.QueueView.isCacheEnabled(QueueView.java:197)
at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)
at com.sun.jmx.mbeanserver.MBeanSupport.getAttributes(MBeanSupport.java:213)
at javax.management.StandardMBean.getAttributes(StandardMBean.java:390)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttributes(DefaultMBeanServerInterceptor.java:709)
at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttributes(JmxMBeanServer.java:705)
at org.apache.activemq.console.filter.MBeansAttributeQueryFilter.getMBeanAttributes(MBeansAttributeQueryFilter.java:150)
at org.apache.activemq.console.filter.MBeansAttributeQueryFilter.getMBeanAttributesCollection(MBeansAttributeQueryFilter.java:85)
at org.apache.activemq.console.filter.MBeansAttributeQueryFilter.query(MBeansAttributeQueryFilter.java:65)
at org.apache.activemq.console.filter.AbstractQueryFilter.query(AbstractQueryFilter.java:43)
at org.apache.activemq.console.util.JmxMBeansUtil.queryMBeans(JmxMBeansUtil.java:77)
at org.apache.activemq.console.util.JmxMBeansUtil.queryMBeansAsMap(JmxMBeansUtil.java:62)
at org.apache.activemq.console.command.QueryCommand.runTask(QueryCommand.java:121)
at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:62)
at org.apache.activemq.console.command.AbstractJmxCommand.execute(AbstractJmxCommand.java:387)
at org.apache.activemq.karaf.commands.ActiveMQCommandSupport.doExecute(ActiveMQCommandSupport.java:49)
at org.apache.karaf.shell.console.OsgiCommandSupport.execute(OsgiCommandSupport.java:38)
at org.apache.felix.gogo.commands.basic.AbstractCommand.execute(AbstractCommand.java:35)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.aries.proxy.impl.ProxyHandler$1.invoke(ProxyHandler.java:54)
at org.apache.aries.proxy.impl.ProxyHandler.invoke(ProxyHandler.java:119)
at org.apache.activemq.karaf.commands.$ActiveMQCommand792184750.execute(Unknown Source)
at org.apache.felix.gogo.runtime.CommandProxy.execute(CommandProxy.java:78)
at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:477)
at org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:403)
at org.apache.felix.gogo.runtime.Pipe.run(Pipe.java:108)
at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:183)
at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:120)
at org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:92)
at org.apache.karaf.shell.console.jline.Console.run(Console.java:197)
at org.apache.karaf.shell.ssh.ShellFactoryImpl$ShellImpl$1.runConsole(ShellFactoryImpl.java:173)
at org.apache.karaf.shell.ssh.ShellFactoryImpl$ShellImpl$1$1.run(ShellFactoryImpl.java:125)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.karaf.jaas.modules.JaasHelper.doAs(JaasHelper.java:47)
at org.apache.karaf.shell.ssh.ShellFactoryImpl$ShellImpl$1.run(ShellFactoryImpl.java:123)
"ActiveMQ BrokerService.worker.1":
at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006c13a6a08> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
at org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1782)
at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.discardExpiredMessage(FilePendingMessageCursor.java:474)
at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.expireOldMessages(FilePendingMessageCursor.java:420) - locked <0x00000006c13a51c8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.onUsageChanged(FilePendingMessageCursor.java:398) - locked <0x00000006c13a51c8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
at org.apache.activemq.usage.Usage$1.run(Usage.java:304)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
"ActiveMQ Transport: tcp:///10.2.41.51:45396@61616":
at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.addMessageLast(FilePendingMessageCursor.java:207) - waiting to lock <0x00000006c13a51c8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.addMessageLast(StoreQueueCursor.java:97) - locked <0x00000006c13a6a58> (a org.apache.activemq.broker.region.cursors.StoreQueueCursor)
at org.apache.activemq.broker.region.Queue.cursorAdd(Queue.java:1796)
at org.apache.activemq.broker.region.Queue.orderedCursorAdd(Queue.java:869)
at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:845)
at org.apache.activemq.broker.region.Queue.send(Queue.java:723)
at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:419)
at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:468)
at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:296)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:152)
at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:293)
at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:152)
at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:157)
at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:546)
at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:768)
at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:338)
at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:188)
at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:300)
at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
at java.lang.Thread.run(Thread.java:745)