Uploaded image for project: 'JBoss A-MQ'
  1. JBoss A-MQ
  2. ENTMQ-1708

The connection to 'tcp://...' is taking a long time to shutdown

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • JBoss A-MQ 6.3
    • JBoss A-MQ 6.2.1
    • broker
    • None

    Description

      The AMQ had this issue since last boot-up .There are 3 Fuse containers in this environment and each is connecting to standalone AMQ in the same machine. It seems that AMQ on second node is having issue.
      Found one Java-level deadlock:
      =============================
      "Thread-4360":
      waiting to lock monitor 0x00007ff1bc027748 (object 0x00000006c13a51c8, a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor),
      which is held by "ActiveMQ BrokerService.worker.1"
      "ActiveMQ BrokerService.worker.1":
      waiting for ownable synchronizer 0x00000006c13a6a08, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
      which is held by "ActiveMQ Transport: tcp:///10.2.41.51:45396@61616"
      "ActiveMQ Transport: tcp:///10.2.41.51:45396@61616":
      waiting to lock monitor 0x00007ff1bc027748 (object 0x00000006c13a51c8, a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor),
      which is held by "ActiveMQ BrokerService.worker.1"

      Java stack information for the threads listed above:
      ===================================================
      "Thread-4360":
      at org.apache.activemq.broker.region.cursors.AbstractPendingMessageCursor.isCacheEnabled(AbstractPendingMessageCursor.java:332)

      • waiting to lock <0x00000006c13a51c8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
        at org.apache.activemq.broker.region.cursors.StoreQueueCursor.isCacheEnabled(StoreQueueCursor.java:308)
        at org.apache.activemq.broker.jmx.QueueView.isCacheEnabled(QueueView.java:197)
        at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
        at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
        at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
        at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)
        at com.sun.jmx.mbeanserver.MBeanSupport.getAttributes(MBeanSupport.java:213)
        at javax.management.StandardMBean.getAttributes(StandardMBean.java:390)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttributes(DefaultMBeanServerInterceptor.java:709)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttributes(JmxMBeanServer.java:705)
        at org.apache.activemq.console.filter.MBeansAttributeQueryFilter.getMBeanAttributes(MBeansAttributeQueryFilter.java:150)
        at org.apache.activemq.console.filter.MBeansAttributeQueryFilter.getMBeanAttributesCollection(MBeansAttributeQueryFilter.java:85)
        at org.apache.activemq.console.filter.MBeansAttributeQueryFilter.query(MBeansAttributeQueryFilter.java:65)
        at org.apache.activemq.console.filter.AbstractQueryFilter.query(AbstractQueryFilter.java:43)
        at org.apache.activemq.console.util.JmxMBeansUtil.queryMBeans(JmxMBeansUtil.java:77)
        at org.apache.activemq.console.util.JmxMBeansUtil.queryMBeansAsMap(JmxMBeansUtil.java:62)
        at org.apache.activemq.console.command.QueryCommand.runTask(QueryCommand.java:121)
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:62)
        at org.apache.activemq.console.command.AbstractJmxCommand.execute(AbstractJmxCommand.java:387)
        at org.apache.activemq.karaf.commands.ActiveMQCommandSupport.doExecute(ActiveMQCommandSupport.java:49)
        at org.apache.karaf.shell.console.OsgiCommandSupport.execute(OsgiCommandSupport.java:38)
        at org.apache.felix.gogo.commands.basic.AbstractCommand.execute(AbstractCommand.java:35)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.aries.proxy.impl.ProxyHandler$1.invoke(ProxyHandler.java:54)
        at org.apache.aries.proxy.impl.ProxyHandler.invoke(ProxyHandler.java:119)
        at org.apache.activemq.karaf.commands.$ActiveMQCommand792184750.execute(Unknown Source)
        at org.apache.felix.gogo.runtime.CommandProxy.execute(CommandProxy.java:78)
        at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:477)
        at org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:403)
        at org.apache.felix.gogo.runtime.Pipe.run(Pipe.java:108)
        at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:183)
        at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:120)
        at org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:92)
        at org.apache.karaf.shell.console.jline.Console.run(Console.java:197)
        at org.apache.karaf.shell.ssh.ShellFactoryImpl$ShellImpl$1.runConsole(ShellFactoryImpl.java:173)
        at org.apache.karaf.shell.ssh.ShellFactoryImpl$ShellImpl$1$1.run(ShellFactoryImpl.java:125)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.apache.karaf.jaas.modules.JaasHelper.doAs(JaasHelper.java:47)
        at org.apache.karaf.shell.ssh.ShellFactoryImpl$ShellImpl$1.run(ShellFactoryImpl.java:123)
        "ActiveMQ BrokerService.worker.1":
        at sun.misc.Unsafe.park(Native Method)
      • parking to wait for <0x00000006c13a6a08> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
        at org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1782)
        at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.discardExpiredMessage(FilePendingMessageCursor.java:474)
        at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.expireOldMessages(FilePendingMessageCursor.java:420)
      • locked <0x00000006c13a51c8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
        at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.onUsageChanged(FilePendingMessageCursor.java:398)
      • locked <0x00000006c13a51c8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
        at org.apache.activemq.usage.Usage$1.run(Usage.java:304)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
        "ActiveMQ Transport: tcp:///10.2.41.51:45396@61616":
        at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.addMessageLast(FilePendingMessageCursor.java:207)
      • waiting to lock <0x00000006c13a51c8> (a org.apache.activemq.broker.region.cursors.FilePendingMessageCursor)
        at org.apache.activemq.broker.region.cursors.StoreQueueCursor.addMessageLast(StoreQueueCursor.java:97)
      • locked <0x00000006c13a6a58> (a org.apache.activemq.broker.region.cursors.StoreQueueCursor)
        at org.apache.activemq.broker.region.Queue.cursorAdd(Queue.java:1796)
        at org.apache.activemq.broker.region.Queue.orderedCursorAdd(Queue.java:869)
        at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:845)
        at org.apache.activemq.broker.region.Queue.send(Queue.java:723)
        at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:419)
        at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:468)
        at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:296)
        at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:152)
        at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
        at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:293)
        at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:152)
        at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:157)
        at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:546)
        at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:768)
        at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:338)
        at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:188)
        at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
        at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
        at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:300)
        at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
        at java.lang.Thread.run(Thread.java:745)

      Attachments

        Activity

          People

            gtully@redhat.com Gary Tully
            rhn-support-kkakarla kodandaRamu kakarla
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: