Uploaded image for project: 'JBoss A-MQ'
  1. JBoss A-MQ
  2. ENTMQ-2329

[AMQ6, JMX, removeMessage(java.lang.String)] JMX operation removeMessage(messageID) is not completing and leading to high CPU usage

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • JBoss A-MQ 6.3.x
    • JBoss A-MQ 6.3
    • broker
    • None
    • Hide

      Using the default configuration for AMQ 6.3.0 R12 carry out the following steps.

      1) load 360,000 messages into a queue

      2) on the hawtio "operations" pane for the queue invoke "removeMessage(java.lang.String)" giving it a message ID in the queue . This appears to complete successfully.

      3) navigate away from panel and back - invoke "removeMessage(java.lang.String)" with the SAME message ID.

      4) repeat "step 3" 4 times

      RESULT:
      The broker will have 5 threads in a RUNNABLE state within these operations

      at org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1338)
      	at org.apache.activemq.broker.jmx.QueueView.removeMessage(QueueView.java:68)
      

      Leave it for 20 minutes and the threads will remain in a RUNNABLE state but do not seem to complete. Also the threads in that state show a high amount of CPU time.

      Show
      Using the default configuration for AMQ 6.3.0 R12 carry out the following steps. 1) load 360,000 messages into a queue 2) on the hawtio "operations" pane for the queue invoke "removeMessage(java.lang.String)" giving it a message ID in the queue . This appears to complete successfully. 3) navigate away from panel and back - invoke "removeMessage(java.lang.String)" with the SAME message ID. 4) repeat "step 3" 4 times RESULT: The broker will have 5 threads in a RUNNABLE state within these operations at org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1338) at org.apache.activemq.broker.jmx.QueueView.removeMessage(QueueView.java:68) Leave it for 20 minutes and the threads will remain in a RUNNABLE state but do not seem to complete. Also the threads in that state show a high amount of CPU time.

    Description

      On a queue with a large number of messages, invoking the JMX operation removeMessage(messageID), with a message ID that has already been removed, is not completing and leading to high CPU usage

      The thread appears to remain in a RUNNABLE state.

      eg stack below

      "qtp980125874-81" #81 prio=5 os_prio=0 tid=0x00007f36789f4000 nid=0x59dd runnable [0x00007f35fdce2000]
         java.lang.Thread.State: RUNNABLE
      	at java.util.HashMap.putVal(HashMap.java:657)
      	at java.util.HashMap.put(HashMap.java:612)
      	at java.util.HashSet.add(HashSet.java:220)
      	at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
      	at org.apache.activemq.broker.region.Queue.removeMatchingMessages(Queue.java:1374)
      	at org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1338)
      	at org.apache.activemq.broker.jmx.QueueView.removeMessage(QueueView.java:68)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
      	at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
      	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
      	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
      	at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
      	at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
      	at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
      	at javax.management.StandardMBean.invoke(StandardMBean.java:405)
      	at org.apache.activemq.broker.jmx.AnnotatedMBean.invoke(AnnotatedMBean.java:243)
      	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
      	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
      	at org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:98)
      	at org.jolokia.handler.ExecHandler.doHandleRequest(ExecHandler.java:40)
      	at org.jolokia.handler.JsonRequestHandler.handleRequest(JsonRequestHandler.java:89)
      	at org.jolokia.backend.MBeanServerExecutorLocal.handleRequest(MBeanServerExecutorLocal.java:109)
      	at org.jolokia.backend.MBeanServerHandler.dispatchRequest(MBeanServerHandler.java:161)
      	at org.jolokia.backend.LocalRequestDispatcher.dispatchRequest(LocalRequestDispatcher.java:99)
      	at org.jolokia.backend.BackendManager.callRequestDispatcher(BackendManager.java:413)
      	at org.jolokia.backend.BackendManager.handleRequest(BackendManager.java:158)
      	at org.jolokia.http.HttpRequestHandler.executeRequest(HttpRequestHandler.java:197)
      	at org.jolokia.http.HttpRequestHandler.handlePostRequest(HttpRequestHandler.java:137)
      	at org.jolokia.http.AgentServlet$3.handleRequest(AgentServlet.java:420)
      	at org.jolokia.http.AgentServlet.handleSecurely(AgentServlet.java:301)
      	at org.jolokia.http.AgentServlet.handle(AgentServlet.java:277)
      	at org.jolokia.http.AgentServlet.doPost(AgentServlet.java:244)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
      	at io.hawt.web.JolokiaConfiguredAgentServlet.access$001(JolokiaConfiguredAgentServlet.java:34)
      	at io.hawt.web.JolokiaConfiguredAgentServlet$1.run(JolokiaConfiguredAgentServlet.java:76)
      	at io.hawt.web.JolokiaConfiguredAgentServlet$1.run(JolokiaConfiguredAgentServlet.java:73)
      	at io.hawt.system.Helpers.doJmxPrivileged(Helpers.java:78)
      	at io.hawt.web.JolokiaConfiguredAgentServlet.service(JolokiaConfiguredAgentServlet.java:73)
      	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
      	at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:201)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at io.hawt.web.AuthenticationFilter$3.run(AuthenticationFilter.java:201)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at io.hawt.web.AuthenticationFilter.executeAs(AuthenticationFilter.java:198)
      	at io.hawt.web.AuthenticationFilter.doFilter(AuthenticationFilter.java:136)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at io.hawt.web.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:37)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at io.hawt.web.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:37)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at io.hawt.web.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:37)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at io.hawt.web.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:37)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at io.hawt.web.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:37)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at io.hawt.web.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:37)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at io.hawt.web.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:37)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at io.hawt.web.HttpHeaderFilter.doFilter(HttpHeaderFilter.java:37)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at io.hawt.web.SessionExpiryFilter.process(SessionExpiryFilter.java:148)
      	at io.hawt.web.SessionExpiryFilter.doFilter(SessionExpiryFilter.java:46)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at io.hawt.web.RedirectFilter.process(RedirectFilter.java:73)
      	at io.hawt.web.RedirectFilter.doFilter(RedirectFilter.java:38)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
      	at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:72)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
      	at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:287)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      	at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:499)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258)
      	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
      	at java.lang.Thread.run(Thread.java:748)
      
         Locked ownable synchronizers:
      	- None
      
      

      Attachments

        1. high-cpu.out
          111 kB
        2. high-cpu-tdump.out
          722 kB

        Activity

          People

            gtully@redhat.com Gary Tully
            rhn-support-pfox Patrick Fox (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: