Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-3506

Creating and deleting addresses over JMX fails at random

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • None
    • AMQ 7.7.0.CR1
    • None

      Download http://messaging-qe-repo.usersys.redhat.com:8081/artifactory/cli-java/amqx-2.9.0.redhat-00005.jar, change broker name from 0.0.0.0 to "amq", enable jmx.

      Run the following against Artemis 2.11 broker. There are three errorneous situations that can happen, when given enough time.

      i=b
      while true; do
      java  -jar amqx-2.9.0.redhat-00005.jar  topic --host 127.0.0.1:1099 --username admin --password admin --action add --name tests.${i}.results.success || break
      java  -jar amqx-2.9.0.redhat-00005.jar  topic --host 127.0.0.1:1099 --username admin --password admin --action add --name tests.${i}.results.failure || break
      java  -jar amqx-2.9.0.redhat-00005.jar  topic --host 127.0.0.1:1099 --username admin --password admin --action remove --name tests.${i}.results.success || break
      java  -jar amqx-2.9.0.redhat-00005.jar  topic --host 127.0.0.1:1099 --username admin --password admin --action remove --name tests.${i}.results.failure || break
      i+=1
      done;
      

      1. The broker fails to find previously created address and error is shown in server log, JMX client gets reported success

      Note that error message mentions an auto-created queue, but the destination was actually created explicitly.

      2020-04-21 23:09:18,736 ERROR [org.apache.activemq.artemis.core.server] AMQ224065: Failed to remove auto-created queue tests.b1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111.results.failure: ActiveMQAddressDoesNotExistException[errorType=ADDRESS_DOES_NOT_EXIST message=AMQ229203: Address Does Not Exist: tests.b1111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111.results.failure]
      	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.removeAddressInfo(ActiveMQServerImpl.java:3303) [artemis-server-2.11.0.jar:2.11.0]
      	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.removeAddressInfo(ActiveMQServerImpl.java:3292) [artemis-server-2.11.0.jar:2.11.0]
      	at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl$AddressQueueReaper.run(PostOfficeImpl.java:1762) [artemis-server-2.11.0.jar:2.11.0]
      	at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent$2.run(ActiveMQScheduledComponent.java:306) [artemis-commons-2.11.0.jar:2.11.0]
      	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42) [artemis-commons-2.11.0.jar:2.11.0]
      	at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31) [artemis-commons-2.11.0.jar:2.11.0]
      	at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:66) [artemis-commons-2.11.0.jar:2.11.0]
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [java.base:]
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [java.base:]
      	at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) [artemis-commons-2.11.0.jar:2.11.0]
      

      2. The broker fails to find previously created address and error is thrown in JMX client

      java.lang.IllegalStateException: AMQ229203: Address Does Not Exist: tests.b111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111.results.failure
              at org.apache.activemq.artemis.core.management.impl.ActiveMQServerControlImpl.deleteAddress(ActiveMQServerControlImpl.java:870)
              at org.apache.activemq.artemis.core.management.impl.ActiveMQServerControlImpl.deleteAddress(ActiveMQServerControlImpl.java:856)
              at jdk.internal.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
              at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.base/java.lang.reflect.Method.invoke(Method.java:567)
              at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
              at jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
              at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.base/java.lang.reflect.Method.invoke(Method.java:567)
              at java.base/sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:260)
              at java.management/com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
              at java.management/com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
              at java.management/com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
              at java.management/com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
              at java.management/com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
              at java.management/javax.management.StandardMBean.invoke(StandardMBean.java:405)
              at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:809)
              at java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
              at java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1466)
              at java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1307)
              at java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1399)
              at java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:827)
              at java.base/jdk.internal.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
              at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.base/java.lang.reflect.Method.invoke(Method.java:567)
              at java.rmi/sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:359)
              at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:200)
              at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:197)
              at java.base/java.security.AccessController.doPrivileged(AccessController.java:690)
              at java.rmi/sun.rmi.transport.Transport.serviceCall(Transport.java:196)
              at java.rmi/sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:562)
              at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:796)
              at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:677)
              at java.base/java.security.AccessController.doPrivileged(AccessController.java:391)
              at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:676)
              at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
              at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
              at java.base/java.lang.Thread.run(Thread.java:835)
      

      3. When multiple instances of the script above are run at the same time, it can happen that error result about one command is delivered to the other command.

      % i=a
      while true; do
      java  -jar amqx-2.9.0.redhat-00005.jar  topic --host 127.0.0.1:1099 --username admin --password admin --action add --name tests.${i}.results.success || break 
      java  -jar amqx-2.9.0.redhat-00005.jar  topic --host 127.0.0.1:1099 --username admin --password admin --action add --name tests.${i}.results.failure || break 
      java  -jar amqx-2.9.0.redhat-00005.jar  topic --host 127.0.0.1:1099 --username admin --password admin --action remove --name tests.${i}.results.success || break
      java  -jar amqx-2.9.0.redhat-00005.jar  topic --host 127.0.0.1:1099 --username admin --password admin --action remove --name tests.${i}.results.failure || break
      i+=1
      done;
      
      topic 'tests.a.results.success' created
      topic 'tests.a.results.failure' created
      Topic 'tests.a.results.success' removed
      Topic 'tests.a.results.failure' removed
      topic 'tests.a1.results.success' created
      topic 'tests.a1.results.failure' created
      Topic 'tests.a1.results.success' removed
      Topic 'tests.a1.results.failure' removed
      topic 'tests.a11.results.success' created
      topic 'tests.a11.results.failure' created
      Topic 'tests.a11.results.success' removed
      Topic 'tests.a11.results.failure' removed
      topic 'tests.a111.results.success' created
      Unable to add queue: org.apache.activemq.artemis:broker="amq",component=addresses,address="tests.b1.results.failure"
      javax.management.InstanceNotFoundException: org.apache.activemq.artemis:broker="amq",component=addresses,address="tests.b1.results.failure"
              at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1083)
              at java.management/com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:637)
              at java.management/com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
              at java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1443)
              at java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1307)
              at java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1399)
              at java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:637)
              at java.base/jdk.internal.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
              at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.base/java.lang.reflect.Method.invoke(Method.java:567)
              at java.rmi/sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:359)
              at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:200)
              at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:197)
              at java.base/java.security.AccessController.doPrivileged(AccessController.java:690)
              at java.rmi/sun.rmi.transport.Transport.serviceCall(Transport.java:196)
              at java.rmi/sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:562)
              at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:796)
              at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:677)
              at java.base/java.security.AccessController.doPrivileged(AccessController.java:391)
              at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:676)
              at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
              at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
              at java.base/java.lang.Thread.run(Thread.java:835)
              at java.rmi/sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:283)
              at java.rmi/sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:260)
              at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
              at jdk.remoteref/jdk.jmx.remote.internal.rmi.PRef.invoke(Unknown Source)
              at java.management.rmi/javax.management.remote.rmi.RMIConnectionImpl_Stub.getAttribute(Unknown Source)
              at java.management.rmi/javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getAttribute(RMIConnector.java:904)
              at java.management/javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:273)
              at com.sun.proxy.$Proxy4.getQueueNames(Unknown Source)
              at com.redhat.amqx.management.artemis.AbstractArtemisManager.getTopics(AbstractArtemisManager.java:77)
              at com.redhat.amqx.management.artemis.TopicArtemisManager.destinationExists(TopicArtemisManager.java:29)
              at com.redhat.amqx.management.artemis.TopicArtemisManager.addDestination(TopicArtemisManager.java:99)
              at com.redhat.amqx.main.actions.TopicAction.run(TopicAction.java:95)
              at com.redhat.amqx.main.Main.main(Main.java:60)
      

      Note that this command is creating an a* addresses, but the error message talks about a b* address.

            dbruscin Domenico Francesco Bruscino
            jdanek@redhat.com Jiri Daněk
            Tiago Bueno Tiago Bueno
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: