Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-494

AMQ7 beta error while consuming large number of messages using amqp python's example.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • A-MQ 7.0.0.ER17
    • A-MQ 7.0.0.ER12
    • None
    • None
    • Hide
      • Attached are python class(send and recv). These are taken from link[1].
      • If we send large number of messages around 1000000 and receive them like
        python send.py -a localhost:5672/jms.queue.ex2 -m 1000000
        -- wait for it to end
        python recv.py -a localhost:5672/jms.queue.ex2 -m 1000000
        -- starts working but at somepoint stops and i get that exception
        
      • While receiving around 75k out of 1000000 messages, we get following error.
        16:03:56,850 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: removing consumer which did not handle a message, consumer=ServerConsumerImpl [id=0, filter=null, binding=LocalQueueBinding [address=jms.queue.ex4, queue=QueueImpl[name=jms.queue.ex4, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=6ec953a9-cc1b-11e6-a214-da072f5c0dec]]@5d489dcd, filter=null, name=jms.queue.ex4, clusterName=jms.queue.ex46ec953a9-cc1b-11e6-a214-da072f5c0dec]], message=PagedReferenceImpl [position=PagePositionImpl [pageNr=3, messageNr=0, recordID=-1], message=PagedMessageImpl [queueIDs=[3076915], transactionID=-1, message=ServerMessage[messageID=3753733,durable=false,userID=null,priority=4, bodySize=1024, timestamp=0,expiration=0, durable=false, address=jms.queue.ex4,properties=TypedProperties[JMS_AMQP_NATIVE=true,NATIVE_MESSAGE_ID=AMQP_ULONG:76724,JMS_AMQP_FirstAcquirer=false,JMSXDeliveryCount=0,JMSXGroupSeq=0,JMS_AMQP_MESSAGE_FORMAT=0]]@540739317], deliveryTime=0, persistedCount=0, deliveryCount=1, subscription=PageSubscriptionImpl [cursorId=3076915, queue=QueueImpl[name=jms.queue.ex4, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=6ec953a9-cc1b-11e6-a214-da072f5c0dec]]@5d489dcd, filter = null]]: java.lang.IllegalStateException: Can't deliver message ActiveMQAMQPInternalErrorException[errorType=INTERNAL_ERROR message=Unknown constructor]
            at org.apache.activemq.artemis.core.protocol.proton.plug.ProtonSessionIntegrationCallback.sendMessage(ProtonSessionIntegrationCallback.java:474) [artemis-amqp-protocol-1.3.0.amq-700005-redhat-1.jar:]
            at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.deliverStandardMessage(ServerConsumerImpl.java:1043) [artemis-server-1.3.0.amq-700005-redhat-1.jar:1.3.0.amq-700005-redhat-1]
            at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.proceedDeliver(ServerConsumerImpl.java:417) [artemis-server-1.3.0.amq-700005-redhat-1.jar:1.3.0.amq-700005-redhat-1]
            at org.apache.activemq.artemis.core.server.impl.QueueImpl.proceedDeliver(QueueImpl.java:2507) [artemis-server-1.3.0.amq-700005-redhat-1.jar:1.3.0.amq-700005-redhat-1]
            at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:1999) [artemis-server-1.3.0.amq-700005-redhat-1.jar:1.3.0.amq-700005-redhat-1]
            at org.apache.activemq.artemis.core.server.impl.QueueImpl.access$1500(QueueImpl.java:99) [artemis-server-1.3.0.amq-700005-redhat-1.jar:1.3.0.amq-700005-redhat-1]
            at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2748) [artemis-server-1.3.0.amq-700005-redhat-1.jar:1.3.0.amq-700005-redhat-1]
            at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:103) [artemis-core-client-1.3.0.amq-700005-redhat-1.jar:1.3.0.amq-700005-redhat-1]
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_77]
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_77]
            at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_77]
        Caused by: ActiveMQAMQPInternalErrorException[errorType=INTERNAL_ERROR message=Unknown constructor]
            at org.proton.plug.context.server.ProtonServerSenderContext.deliverMessage(ProtonServerSenderContext.java:433) [artemis-proton-plug-1.3.0.amq-700005-redhat-1.jar:1.3.0.amq-700005-redhat-1]
            at org.apache.activemq.artemis.core.protocol.proton.plug.ProtonSessionIntegrationCallback.sendMessage(ProtonSessionIntegrationCallback.java:467) [artemis-amqp-protocol-1.3.0.amq-700005-redhat-1.jar:]
            ... 10 more
        Caused by: org.apache.qpid.proton.codec.DecodeException: Unknown constructor
            at org.apache.qpid.proton.codec.DecoderImpl.readObject(DecoderImpl.java:883) [proton-j-0.12.0.redhat-1.jar:0.12.0.redhat-1]
            at org.apache.qpid.proton.codec.DecoderImpl.readConstructor(DecoderImpl.java:62) [proton-j-0.12.0.redhat-1.jar:0.12.0.redhat-1]
            at org.apache.qpid.proton.codec.DecoderImpl.readObject(DecoderImpl.java:880) [proton-j-0.12.0.redhat-1.jar:0.12.0.redhat-1]
            at org.apache.qpid.proton.codec.DecoderImpl.readConstructor(DecoderImpl.java:62) [proton-j-0.12.0.redhat-1.jar:0.12.0.redhat-1]
            at org.apache.qpid.proton.codec.DecoderImpl.readObject(DecoderImpl.java:880) [proton-j-0.12.0.redhat-1.jar:0.12.0.redhat-1]
            at org.apache.qpid.proton.message.impl.MessageImpl.decode(MessageImpl.java:672) [proton-j-0.12.0.redhat-1.jar:0.12.0.redhat-1]
            at org.apache.qpid.proton.message.impl.MessageImpl.decode(MessageImpl.java:574) [proton-j-0.12.0.redhat-1.jar:0.12.0.redhat-1]
            at org.apache.activemq.artemis.core.protocol.proton.converter.message.AMQPNativeOutboundTransformer.transform(AMQPNativeOutboundTransformer.java:43) [artemis-amqp-protocol-1.3.0.amq-700005-redhat-1.jar:]
            at org.apache.activemq.artemis.core.protocol.proton.converter.ProtonMessageConverter.outbound(ProtonMessageConverter.java:99) [artemis-amqp-protocol-1.3.0.amq-700005-redhat-1.jar:]
            at org.apache.activemq.artemis.core.protocol.proton.plug.ProtonSessionIntegrationCallback.encodeMessage(ProtonSessionIntegrationCallback.java:281) [artemis-amqp-protocol-1.3.0.amq-700005-redhat-1.jar:]
            at org.proton.plug.context.server.ProtonServerSenderContext.deliverMessage(ProtonServerSenderContext.java:429) [artemis-proton-plug-1.3.0.amq-700005-redhat-1.jar:1.3.0.amq-700005-redhat-1]
            ... 11 more
        
      • Even if we restart broker multiple times, issue still remains.
      • It is observed that while receiving if we set count to 50K than error is not seen. So seems to be related to ThreadPoolExecutor which we see in stack-trace too.

      [1] https://qpid.apache.org/releases/qpid-proton-0.10/proton/python/tutorial/tutorial.html#asynchronous-send-and-receive

      Show
      Attached are python class(send and recv). These are taken from link [1] . If we send large number of messages around 1000000 and receive them like python send.py -a localhost:5672/jms.queue.ex2 -m 1000000 -- wait for it to end python recv.py -a localhost:5672/jms.queue.ex2 -m 1000000 -- starts working but at somepoint stops and i get that exception While receiving around 75k out of 1000000 messages, we get following error. 16:03:56,850 WARN [org.apache.activemq.artemis.core.server] AMQ222151: removing consumer which did not handle a message, consumer=ServerConsumerImpl [id=0, filter= null , binding=LocalQueueBinding [address=jms.queue.ex4, queue=QueueImpl[name=jms.queue.ex4, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=6ec953a9-cc1b-11e6-a214-da072f5c0dec]]@5d489dcd, filter= null , name=jms.queue.ex4, clusterName=jms.queue.ex46ec953a9-cc1b-11e6-a214-da072f5c0dec]], message=PagedReferenceImpl [position=PagePositionImpl [pageNr=3, messageNr=0, recordID=-1], message=PagedMessageImpl [queueIDs=[3076915], transactionID=-1, message=ServerMessage[messageID=3753733,durable= false ,userID= null ,priority=4, bodySize=1024, timestamp=0,expiration=0, durable= false , address=jms.queue.ex4,properties=TypedProperties[JMS_AMQP_NATIVE= true ,NATIVE_MESSAGE_ID=AMQP_ULONG:76724,JMS_AMQP_FirstAcquirer= false ,JMSXDeliveryCount=0,JMSXGroupSeq=0,JMS_AMQP_MESSAGE_FORMAT=0]]@540739317], deliveryTime=0, persistedCount=0, deliveryCount=1, subscription=PageSubscriptionImpl [cursorId=3076915, queue=QueueImpl[name=jms.queue.ex4, postOffice=PostOfficeImpl [server=ActiveMQServerImpl::serverUUID=6ec953a9-cc1b-11e6-a214-da072f5c0dec]]@5d489dcd, filter = null ]]: java.lang.IllegalStateException: Can't deliver message ActiveMQAMQPInternalErrorException[errorType=INTERNAL_ERROR message=Unknown constructor] at org.apache.activemq.artemis.core.protocol.proton.plug.ProtonSessionIntegrationCallback.sendMessage(ProtonSessionIntegrationCallback.java:474) [artemis-amqp-protocol-1.3.0.amq-700005-redhat-1.jar:] at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.deliverStandardMessage(ServerConsumerImpl.java:1043) [artemis-server-1.3.0.amq-700005-redhat-1.jar:1.3.0.amq-700005-redhat-1] at org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.proceedDeliver(ServerConsumerImpl.java:417) [artemis-server-1.3.0.amq-700005-redhat-1.jar:1.3.0.amq-700005-redhat-1] at org.apache.activemq.artemis.core.server.impl.QueueImpl.proceedDeliver(QueueImpl.java:2507) [artemis-server-1.3.0.amq-700005-redhat-1.jar:1.3.0.amq-700005-redhat-1] at org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:1999) [artemis-server-1.3.0.amq-700005-redhat-1.jar:1.3.0.amq-700005-redhat-1] at org.apache.activemq.artemis.core.server.impl.QueueImpl.access$1500(QueueImpl.java:99) [artemis-server-1.3.0.amq-700005-redhat-1.jar:1.3.0.amq-700005-redhat-1] at org.apache.activemq.artemis.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2748) [artemis-server-1.3.0.amq-700005-redhat-1.jar:1.3.0.amq-700005-redhat-1] at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:103) [artemis-core-client-1.3.0.amq-700005-redhat-1.jar:1.3.0.amq-700005-redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_77] at java.lang. Thread .run( Thread .java:745) [rt.jar:1.8.0_77] Caused by: ActiveMQAMQPInternalErrorException[errorType=INTERNAL_ERROR message=Unknown constructor] at org.proton.plug.context.server.ProtonServerSenderContext.deliverMessage(ProtonServerSenderContext.java:433) [artemis-proton-plug-1.3.0.amq-700005-redhat-1.jar:1.3.0.amq-700005-redhat-1] at org.apache.activemq.artemis.core.protocol.proton.plug.ProtonSessionIntegrationCallback.sendMessage(ProtonSessionIntegrationCallback.java:467) [artemis-amqp-protocol-1.3.0.amq-700005-redhat-1.jar:] ... 10 more Caused by: org.apache.qpid.proton.codec.DecodeException: Unknown constructor at org.apache.qpid.proton.codec.DecoderImpl.readObject(DecoderImpl.java:883) [proton-j-0.12.0.redhat-1.jar:0.12.0.redhat-1] at org.apache.qpid.proton.codec.DecoderImpl.readConstructor(DecoderImpl.java:62) [proton-j-0.12.0.redhat-1.jar:0.12.0.redhat-1] at org.apache.qpid.proton.codec.DecoderImpl.readObject(DecoderImpl.java:880) [proton-j-0.12.0.redhat-1.jar:0.12.0.redhat-1] at org.apache.qpid.proton.codec.DecoderImpl.readConstructor(DecoderImpl.java:62) [proton-j-0.12.0.redhat-1.jar:0.12.0.redhat-1] at org.apache.qpid.proton.codec.DecoderImpl.readObject(DecoderImpl.java:880) [proton-j-0.12.0.redhat-1.jar:0.12.0.redhat-1] at org.apache.qpid.proton.message.impl.MessageImpl.decode(MessageImpl.java:672) [proton-j-0.12.0.redhat-1.jar:0.12.0.redhat-1] at org.apache.qpid.proton.message.impl.MessageImpl.decode(MessageImpl.java:574) [proton-j-0.12.0.redhat-1.jar:0.12.0.redhat-1] at org.apache.activemq.artemis.core.protocol.proton.converter.message.AMQPNativeOutboundTransformer.transform(AMQPNativeOutboundTransformer.java:43) [artemis-amqp-protocol-1.3.0.amq-700005-redhat-1.jar:] at org.apache.activemq.artemis.core.protocol.proton.converter.ProtonMessageConverter.outbound(ProtonMessageConverter.java:99) [artemis-amqp-protocol-1.3.0.amq-700005-redhat-1.jar:] at org.apache.activemq.artemis.core.protocol.proton.plug.ProtonSessionIntegrationCallback.encodeMessage(ProtonSessionIntegrationCallback.java:281) [artemis-amqp-protocol-1.3.0.amq-700005-redhat-1.jar:] at org.proton.plug.context.server.ProtonServerSenderContext.deliverMessage(ProtonServerSenderContext.java:429) [artemis-proton-plug-1.3.0.amq-700005-redhat-1.jar:1.3.0.amq-700005-redhat-1] ... 11 more Even if we restart broker multiple times, issue still remains. It is observed that while receiving if we set count to 50K than error is not seen. So seems to be related to ThreadPoolExecutor which we see in stack-trace too. [1] https://qpid.apache.org/releases/qpid-proton-0.10/proton/python/tutorial/tutorial.html#asynchronous-send-and-receive

    Attachments

      1. data.zip
        12.07 MB
      2. journal.zip
        23 kB
      3. sendAndrecv.zip
        1 kB

      Issue Links

        Activity

          People

            rh-ee-ataylor Andy Taylor
            rhn-support-cpandey Chandra Shekhar Pandey (Inactive)
            Petr Matousek Petr Matousek
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: