Uploaded image for project: 'AMQ Clients'
  1. AMQ Clients
  2. ENTMQCL-994

[qpid-jms-client 0.26.0.redhat-1, amqp-camel] after failover, consumer registered but not consuming messages

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 1.2.2.GA
    • 1.2.0.GA
    • qpid-jms
    • None
    • Hide
      • setup AMQ 7.2.0 GA replicated.
      • use default acceptor configuration
      • fill the broker with 40,000 messages using the amqp-producer (attached) using command
        mvn exec:java -Dexec.mainClass=org.apache.qpid.jms.example.FailoverSender
      • start the camel route attached (camel-amqp-reproducer) using the command
        mvn exec:java -Dexec.mainClass=com.acme.sample.Driver
      • stop master broker

      RESULT

      • "./artemis queue stat " on the slave broker shows the consumers (25 consumers) have reattached but all the messages are in a delivering state.
      • the camel route does not receive messages (route just logs the message) after the failover.
      Show
      setup AMQ 7.2.0 GA replicated. use default acceptor configuration fill the broker with 40,000 messages using the amqp-producer (attached) using command mvn exec:java -Dexec.mainClass=org.apache.qpid.jms.example.FailoverSender start the camel route attached (camel-amqp-reproducer) using the command mvn exec:java -Dexec.mainClass=com.acme.sample.Driver stop master broker RESULT "./artemis queue stat " on the slave broker shows the consumers (25 consumers) have reattached but all the messages are in a delivering state. the camel route does not receive messages (route just logs the message) after the failover.
    • Irving

    Description

      Using qpid-jms-client 0.26.0.redhat-1 (in camel-amq in Fuse 6.3 R8) against a AMQ 7.2.0 broker, after a broker failover
      the consumers did not appear to receive messages. The consumers where reported connected
      to the broker and messages where reported inflight. From the broker logging the messages appeared to be dispatched.

      The consumer threads appeared to be in this state

      "Camel (myCamel) thread #2 - JmsConsumer[testqueue6]" #16 daemon prio=5 os_prio=31 tid=0x00007fef286d5800 nid=0x6103 waiting on condition [0x000070000f48e000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x0000000774649a88> (a java.util.concurrent.CountDownLatch$Sync)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
      	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
      	at org.apache.qpid.jms.provider.ProviderFuture.sync(ProviderFuture.java:97)
      	at org.apache.qpid.jms.JmsConnection.pull(JmsConnection.java:825)
      	at org.apache.qpid.jms.JmsConnection.pull(JmsConnection.java:814)
      	at org.apache.qpid.jms.JmsMessageConsumer.performPullIfRequired(JmsMessageConsumer.java:694)
      	at org.apache.qpid.jms.JmsMessageConsumer.dequeue(JmsMessageConsumer.java:316)
      	at org.apache.qpid.jms.JmsMessageConsumer.receive(JmsMessageConsumer.java:197)
      	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:430)
      	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:310)
      	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
      	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1103)
      	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1095)
      	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:992)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      Camel configuration used

      <!-- ActiveMQ Connection Factory Bean -->
        <bean class="org.apache.qpid.jms.JmsConnectionFactory" id="jmsConnectionFactory">
           <property name="remoteURI" value="failover:(amqp://localhost:61616,amqp://localhost:61716)?failover.initialReconnectDelay=5000"/>
           <property name="username" value="admin"/>
           <property name="password" value="admin"/>
        </bean>
      
      
        <!-- JMS (Java Message Service) Bean -->
        <bean class="org.apache.camel.component.jms.JmsConfiguration" id="jmsConfig">
           <property name="connectionFactory" ref="jmsConnectionFactory"/>
           <property name="concurrentConsumers" value="25"/>
        </bean>
      
        <!-- ActiveMQ Bean -->
        <bean class="org.apache.camel.component.amqp.AMQPComponent" id="activemq">
           <property name="configuration" ref="jmsConfig"/>
        </bean>
      

      NOTE: I could consistently get this issue to occur with 25 concurrent consumers in my camel route. If I use 5 or 10 the issue did not seem to appear (at least not consistently)

      Attachments

        1. amqp-producer.tar.gz
          5 kB
        2. camel_heap.hprof.gz_parta
          25.00 MB
        3. camel_heap.hprof.gz_partb
          25.00 MB
        4. camel_heap.hprof.gz_partc
          24.46 MB
        5. camel_thread_dumps.tar.gz
          221 kB
        6. camel_threaddumps.tar.gz
          119 kB
        7. camel.log.gz
          22.52 MB
        8. camel.log.gz
          1.88 MB
        9. camel-amqp-reproducer.tar.gz
          20 kB
        10. tdumps-camel.tar
          301 kB
        11. trace.log.gz
          999 kB

        Issue Links

          Activity

            People

              gemmellr Robbie Gemmell
              rhn-support-pfox Patrick Fox (Inactive)
              Jiri Daněk Jiri Daněk
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: