Uploaded image for project: 'AMQ Clients'
  1. AMQ Clients
  2. ENTMQCL-994

[qpid-jms-client 0.26.0.redhat-1, amqp-camel] after failover, consumer registered but not consuming messages

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 1.2.2.GA
    • 1.2.0.GA
    • qpid-jms
    • None
    • Hide
      • setup AMQ 7.2.0 GA replicated.
      • use default acceptor configuration
      • fill the broker with 40,000 messages using the amqp-producer (attached) using command
        mvn exec:java -Dexec.mainClass=org.apache.qpid.jms.example.FailoverSender
      • start the camel route attached (camel-amqp-reproducer) using the command
        mvn exec:java -Dexec.mainClass=com.acme.sample.Driver
      • stop master broker

      RESULT

      • "./artemis queue stat " on the slave broker shows the consumers (25 consumers) have reattached but all the messages are in a delivering state.
      • the camel route does not receive messages (route just logs the message) after the failover.
      Show
      setup AMQ 7.2.0 GA replicated. use default acceptor configuration fill the broker with 40,000 messages using the amqp-producer (attached) using command mvn exec:java -Dexec.mainClass=org.apache.qpid.jms.example.FailoverSender start the camel route attached (camel-amqp-reproducer) using the command mvn exec:java -Dexec.mainClass=com.acme.sample.Driver stop master broker RESULT "./artemis queue stat " on the slave broker shows the consumers (25 consumers) have reattached but all the messages are in a delivering state. the camel route does not receive messages (route just logs the message) after the failover.
    • Irving

    Description

      Using qpid-jms-client 0.26.0.redhat-1 (in camel-amq in Fuse 6.3 R8) against a AMQ 7.2.0 broker, after a broker failover
      the consumers did not appear to receive messages. The consumers where reported connected
      to the broker and messages where reported inflight. From the broker logging the messages appeared to be dispatched.

      The consumer threads appeared to be in this state

      "Camel (myCamel) thread #2 - JmsConsumer[testqueue6]" #16 daemon prio=5 os_prio=31 tid=0x00007fef286d5800 nid=0x6103 waiting on condition [0x000070000f48e000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x0000000774649a88> (a java.util.concurrent.CountDownLatch$Sync)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
      	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
      	at org.apache.qpid.jms.provider.ProviderFuture.sync(ProviderFuture.java:97)
      	at org.apache.qpid.jms.JmsConnection.pull(JmsConnection.java:825)
      	at org.apache.qpid.jms.JmsConnection.pull(JmsConnection.java:814)
      	at org.apache.qpid.jms.JmsMessageConsumer.performPullIfRequired(JmsMessageConsumer.java:694)
      	at org.apache.qpid.jms.JmsMessageConsumer.dequeue(JmsMessageConsumer.java:316)
      	at org.apache.qpid.jms.JmsMessageConsumer.receive(JmsMessageConsumer.java:197)
      	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveMessage(AbstractPollingMessageListenerContainer.java:430)
      	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:310)
      	at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263)
      	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1103)
      	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1095)
      	at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:992)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      Camel configuration used

      <!-- ActiveMQ Connection Factory Bean -->
        <bean class="org.apache.qpid.jms.JmsConnectionFactory" id="jmsConnectionFactory">
           <property name="remoteURI" value="failover:(amqp://localhost:61616,amqp://localhost:61716)?failover.initialReconnectDelay=5000"/>
           <property name="username" value="admin"/>
           <property name="password" value="admin"/>
        </bean>
      
      
        <!-- JMS (Java Message Service) Bean -->
        <bean class="org.apache.camel.component.jms.JmsConfiguration" id="jmsConfig">
           <property name="connectionFactory" ref="jmsConnectionFactory"/>
           <property name="concurrentConsumers" value="25"/>
        </bean>
      
        <!-- ActiveMQ Bean -->
        <bean class="org.apache.camel.component.amqp.AMQPComponent" id="activemq">
           <property name="configuration" ref="jmsConfig"/>
        </bean>
      

      NOTE: I could consistently get this issue to occur with 25 concurrent consumers in my camel route. If I use 5 or 10 the issue did not seem to appear (at least not consistently)

      Attachments

        1. amqp-producer.tar.gz
          5 kB
          Patrick Fox
        2. camel_heap.hprof.gz_parta
          25.00 MB
          Patrick Fox
        3. camel_heap.hprof.gz_partb
          25.00 MB
          Patrick Fox
        4. camel_heap.hprof.gz_partc
          24.46 MB
          Patrick Fox
        5. camel_thread_dumps.tar.gz
          221 kB
          Patrick Fox
        6. camel_threaddumps.tar.gz
          119 kB
          Patrick Fox
        7. camel.log.gz
          22.52 MB
          Patrick Fox
        8. camel.log.gz
          1.88 MB
          Patrick Fox
        9. camel-amqp-reproducer.tar.gz
          20 kB
          Patrick Fox
        10. tdumps-camel.tar
          301 kB
          Patrick Fox
        11. trace.log.gz
          999 kB
          Patrick Fox

        Issue Links

          Activity

            People

              gemmellr Robbie Gemmell
              rhn-support-pfox Patrick Fox (Inactive)
              Jiri Daněk Jiri Daněk
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: