Uploaded image for project: 'JBoss A-MQ'
  1. JBoss A-MQ
  2. ENTMQ-1128

With ActiveMQ RAR, consumers do not reconnect after being closed by slow-consumer detection, with failover: protocol

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • JBoss A-MQ 6.2.1
    • 6.1.1
    • None
    • Hide

      1. Configure EAP with the ActiveMQ resource adapter, as described here:

      https://access.redhat.com/documentation/en-US/Red_Hat_JBoss_A-MQ/6.1/html-single/Integrating_with_JBoss_Enterprise_Application_Platform/#DeployRar-Failover

      The connection protocol should include failover: but only a single broker is needed to reproduce the problem. So in standalone.xml (or whatever), we have:

      <config-property name="ServerUrl">
                              failover:(tcp://localhost:61616)?maxReconnectAttempts=0
      </config-property>
      

      2. Enable slow-consumer detection in the broker:

       <slowConsumerStrategy>
                          <abortSlowAckConsumerStrategy abortConnection="true" maxTimeSinceLastAck="3000" checkPeriod="3000" ignoreIdleConsumers="true"/>
      </slowConsumerStrategy
      

      I have set the slow consumer timeout to a short value (3 seconds) just to make the test quicker.

      3. Build the attached project eap-ora-mdb-demo.zip and deploy the resulting JAR on EAP (e.g., copy to standalone/deployments/) This test project implements a single MDB that writes a log message, waits two seconds, and then writes another. Please ignore all the commented-out Oracle stuff – I thought that this issue was related to two-phase commit, but it is not.

      4. Using an JMS client, post a message on queue/testqueue, and check that a message appears in the EAP log (so it's all working)

      5. Post a large number of messages to queue/testqueue

      6. While messages are being consumed, pause the EAP JVM by doing "kill -STOP" on its PID

      7. Wait for the broker log to show that the consumer has been disconnected for being too slow. Check that the physical connection has been broker, e.g., using netstat

      8. Resume EAP using "kill -CONT"

      9. Note that, eventually, connections are reestablished, but no messages are consumed. The stack dump shows threads stuck in ResponseCorrelator.

      Show
      1. Configure EAP with the ActiveMQ resource adapter, as described here: https://access.redhat.com/documentation/en-US/Red_Hat_JBoss_A-MQ/6.1/html-single/Integrating_with_JBoss_Enterprise_Application_Platform/#DeployRar-Failover The connection protocol should include failover: but only a single broker is needed to reproduce the problem. So in standalone.xml (or whatever), we have: <config-property name= "ServerUrl" > failover:(tcp: //localhost:61616)?maxReconnectAttempts=0 </config-property> 2. Enable slow-consumer detection in the broker: <slowConsumerStrategy> <abortSlowAckConsumerStrategy abortConnection= " true " maxTimeSinceLastAck= "3000" checkPeriod= "3000" ignoreIdleConsumers= " true " /> </slowConsumerStrategy I have set the slow consumer timeout to a short value (3 seconds) just to make the test quicker. 3. Build the attached project eap-ora-mdb-demo.zip and deploy the resulting JAR on EAP (e.g., copy to standalone/deployments/) This test project implements a single MDB that writes a log message, waits two seconds, and then writes another. Please ignore all the commented-out Oracle stuff – I thought that this issue was related to two-phase commit, but it is not. 4. Using an JMS client, post a message on queue/testqueue, and check that a message appears in the EAP log (so it's all working) 5. Post a large number of messages to queue/testqueue 6. While messages are being consumed, pause the EAP JVM by doing "kill -STOP" on its PID 7. Wait for the broker log to show that the consumer has been disconnected for being too slow. Check that the physical connection has been broker, e.g., using netstat 8. Resume EAP using "kill -CONT" 9. Note that, eventually, connections are reestablished, but no messages are consumed. The stack dump shows threads stuck in ResponseCorrelator.

    Description

      When slow consumer detection is enabled in the ActiveMQ broker, and an application on EAP uses MDBs to consume from a broker queue, and the connection protocol is failover:xxx, then some consumers do not reconnect after slow consumer detection aborts them. Physical connections are re-established to the broker, but the consumers are not recreated. No further messages ar consumers, and the application stalls. A restart of EAP is needed to recover.

      Attachments

        1. activemq.xml
          5 kB
        2. eap-mdb-ora-demo.zip
          13 kB
        3. jstack.out
          130 kB
        4. server_trace.log
          115 kB
        5. standalone.xml
          20 kB
        6. threads.txt
          77 kB

        Activity

          People

            gtully@redhat.com Gary Tully
            rhn-support-kboone Kevin Boone
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: