Uploaded image for project: 'JBoss A-MQ'
  1. JBoss A-MQ
  2. ENTMQ-1681

Issue with message ordering after transaction rollback

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • JBoss A-MQ 6.3
    • JBoss A-MQ 6.2.1
    • broker
    • None
    • CR1
    • Hide

      In order to reproduce the problem, use the attached reproducer in the following way:

      A. Unzip a plain vanilla A-MQ 6.2.1R1 broker (6.2.1.redhat-090)
      B. In activemq.xml, change the queue policy so that there is no memory limit on queues:
      <policyEntry queue=">" producerFlowControl="true">
      C. Activate (uncomment) the admin user in etc/users.properties
      D. Start the broker
      E. Run the provided "ConsumerRollbackTest" reproducer as below:
      java ConsumerRollbackTest tcp://broker_host:61616?jms.prefetchPolicy.queuePrefetch=1000 admin admin TestConsumerRollbackQueue 10000 800 10

      Note: your CLASSPATH must contain the below libraries:
      activemq-client-5.11.0.redhat-621090.jar
      geronimo-j2ee-management_1.1_spec-1.0.1.jar
      hawtbuf-1.11.jar
      jboss-jms-api_1.1_spec-1.0.1.Final-redhat-2.jar
      slf4j-api-1.7.10.jar

      The reproducer does the following: it enqueues 10.000 messages with a "MessageRank" property indicating their order from 0 to 9999.
      Then it executes the scenario described above - the initial prefetch value is 1000 and messages read are committed in a transaction every 800 messages.

      At step 3., 10 messages are consumed and rolled-back.
      At step 6., the reproducer logs messages like the below

      "SEVERE: Having read out-of-sequence message with JMSMessageId=.....1:1:1:201, MessageRank=200. Highest MessageRank so far : 1199. Having read in total 991 messages !"
      ... which means that after having read a message with MessageRank 1199, the consumer actually read a message with MessageRank 200.

      (Please note that in order to exactly run the required scenario, the reproducer might automatically execute several times in a loop, self-tuning at each iteration the receive time-out value used at step 1.)

      Show
      In order to reproduce the problem, use the attached reproducer in the following way: A. Unzip a plain vanilla A-MQ 6.2.1R1 broker (6.2.1.redhat-090) B. In activemq.xml, change the queue policy so that there is no memory limit on queues: <policyEntry queue=">" producerFlowControl="true"> C. Activate (uncomment) the admin user in etc/users.properties D. Start the broker E. Run the provided "ConsumerRollbackTest" reproducer as below: java ConsumerRollbackTest tcp://broker_host:61616?jms.prefetchPolicy.queuePrefetch=1000 admin admin TestConsumerRollbackQueue 10000 800 10 Note: your CLASSPATH must contain the below libraries: activemq-client-5.11.0.redhat-621090.jar geronimo-j2ee-management_1.1_spec-1.0.1.jar hawtbuf-1.11.jar jboss-jms-api_1.1_spec-1.0.1.Final-redhat-2.jar slf4j-api-1.7.10.jar The reproducer does the following: it enqueues 10.000 messages with a "MessageRank" property indicating their order from 0 to 9999. Then it executes the scenario described above - the initial prefetch value is 1000 and messages read are committed in a transaction every 800 messages. At step 3., 10 messages are consumed and rolled-back. At step 6., the reproducer logs messages like the below "SEVERE: Having read out-of-sequence message with JMSMessageId=.....1:1:1:201, MessageRank=200. Highest MessageRank so far : 1199. Having read in total 991 messages !" ... which means that after having read a message with MessageRank 1199, the consumer actually read a message with MessageRank 200. (Please note that in order to exactly run the required scenario, the reproducer might automatically execute several times in a loop, self-tuning at each iteration the receive time-out value used at step 1.)

    Description

      We have been investigating a faulty messaging ordering behaviour for a couple of days, and we finally determined the scenario in which it happens.
      It is linked the way A-MQ handles the roll-back of transactions and the redelivery of messages.

      Scenario:
      1. Consume messages with a prefetch value >0
      2. Upon not receiving messages any more in the consumer (prefetch buffer empty, although there are still messages in the queue), the consumer is closed and a new consumer is created with prefetch=0
      3. This new "zero-prefetch" consumer consumes some messages
      4. This consumer's session rolls back (on purpose)
      -> Conceptually, the messages consumed in 3. go back at the head of the queue since the "maximumRedeliveries" is set to e.g. 10
      5. The consumer, the session and the connection are properly closed
      6. A new connection, session and consumer are created.
      -> the first messages this new consumer consumes are not the messages which consumption was rolled-back at step 4. (although they should be).

      In essence, this means that there are situations where a single consumer may consume messages in an order which is different from the order created at the time of enqueue.
      (Please note that no message priority is used, and that the initialRedeliveryDelay and redeliveryDelay values are set to zero).

      Attachments

        Activity

          People

            gtully@redhat.com Gary Tully
            rhn-support-ger-jan Gerhardus Johannes Petrus Maria te Dorsthorst
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: