Details

    • Target Release:
    • Sprint:
      AMQ Broker 1833
    • Steps to Reproduce:
      Hide
      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
      cd eap-tests-hornetq/scripts/
      groovy -DEAP_ZIP_URL=https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messaging-testing-prepare/97/artifact/jboss-eap.zip PrepareServers7.groovy
      export WORKSPACE=$PWD
      export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
      export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
      export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
      export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap
      
      cd ../jboss-hornetq-testsuite/
      
      mvn clean test -Dtest=ColocatedClusterFailoverTestCase#testFailbackWithMdbsShutdownDeployUsingCLI -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1531731315-SNAPSHOT -DfailIfNoTests=false -Deap=7x | tee log
      
      Show
      git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ groovy -DEAP_ZIP_URL=https: //eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messaging-testing-prepare/97/artifact/jboss-eap.zip PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=ColocatedClusterFailoverTestCase#testFailbackWithMdbsShutdownDeployUsingCLI -Deap7.org.jboss.qa.hornetq.apps.clients.version=7.1531731315-SNAPSHOT -DfailIfNoTests= false -Deap=7x | tee log

      Description

      PR: https://github.com/wildfly/wildfly/pull/11355

      Scenario:

      • There are two Wildfly servers each containing two Artemis brokers
      • Artemis brokers creates two live-backup pairs. Each Wildfly instance contains one live and one backup
      • Second server has MDB which resends messages from InQueue to OutQueue
      • The second server is stopped and restarted

      After the second server is restarted, everything works correctly for a while. Failback performs successfully, MDB resends messages. However after some time the server starts to print many errors [1]. It looks like some messages where corrupted during failover/failback and cannot be delivered.

      The reproducer always hits the issue. The issue is not relevant to particular database, I hit it with Oracle12c and DB2 11.1. I also tried to reduce amount of messages and their size, but I still hit the issue. So it is not related to paging or large messages.

      [1]

      15:20:50,010 ERROR [org.apache.activemq.artemis.ra] (Thread-7 (ActiveMQ-client-global-threads)) AMQ154004: Failed to deliver message: java.lang.IndexOutOfBoundsException
              at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readSimpleStringInternal(ChannelBufferWrapper.java:93) [artemis-commons-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
              at org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readNullableSimpleString(ChannelBufferWrapper.java:73) [artemis-commons-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
              at org.apache.activemq.artemis.reader.TextMessageUtil.readBodyText(TextMessageUtil.java:37) [artemis-core-client-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
              at org.apache.activemq.artemis.jms.client.ActiveMQTextMessage.doBeforeReceive(ActiveMQTextMessage.java:112) [artemis-jms-client-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
              at org.apache.activemq.artemis.ra.inflow.ActiveMQMessageHandler.onMessage(ActiveMQMessageHandler.java:295) [artemis-ra-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
              at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1001) [artemis-core-client-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
              at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.access$400(ClientConsumerImpl.java:49) [artemis-core-client-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
              at org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1124) [artemis-core-client-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
              at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:122) [artemis-commons-1.5.5.jbossorg-012.jar:1.5.5.jbossorg-012]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_171]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_171]
              at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_171]
      

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  martyn-taylor Martyn Taylor
                  Reporter:
                  mnovak Miroslav Novak
                  Tester:
                  Michal Toth
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: