Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-3089

Failback fails in colocated HA toplogy with shared store and netty connectors

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Blocker
    • 7.0.0.ER7
    • 7.0.0.ER4
    • ActiveMQ
    • Regression
    • Hide

      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git

      cd eap-tests-hornetq/scripts/

      git checkout refactoring_modules

      groovy -DEAP_VERSION=7.0.0.ER4 PrepareServers7.groovy

      export WORKSPACE=$PWD

      export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap

      export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap

      export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap

      export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap

      mkdir journal_A
      mkdir journal_B
      export JOURNAL_DIRECTORY_A=$WORKSPACE/journal_A
      export JOURNAL_DIRECTORY_B=$WORKSPACE/journal_B

      cd ../jboss-hornetq-testsuite/

      mvn clean test -Dtest=ColocatedClusterFailoverTestCase#testFailbackClientAckQueueNIO -DfailIfNoTests=false -Deap=7x | tee log

      Show
      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ git checkout refactoring_modules groovy -DEAP_VERSION=7.0.0.ER4 PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap mkdir journal_A mkdir journal_B export JOURNAL_DIRECTORY_A=$WORKSPACE/journal_A export JOURNAL_DIRECTORY_B=$WORKSPACE/journal_B cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=ColocatedClusterFailoverTestCase#testFailbackClientAckQueueNIO -DfailIfNoTests=false -Deap=7x | tee log

    Description

      We have manually (like EAP6) configured colocated topology with shared journal.
      Client is connected to server1 and sends messages to queue. While sending, server1 is killed when "processRoute" method is invoked in PostOfficeImpl on server1. Client successfully executes failover sequence to backup. Then server1 is started again.

      Client is stuck (probably on send() ) method.
      Server1: Live obtains live lock sucesfully, backup obtains backup log sucessfully and is announced, cluster is connected. Log starts spamming messages:
      13:53:33,730 WARN [org.apache.activemq.artemis.core.client] (Thread-0 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$2@70f35326-1682410692)) AMQ212006: Waiting 2,000 milliseconds before next retry. RetryInterval=2,000 and multiplier=1

      Server2: Live and backup are announced and obtain their locks. And in some cases starts spaming log with: 13:47:40,718 WARN [org.apache.activemq.artemis.core.client] (Thread-20 (ActiveMQ-client-global-threads-785057483)) AMQ212006: Waiting 2,000 milliseconds before next retry. RetryInterval=2,000 and multiplier=1

      Here are thread dumps from client: [dump-client.txt] and from server1: [dump-server.txt]

      In attachment are configs of both servers.

      Attachments

        1. dump-client.txt
          33 kB
        2. dump-server.txt
          222 kB
        3. logs_3089.zip
          4.20 MB
        4. standalone-full-ha.xml
          31 kB
        5. standalone-full-ha-2.xml
          31 kB
        6. traces.zip
          2.36 MB

        Issue Links

          is blocked by

          Component Upgrade - A task tracking an update to a bundled component or revision of component upstream of the project. JBEAP-3577 Upgrade Artemis to 1.1.0.wildfly-014

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed
          relates to

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-3918 (7.0.z) AMQ224000: Failure in initialisation: java.lang.NullPointerException

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              mtaylor1@redhat.com Martyn Taylor (Inactive)
              okalman@redhat.com Ondřej Kalman (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: