Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-11059

Clients do not failover to backup even it if backup started and keep retrying connection

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Critical
    • None
    • 14.0.0.Final
    • JMS
    • None
    • Hide

      Issue is intermittent - not 100% reproducer:

      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
cd eap-tests-hornetq/scripts/
git checkout eap7.2.x-artemis2.x-ha-jdbc
groovy -DEAP_ZIP_URL=https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messaging-testing-prepare/362/artifact/jboss-eap.zip PrepareServers7.groovy
export WORKSPACE=$PWD
export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap

cd ../jboss-hornetq-testsuite/

mvn clean test -Dtest=NettyColocatedClusterFailoverTestCase#testFailbackClientAckQueueNIO -Deap7.clients.version=7.1531981138-SNAPSHOT | tee log
      Show
      Issue is intermittent - not 100% reproducer : git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ git checkout eap7.2.x-artemis2.x-ha-jdbc groovy -DEAP_ZIP_URL=https: //eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-messaging-testing-prepare/362/artifact/jboss-eap.zip PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=NettyColocatedClusterFailoverTestCase#testFailbackClientAckQueueNIO -Deap7.clients.version=7.1531981138-SNAPSHOT | tee log

    Description

      Test Scenario:

      • Start 2 servers in colocated topology with shared store
      • Start procuder and consumer sending/receiving messages to/from queue on 1st server(to live)
      • Kill 1st server and wait for backup on 2nd server to start
      • Clients failover to backup to 2nd server and wait for them to finish
      • Check that number of sent/received messages is the same

      Result:
      Sometimes happens that clients do not failover to backup to 2nd server and retry connection. I can see that backup on 2nd server started and created cluster with colocated live server but clients did not connect to it.

      Clients are retrying connection:

      07:32:16,676 Thread-36 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:794] Trying reconnection attempt 163/-1
07:32:16,676 Thread-39 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:446] Connector NettyConnector [host=rhel7-large-42748, port=5445, httpEnabled=fa
lse, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using native epoll
07:32:16,676 Thread-36 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1080] Trying to connect with connectorFactory = org.apache.activemq.artemis.co
re.remoting.impl.netty.NettyConnectorFactory@6568a014, connectorConfig=TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&useNio=true
&host=rhel7-large-42748&useNioGlobalWorkerPool=true&blockOnNonDurableSend=false&retryIntervalMultiplier=1-0&maxRetryInterval=2000&producerMaxRate=-1&blockOnDurableSend=true&connectionTTL=60000&compressLargeMessa
ge=false&reconnectAttempts=-1&ignoreJTA=false&cacheLargeMessagesClient=false&scheduledThreadPoolMaxSize=5&useGlobalPools=true&callFailoverTimeout=60000&initialConnectAttempts=1&clientFailureCheckPeriod=30000&blo
ckOnAcknowledge=true&consumerWindowSize=1048576&minLargeMessageSize=102400&finalizeChecks=false&autoGroup=false&threadPoolMaxSize=30&confirmationWindowSize=-1&transactionBatchSize=1048576&callTimeout=30000&preAc
knowledge=false&enable1xPrefixes=true&cacheDestinations=false&connectionLoadBalancingPolicyClassName=org-apache-activemq-artemis-api-core-client-loadbalance-RoundRobinConnectionLoadBalancingPolicy&dupsOKBatchSiz
e=1048576&incomingInterceptorList=&initialMessagePacketSize=1500&consumerMaxRate=-1&enableSharedClientID=true&HA=true&retryInterval=2000&factoryType=0&failoverOnInitialConnection=false&outgoingInterceptorList=&p
roducerWindowSize=65536
07:32:16,676 Thread-39 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:600] AMQ211002: Started EPOLL Netty Connector version 4.1.16.Final to rhel7-larg
e-42748:5445
07:32:16,676 Thread-36 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector:446] Connector NettyConnector [host=rhel7-large-42748, port=5445, httpEnabled=fa
lse, httpUpgradeEnabled=false, useServlet=false, servletPath=/messaging/ActiveMQServlet, sslEnabled=false, useNio=true] using native epoll
07:32:16,676 Thread-38 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:794] Trying reconnection attempt 163/-1
07:32:16,676 Thread-38 (ActiveMQ-client-global-threads) DEBUG [org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl:1080] Trying to connect with connectorFactory = org.apache.activemq.artemis.co
re.remoting.impl.netty.NettyConnectorFactory@43d7f9fb, connectorConfig=TransportConfiguration(name=null, factory=org-apache-activemq-artemis-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&useNio=true
&host=rhel7-large-42748&useNioGlobalWorkerPool=true&blockOnNonDurableSend=false&retryIntervalMultiplier=1-0&maxRetryInterval=2000&producerMaxRate=-1&blockOnDurableSend=true&connectionTTL=60000&compressLargeMessa
ge=false&reconnectAttempts=-1&ignoreJTA=false&cacheLargeMessagesClient=false&scheduledThreadPoolMaxSize=5&useGlobalPools=true&callFailoverTimeout=60000&initialConnectAttempts=1&clientFailureCheckPeriod=30000&blo
ckOnAcknowledge=true&consumerWindowSize=1048576&minLargeMessageSize=102400&finalizeChecks=false&autoGroup=false&threadPoolMaxSize=30&confirmationWindowSize=-1&transactionBatchSize=1048576&callTimeout=30000&preAc
knowledge=false&enable1xPrefixes=true&cacheDestinations=false&connectionLoadBalancingPolicyClassName=org-apache-activemq-artemis-api-core-client-loadbalance-RoundRobinConnectionLoadBalancingPolicy&dupsOKBatchSiz
e=1048576&incomingInterceptorList=&initialMessagePacketSize=1500&consumerMaxRate=-1&enableSharedClientID=true&HA=true&retryInterval=2000&factoryType=0&failoverOnInitialConnection=false&outgoingInterceptorList=&p
roducerWindowSize=65536

      Attachments

        Issue Links

          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. ENTMQBR-1818 [Artemis 2.x upgrade] Artemis RA connections disconnected after failover

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed
          relates to

          Bug - A problem that impairs or prevents the functions of the product. WFWIP-65 [Artemis 2.x upgrade] Artemis RA connections disconnected after failover

          • Blocker - To be worked above all other priorities. This term is chosen when the severity of the issue is very high, has no workaround, or the effort for the change is comparatively low. Issues that may be very publicly visible and could generate significant media attention may also drive a higher priority.
          • Closed

          Activity

            People

              mtaylor1@redhat.com Martyn Taylor (Inactive)
              mnovak1@redhat.com Miroslav Novak
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: