Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-882

Standby slave does not announce replication to master when primary slave is down

    XMLWordPrintable

Details

    • Documentation (Ref Guide, User Guide, etc.), Release Notes
    • Hide
      This issue occurred in past releases when multiple backup brokers, also referred to as slaves, served a single live (master) broker. In this scenario, if a primary backup broker failed, the secondary backup tried to replicate. But that operation failed, the secondary backup could not take over for the primary backup, and as a result, high-availability was lost. This issue is now resolved.
      Show
      This issue occurred in past releases when multiple backup brokers, also referred to as slaves, served a single live (master) broker. In this scenario, if a primary backup broker failed, the secondary backup tried to replicate. But that operation failed, the secondary backup could not take over for the primary backup, and as a result, high-availability was lost. This issue is now resolved.
    • Documented as Resolved Issue
    • Hide

      Configurations attached.

      1. Start master node
      2. Start slave node and wait for it to become replication partner for master
      3. Start standby node
      4. Produce messages to master node
      5. Kill slave node
      (standby does not announce as backup and remains in Backup Activation Loop
      6. Kill master node
      (standby still waiting to become backup)

      Show
      Configurations attached. 1. Start master node 2. Start slave node and wait for it to become replication partner for master 3. Start standby node 4. Produce messages to master node 5. Kill slave node (standby does not announce as backup and remains in Backup Activation Loop 6. Kill master node (standby still waiting to become backup)
    • AMQ Broker 1836, AMQ Broker 1839

    Description

      When testing failover in a scenario with 1 master and 2 slaves, the example scenario in which the master is killed first worrks correctly - the primary backup becomes the master and the secondary backup becomes the replication node.

      If, however, the primary backup is killed first, the secondary backup remains stopped and does not announce as the replication slave. Instead it continues to log:

      13:31:44,373 WARN  [org.apache.activemq.artemis.core.server] AMQ222040: Server is stopped

      When the master is brought down, the secondary slave remains stopped.

      Looking at the thread dumps of the secondary backup for this scenario, (taken when the primary is killed), it appears the secondary is stuck looping in NamedLiveNodeLocatorForReplication::locateNode(...).

      "AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=null" #18 prio=5 os_prio=0 tid=0x00007f1920803800 nid=0x642b waiting on condition [0x00007f19028e8000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000c04b7170> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at org.apache.activemq.artemis.core.server.impl.NamedLiveNodeLocatorForReplication.locateNode(NamedLiveNodeLocatorForReplication.java:67)
        at org.apache.activemq.artemis.core.server.impl.NamedLiveNodeLocatorForReplication.locateNode(NamedLiveNodeLocatorForReplication.java:54)
        at org.apache.activemq.artemis.core.server.impl.SharedNothingBackupActivation.run(SharedNothingBackupActivation.java:195)
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2793)

   Locked ownable synchronizers:
        - None

      If multiple slaves are configured for a master, nth slave should become the active slave if the current slave(s) are offline.

      This is https://issues.apache.org/jira/browse/ARTEMIS-2075 upstream

      Attachments

        Issue Links

          is cloned by

          Task - Represents a small unit of work that is not end-user facing. ENTMQBR-1954 Create test for Standby slave does not announce replication to master when primary slave is down

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed
          is duplicated by

          Bug - A problem that impairs or prevents the functions of the product. ENTMQBR-1021 [HA, MS1S2] When backup slave1 is killed, slave2 can't take role of backup, leaving HA on master only

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed
          is related to

          ARTEMIS-1285 Loading...

          Activity

            People

              rh-ee-ataylor Andy Taylor
              rhn-support-dhawkins Duane Hawkins
              Roman Vais Roman Vais
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 2 days
                  2d
                  Remaining:
                  Remaining Estimate - 2 days
                  2d
                  Logged:
                  Time Spent - Not Specified
                  Not Specified