Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-14032

[GSS](7.2.z) ARTEMIS-2069 - Backup doesn't activate after shared store is reconnected

    XMLWordPrintable

Details

    Description

      Scenario

      1. Start live backup server pair in dedicated topology with shared store HA, with journal located on NFS
      2. NFS mounted on backup server fails
      3. Reconnect NFS on backup server
      4. Try to shut down live EAP server
      5. Backup doesn't activate

      What happens
      Backup is waiting for live to fail by checking its file lock. In case the connection to shared storage fails, backup logs following error.

      05:50:57,896 ERROR [org.apache.activemq.artemis.core.server] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=836c9b1e-f067-11e7-8763-001b21862475) AMQ224000: Failure in initialisation: java.io.IOException: Input/output error
	at sun.nio.ch.FileDispatcherImpl.lock0(Native Method) [rt.jar:1.8.0_151]
	at sun.nio.ch.FileDispatcherImpl.lock(FileDispatcherImpl.java:90) [rt.jar:1.8.0_151]
	at sun.nio.ch.FileChannelImpl.tryLock(FileChannelImpl.java:1115) [rt.jar:1.8.0_151]
	at org.apache.activemq.artemis.core.server.impl.FileLockNodeManager.tryLock(FileLockNodeManager.java:299) [artemis-server-1.5.5.008-redhat-1.jar:1.5.5.008-redhat-1]
	at org.apache.activemq.artemis.core.server.impl.FileLockNodeManager.lock(FileLockNodeManager.java:316) [artemis-server-1.5.5.008-redhat-1.jar:1.5.5.008-redhat-1]
	at org.apache.activemq.artemis.core.server.impl.FileLockNodeManager.awaitLiveNode(FileLockNodeManager.java:127) [artemis-server-1.5.5.008-redhat-1.jar:1.5.5.008-redhat-1]
	at org.apache.activemq.artemis.core.server.impl.SharedStoreBackupActivation.run(SharedStoreBackupActivation.java:77) [artemis-server-1.5.5.008-redhat-1.jar:1.5.5.008-redhat-1]
	at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2496) [artemis-server-1.5.5.008-redhat-1.jar:1.5.5.008-redhat-1]

      Exception is caught in SharedStoreBackupActivation.run, and causes termination of backup activation process.

      In case the NFS is reconnected later, backup server doesn't continue in activation process and it doesn't wait for live to fail. In case the live fails, backup doesn't activate, even though it has a connection to shared storage.

      Backup should retry checking live lock even in case the storage is unavailable. It should log warning/error messages that storage is unavailable, but it should not terminate the activation process. This would allow backup to continue its duties when the storage is reconnected.

      Attachments

        Issue Links

          clones

          Bug - A problem that impairs or prevents the functions of the product. WFLY-10968 Backup doesn't activate after shared store is reconnected

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed
          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. ENTMQBR-2147 (7.2.z) Backup doesn't activate after shared store is reconnected

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-15374 [GSS](7.1.z) Backup doesn't activate after shared store is reconnected

          • Critical - To be worked on immediately following BLOCKER issues.
          • Closed
          is incorporated by

          Component Upgrade - A task tracking an update to a bundled component or revision of component upstream of the project. JBEAP-17062 [GSS](7.2.z) Upgrade Artemis from 2.7.0.redhat-00057 to 2.9.0.redhat-00005

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          links to

          Web Link ARTEMIS-2069

          Activity

            People

              thofman Tomas Hofman
              mstyk_jira Martin Styk (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: