Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-10704

AMQ142032: Error reading journal file arises in scenarios when server is killed during journal compacting

    XMLWordPrintable

Details

    • Workaround Exists
    • Hide

      Remove the truncated control file.

      Show
      Remove the truncated control file.
    • Hide 
      git clone git://git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git
cd eap-tests-hornetq/scripts/
git checkout 1abcd9b69f11aeec8007786f43c4c207a70eb9b6
groovy -DEAP_VERSION=7.1.0.ER1 PrepareServers7.groovy
export WORKSPACE=$PWD
export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap
export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap
export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap
export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap

cd ../jboss-hornetq-testsuite/

mvn clean test -Dtest=DedicatedFailoverTestCase#testKillAtWriteControlFile -DfailIfNoTests=false -Deap=7x | tee log
      Show
      git clone git: //git.app.eng.bos.redhat.com/jbossqe/eap-tests-hornetq.git cd eap-tests-hornetq/scripts/ git checkout 1abcd9b69f11aeec8007786f43c4c207a70eb9b6 groovy -DEAP_VERSION=7.1.0.ER1 PrepareServers7.groovy export WORKSPACE=$PWD export JBOSS_HOME_1=$WORKSPACE/server1/jboss-eap export JBOSS_HOME_2=$WORKSPACE/server2/jboss-eap export JBOSS_HOME_3=$WORKSPACE/server3/jboss-eap export JBOSS_HOME_4=$WORKSPACE/server4/jboss-eap cd ../jboss-hornetq-testsuite/ mvn clean test -Dtest=DedicatedFailoverTestCase#testKillAtWriteControlFile -DfailIfNoTests= false -Deap=7x | tee log
    • AMQ Sprint 3

    Description

      In test scenarios where EAP is killed during journal compacting it may happen that backup fails to load the journal.

      Customer impact: During the failover backup server may fail to start because of this issue. It leads to unavailability of the service and breaks meaning of HA.

      You can see the same exception even if you copy the corrupted journal in attachment into the fresh EAP installation and run the server with the standalone-full-ha profile.

      The stack trace of the exception [1] shows that the error arises when Backup reads JournalCompactor control file. In this case the Live only created the file but it didn't write any record to it. The Backup doesn't expect that the file could have 0 length.

      [1]

      15:32:43,529 WARN  [org.apache.activemq.artemis.journal] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=4b1f5a18-2c49-11e7-9fdb-001b217d6d93) AMQ142032: Error reading journal file: java.lang.Il
legalArgumentException
        at java.nio.Buffer.position(Buffer.java:244) [rt.jar:1.8.0_121]
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:418) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.journal.impl.JournalCompactor.readControlFile(JournalCompactor.java:79) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.checkControlFile(JournalImpl.java:2720) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1666) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1326) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1310) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:837) [artemis-server-1.5.4.003-redhat-1.jar:1.5.4.003-redh
at-1]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2213) [artemis-server-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2067) [artemis-server-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.server.impl.SharedStoreBackupActivation.run(SharedStoreBackupActivation.java:90) [artemis-server-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2491) [artemis-server-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]

15:32:43,530 ERROR [org.apache.activemq.artemis.core.server] (AMQ119000: Activation for server ActiveMQServerImpl::serverUUID=4b1f5a18-2c49-11e7-9fdb-001b217d6d93) AMQ224000: Failure in initialisation: java.lang
.Exception
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:702) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.journal.impl.JournalCompactor.readControlFile(JournalCompactor.java:79) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.checkControlFile(JournalImpl.java:2720) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1666) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1326) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1310) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:837) [artemis-server-1.5.4.003-redhat-1.jar:1.5.4.003-redh
at-1]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2213) [artemis-server-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:2067) [artemis-server-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.server.impl.SharedStoreBackupActivation.run(SharedStoreBackupActivation.java:90) [artemis-server-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$ActivationThread.run(ActiveMQServerImpl.java:2491) [artemis-server-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
Caused by: java.lang.IllegalArgumentException
        at java.nio.Buffer.position(Buffer.java:244) [rt.jar:1.8.0_121]
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:418) [artemis-journal-1.5.4.003-redhat-1.jar:1.5.4.003-redhat-1]
        ... 10 more

      Attachments

        1. journal.zip
          292 kB
        2. node-1-logs.zip
          939 kB
        3. node-2-logs.zip
          186 kB
        4. test.log
          3.94 MB

        Issue Links

          is blocked by

          Component Upgrade - A task tracking an update to a bundled component or revision of component upstream of the project. JBEAP-12044 Upgrade Artemis 1.5.5.jbossorg-006

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Verified
          is incorporated by

          Component Upgrade - A task tracking an update to a bundled component or revision of component upstream of the project. JBEAP-12044 Upgrade Artemis 1.5.5.jbossorg-006

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Verified
          cloned to

          ARTEMIS-1275 Loading...

          Activity

            People

              csuconic@redhat.com Clebert Suconic
              eduda_jira Erich Duda (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: