Uploaded image for project: 'JBoss Transaction Manager'
  1. JBoss Transaction Manager
  2. JBTM-1136

XTS Crash Recovery fail: Could not start container

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Rejected
    • Affects Version/s: 4.16.3, 4.17.0.M1/5.0.0.M1
    • Fix Version/s: 4.17.0, 5.0.0.M2
    • Component/s: XTS
    • Labels:
      None

      Description

      http://172.17.131.2/job/jbossts-branch416-java6/193
      http://172.17.131.2/job/jbossts-branch416-java6/194
      http://172.17.131.2/job/jbossts-branch416-java7/137

      http://172.17.131.2/job/narayana-xts-recovery-java6/90
      http://172.17.131.2/job/narayana-xts-recovery-java6/94

      It looks like booting jboss-as with the following error and xtstests.war can not be deployed. the arquillian thinks the container does not start.

      02:51:57,012 ERROR [org.jboss.as.controller.management-operation] (ServerService Thread Pool -- 50) JBAS014612: Operation ("add") failed - address: ([("subsystem" => "osgi")]): org.jboss.msc.service.DuplicateServiceException: Service jbosgi.integration.PersistentBundlesHandler is already registered
      	at org.jboss.msc.service.ServiceRegistrationImpl.setInstance(ServiceRegistrationImpl.java:154) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
      	at org.jboss.msc.service.ServiceControllerImpl.startInstallation(ServiceControllerImpl.java:227) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
      	at org.jboss.msc.service.ServiceContainerImpl.install(ServiceContainerImpl.java:560) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
      	at org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:201) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
      	at org.jboss.msc.service.ServiceControllerImpl$ChildServiceTarget.install(ServiceControllerImpl.java:2228) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
      	at org.jboss.msc.service.ServiceBuilderImpl.install(ServiceBuilderImpl.java:307) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
      	at org.jboss.as.controller.OperationContextImpl$ContextServiceBuilder.install(OperationContextImpl.java:955) [jboss-as-controller-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]
      	at org.jboss.as.osgi.service.PersistentBundlesIntegration.addService(PersistentBundlesIntegration.java:76) [jboss-as-osgi-service-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]
      	at org.jboss.as.osgi.parser.OSGiSubsystemAdd$2.execute(OSGiSubsystemAdd.java:130) [jboss-as-osgi-service-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]
      	at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:385) [jboss-as-controller-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]
      	at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:272) [jboss-as-controller-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]
      	at org.jboss.as.controller.AbstractOperationContext.completeStep(AbstractOperationContext.java:200) [jboss-as-controller-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]
      	at org.jboss.as.osgi.parser.OSGiSubsystemAdd$1.execute(OSGiSubsystemAdd.java:103) [jboss-as-osgi-service-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]
      	at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:385) [jboss-as-controller-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]
      	at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:272) [jboss-as-controller-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]
      	at org.jboss.as.controller.AbstractOperationContext.completeStep(AbstractOperationContext.java:200) [jboss-as-controller-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]
      	at org.jboss.as.controller.ParallelBootOperationStepHandler$ParallelBootTask.run(ParallelBootOperationStepHandler.java:311) [jboss-as-controller-7.1.2.Final-SNAPSHOT.jar:7.1.2.Final-SNAPSHOT]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_03]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_03]
      	at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_03]
      	at org.jboss.threads.JBossThread.run(JBossThread.java:122)
      

        Gliffy Diagrams

          Issue Links

            Activity

            Hide
            paul.robinson Paul Robinson added a comment -

            It looks like this error is preventing the XTS test application from being accessible. Either, the error appears during deployment of the XTS test war, or it is a failure in a dependent service that prevents the XTS test war from being available.

            I think we need to reproduce the scenario in which this occurs. It looks like we could take a build of the latest AS7 and then try booting it many times. We should deploy the XTS test war in the same way that the test is doing this.

            Once we can reproduce the problem, I think we should try an older build of AS7 to prove that it is a new issue and then, of course, we'll need to JIRA something with the AS7 project.

            Show
            paul.robinson Paul Robinson added a comment - It looks like this error is preventing the XTS test application from being accessible. Either, the error appears during deployment of the XTS test war, or it is a failure in a dependent service that prevents the XTS test war from being available. I think we need to reproduce the scenario in which this occurs. It looks like we could take a build of the latest AS7 and then try booting it many times. We should deploy the XTS test war in the same way that the test is doing this. Once we can reproduce the problem, I think we should try an older build of AS7 to prove that it is a new issue and then, of course, we'll need to JIRA something with the AS7 project.
            Hide
            paul.robinson Paul Robinson added a comment -

            We should also check what version of Arquillian is now in AS7. In my experience, Arquillian is very sensitive to version mismatch between client and server.

            Show
            paul.robinson Paul Robinson added a comment - We should also check what version of Arquillian is now in AS7. In my experience, Arquillian is very sensitive to version mismatch between client and server.
            Hide
            paul.robinson Paul Robinson added a comment -

            Looks like the update to use Arquillian 1.0.0.Final didn't help.

            Build 90 passed (this was when the upgrade was done)

            http://172.17.131.2/view/XTS%20recovery/job/jbossts-branch416-xts-recovery-java6/90/

            But, build 93 failed, with this error:

            http://172.17.131.2/view/XTS%20recovery/job/jbossts-branch416-xts-recovery-java6/93/

            Show
            paul.robinson Paul Robinson added a comment - Looks like the update to use Arquillian 1.0.0.Final didn't help. Build 90 passed (this was when the upgrade was done) http://172.17.131.2/view/XTS%20recovery/job/jbossts-branch416-xts-recovery-java6/90/ But, build 93 failed, with this error: http://172.17.131.2/view/XTS%20recovery/job/jbossts-branch416-xts-recovery-java6/93/
            Hide
            paul.robinson Paul Robinson added a comment -

            I've looked at the code and this is what it does:

            1) Arquillian starts the container
            2) Arquillian deploys the xtstest.war artifact, which is a shrinkwrapped physical war defined in the @Deployment method.
            3) The server is crashed by Byteman
            4) The test asks Arquillien to start the server again
            5) When the server starts, it is still configured to deploy the xtstest.war which it attempts to do. This occasionally fails, which is this bug. You will notice in the log that there is the absence of 'Deployed "xtstest.war"' in the cases where this bug occurs.

            I suspect this is not a "regular" use case, so it could be caused by a bug in Arquillian. We should try and re-produce this as at the moment we don't have a lot of information to hand over.

            Show
            paul.robinson Paul Robinson added a comment - I've looked at the code and this is what it does: 1) Arquillian starts the container 2) Arquillian deploys the xtstest.war artifact, which is a shrinkwrapped physical war defined in the @Deployment method. 3) The server is crashed by Byteman 4) The test asks Arquillien to start the server again 5) When the server starts, it is still configured to deploy the xtstest.war which it attempts to do. This occasionally fails, which is this bug. You will notice in the log that there is the absence of 'Deployed "xtstest.war"' in the cases where this bug occurs. I suspect this is not a "regular" use case, so it could be caused by a bug in Arquillian. We should try and re-produce this as at the moment we don't have a lot of information to hand over.
            Show
            paul.robinson Paul Robinson added a comment - It looks like Arquillian times out when deploying the xtstest.war: http://172.17.131.2/view/Narayana+BlackTie/job/narayana-java6-idlj/3/artifact/XTS/sar/crash-recovery-tests/target/surefire-reports/com.arjuna.qa.junit.TestATCrashDuringCommit.txt
            Hide
            tomjenkinson Tom Jenkinson added a comment -

            Moving out to .Final as there is a fix for AS7 that is required. Note this is not an XTS issue so when the underlying issue is resolved we can mark this as rejected anyway as it is an AS7 bug in unrelated code.

            Show
            tomjenkinson Tom Jenkinson added a comment - Moving out to .Final as there is a fix for AS7 that is required. Note this is not an XTS issue so when the underlying issue is resolved we can mark this as rejected anyway as it is an AS7 bug in unrelated code.
            Hide
            zhfeng Amos Feng added a comment -

            the fix has merged in the upstream jboss-as and we verified both on narayana-xts-recovery-java6 and jbossts-branch416-xts-recovery-java6

            Show
            zhfeng Amos Feng added a comment - the fix has merged in the upstream jboss-as and we verified both on narayana-xts-recovery-java6 and jbossts-branch416-xts-recovery-java6

              People

              • Assignee:
                zhfeng Amos Feng
                Reporter:
                zhfeng Amos Feng
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development