Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-11706

Deadlock in cleanup following DatabaseCertLoginModuleTestCase

    Details

      Description

      A test hung due to a server-side deadlock.

      https://ci.wildfly.org/viewLog.html?buildId=139407&buildTypeId=WFPR&tab=buildLog&branch_WF_PullRequest=pull%2F12062

      Build log output stopped at this point:

      [18:27:07][Step 2/3] [INFO] Running org.jboss.as.test.manualmode.security.OutboundLdapConnectionTestCase
      [18:27:42][Step 2/3] [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 34.744 s - in org.jboss.as.test.manualmode.security.OutboundLdapConnectionTestCase
      [18:27:43][Step 2/3] [INFO] Running org.jboss.as.test.manualmode.web.ssl.DatabaseCertLoginModuleTestCase

      The 'main' thread of the test shows the test was shutting down the server:

      "main" #1 prio=5 os_prio=0 tid=0xf6507c00 nid=0x205c in Object.wait() [0xf667d000]
         java.lang.Thread.State: WAITING (on object monitor)
      	at java.lang.Object.wait(Native Method)
      	- waiting on <0xc807a168> (a java.lang.UNIXProcess)
      	at java.lang.Object.wait(Object.java:502)
      	at java.lang.UNIXProcess.waitFor(UNIXProcess.java:395)
      	- locked <0xc807a168> (a java.lang.UNIXProcess)
      	at org.wildfly.core.launcher.ProcessHelper.destroyProcess(ProcessHelper.java:60)
      	at org.jboss.as.arquillian.container.managed.ManagedDeployableContainer.stopInternal(ManagedDeployableContainer.java:294)
      	at org.jboss.as.arquillian.container.CommonDeployableContainer.stop(CommonDeployableContainer.java:135)
      	at org.jboss.arquillian.container.impl.ContainerImpl.stop(ContainerImpl.java:217)
      	at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController$9.perform(ContainerLifecycleController.java:178)
      	at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController$9.perform(ContainerLifecycleController.java:172)
      	at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController.forContainer(ContainerLifecycleController.java:255)
      	at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController.stopContainer(ContainerLifecycleController.java:171)
      

      The server-side thread dump shows the hang:

      Found one Java-level deadlock:
      =============================
      "Periodic Recovery":
        waiting to lock monitor 0xbcc6dbc0 (object 0xc8f80918, a java.util.concurrent.atomic.AtomicInteger),
        which is held by "ServerService Thread Pool -- 8"
      "ServerService Thread Pool -- 8":
        waiting to lock monitor 0xc1a6ffc8 (object 0xc8f64470, a com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule),
        which is held by "Periodic Recovery"
      
      Java stack information for the threads listed above:
      ===================================================
      "Periodic Recovery":
      	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.setScanState(XARecoveryModule.java:1088)
      	- waiting to lock <0xc8f80918> (a java.util.concurrent.atomic.AtomicInteger)
      	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:240)
      	- locked <0xc8f64470> (a com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule)
      	at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:816)
      	at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:382)
      "ServerService Thread Pool -- 8":
      	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.removeXAResourceRecoveryHelper(XARecoveryModule.java:119)
      	- waiting to lock <0xc8f64470> (a com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule)
      	- locked <0xc8f80918> (a java.util.concurrent.atomic.AtomicInteger)
      	at com.arjuna.ats.jbossatx.jta.RecoveryManagerService.removeXAResourceRecovery(RecoveryManagerService.java:129)
      	at org.jboss.jca.core.tx.jbossts.XAResourceRecoveryRegistryImpl.removeXAResourceRecovery(XAResourceRecoveryRegistryImpl.java:63)
      	at org.jboss.as.connector.subsystems.datasources.XaDataSourceService.stopService(XaDataSourceService.java:66)
      	- locked <0xc8f95a38> (a org.jboss.as.connector.subsystems.datasources.XaDataSourceService)
      	at org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService$1.run(AbstractDataSourceService.java:188)
      	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
      	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
      	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
      	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
      	at java.lang.Thread.run(Thread.java:748)
      	at org.jboss.threads.JBossThread.run(JBossThread.java:485)
      
      Found 1 deadlock.
      

      I'll attach the full thread dump.

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  ochaloup Ondrej Chaloupka
                  Reporter:
                  brian.stansberry Brian Stansberry
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: