Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-9531

Deadlock in model controller encountered in basic test suite

    Details

    • Estimated Difficulty:
      High

      Description

      A Java-level deadlock was encountered while running org.jboss.as.test.integration.jca.workmanager.LongRunningThreadsCheckTestCase. The thread dump is attached.

      This occurred during testing of the new thread pool (WFLY-5332 and related), but it seems unlikely to be caused by this change, unless the new pool's scheduling behavior differed enough from the default that it exposed an already-existing race condition. The deadlock was hit just once in more than 10 runs.

      This is the deadlock:

      Found one Java-level deadlock:
      =============================
      "DeploymentScanner-threads - 2":
        waiting for ownable synchronizer 0x000000075357cef0, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
        which is held by "MSC service thread 1-1"
      "MSC service thread 1-1":
        waiting for ownable synchronizer 0x000000071dff9b08, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
        which is held by "MSC service thread 1-2"
      "MSC service thread 1-2":
        waiting for ownable synchronizer 0x000000075357cef0, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
        which is held by "MSC service thread 1-1"
      

      Here is the specific deadlock info (also in the attachment), showing that it has something to do with the model controller:

      Java stack information for the threads listed above:
      ===================================================
      "DeploymentScanner-threads - 2":
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x000000075357cef0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
      	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
      	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
      	at org.jboss.as.controller.registry.ConcreteResourceRegistration.getOperationEntry(ConcreteResourceRegistration.java:304)
      	at org.jboss.as.controller.registry.NodeSubregistry.getOperationEntry(NodeSubregistry.java:186)
      	at org.jboss.as.controller.registry.ConcreteResourceRegistration.getOperationEntry(ConcreteResourceRegistration.java:300)
      	at org.jboss.as.controller.registry.NodeSubregistry.getOperationEntry(NodeSubregistry.java:186)
      	at org.jboss.as.controller.registry.ConcreteResourceRegistration.getOperationEntry(ConcreteResourceRegistration.java:300)
      	at org.jboss.as.controller.registry.AbstractResourceRegistration.getOperationEntry(AbstractResourceRegistration.java:180)
      	at org.jboss.as.controller.registry.AbstractResourceRegistration.getOperationEntry(AbstractResourceRegistration.java:175)
      	at org.jboss.as.controller.OperationContextImpl.getAuthorizationAction(OperationContextImpl.java:1890)
      	at org.jboss.as.controller.OperationContextImpl.getBasicAuthorizationResponse(OperationContextImpl.java:1832)
      	at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1756)
      	at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1306)
      	at org.jboss.as.controller.operations.global.ReadResourceHandler.doExecuteInternal(ReadResourceHandler.java:316)
      	at org.jboss.as.controller.operations.global.ReadResourceHandler.doExecute(ReadResourceHandler.java:171)
      	at org.jboss.as.controller.operations.global.GlobalOperationHandlers$AbstractMultiTargetHandler.execute(GlobalOperationHandlers.java:231)
      	at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:982)
      	at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:726)
      	at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:450)
      	at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1402)
      	at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:418)
      	at org.jboss.as.controller.ModelControllerImpl.lambda$execute$1(ModelControllerImpl.java:243)
      	at org.jboss.as.controller.ModelControllerImpl$$Lambda$656/1109010279.run(Unknown Source)
      	at org.wildfly.security.auth.server.SecurityIdentity$$Lambda$657/2059677950.run(Unknown Source)
      	at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:263)
      	at org.wildfly.security.auth.server.SecurityIdentity.runAs(SecurityIdentity.java:229)
      	at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:243)
      	at org.jboss.as.controller.ModelControllerClientFactoryImpl$LocalClient.executeOperation(ModelControllerClientFactoryImpl.java:131)
      	at org.jboss.as.controller.ModelControllerClientFactoryImpl$1$$Lambda$654/385827253.apply(Unknown Source)
      	at org.jboss.as.controller.ModelControllerClientFactoryImpl.lambda$executeInVm$0(ModelControllerClientFactoryImpl.java:296)
      	at org.jboss.as.controller.ModelControllerClientFactoryImpl$$Lambda$655/955046503.run(Unknown Source)
      	at org.jboss.as.controller.access.InVmAccess.runInVm(InVmAccess.java:85)
      	at org.jboss.as.controller.ModelControllerClientFactoryImpl.executeInVm(ModelControllerClientFactoryImpl.java:296)
      	at org.jboss.as.controller.ModelControllerClientFactoryImpl.access$000(ModelControllerClientFactoryImpl.java:54)
      	at org.jboss.as.controller.ModelControllerClientFactoryImpl$1.executeOperation(ModelControllerClientFactoryImpl.java:77)
      	at org.jboss.as.controller.LocalModelControllerClient.execute(LocalModelControllerClient.java:54)
      	at org.jboss.as.controller.LocalModelControllerClient.execute(LocalModelControllerClient.java:39)
      	at org.jboss.as.server.deployment.scanner.DefaultDeploymentOperations$$Lambda$652/737892850.apply(Unknown Source)
      	at org.jboss.as.server.deployment.scanner.DefaultDeploymentOperations$Execution$1.execute(DefaultDeploymentOperations.java:135)
      	at org.jboss.as.server.deployment.scanner.DefaultDeploymentOperations.getDeploymentsStatus(DefaultDeploymentOperations.java:80)
      	at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1687)
      	at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$ScanContext.<init>(FileSystemDeploymentService.java:1636)
      	at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:589)
      	at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService.scan(FileSystemDeploymentService.java:493)
      	at org.jboss.as.server.deployment.scanner.FileSystemDeploymentService$DeploymentScanRunnable.run(FileSystemDeploymentService.java:255)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      	at org.jboss.threads.JBossThread.run(JBossThread.java:484)
      "MSC service thread 1-1":
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x000000071dff9b08> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
      	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
      	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
      	at org.jboss.as.controller.registry.ConcreteResourceRegistration.getCapabilities(ConcreteResourceRegistration.java:607)
      	at org.jboss.as.controller.registry.NodeSubregistry.unregisterSubModel(NodeSubregistry.java:172)
      	at org.jboss.as.controller.registry.ConcreteResourceRegistration.unregisterSubModel(ConcreteResourceRegistration.java:273)
      	at org.jboss.as.connector.services.resourceadapters.IronJacamarActivationResourceService.stop(IronJacamarActivationResourceService.java:283)
      	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1761)
      	at org.jboss.msc.service.ServiceControllerImpl$StopTask.execute(ServiceControllerImpl.java:1734)
      	at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1521)
      	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1964)
      	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1467)
      	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1346)
      	at java.lang.Thread.run(Thread.java:748)
      "MSC service thread 1-2":
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x000000075357cef0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
      	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
      	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
      	at org.jboss.as.controller.registry.ConcreteResourceRegistration.getSubregistry(ConcreteResourceRegistration.java:571)
      	at org.jboss.as.controller.registry.ConcreteResourceRegistration.getChildAddresses(ConcreteResourceRegistration.java:829)
      	at org.jboss.as.controller.registry.NodeSubregistry.getChildAddresses(NodeSubregistry.java:349)
      	at org.jboss.as.controller.registry.ConcreteResourceRegistration.getChildAddresses(ConcreteResourceRegistration.java:833)
      	at org.jboss.as.controller.registry.NodeSubregistry.getChildAddresses(NodeSubregistry.java:349)
      	at org.jboss.as.controller.registry.ConcreteResourceRegistration.getChildAddresses(ConcreteResourceRegistration.java:833)
      	at org.jboss.as.controller.registry.AbstractResourceRegistration.getChildAddresses(AbstractResourceRegistration.java:326)
      	at org.jboss.as.controller.registry.AbstractResourceRegistration.getChildAddresses(AbstractResourceRegistration.java:323)
      	at org.jboss.as.controller.registry.ConcreteResourceRegistration.unregisterSubModel(ConcreteResourceRegistration.java:264)
      	at org.jboss.as.connector.services.resourceadapters.IronJacamarActivationResourceService.stop(IronJacamarActivationResourceService.java:281)
      	at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1761)
      	at org.jboss.msc.service.ServiceControllerImpl$StopTask.execute(ServiceControllerImpl.java:1734)
      	at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1521)
      	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1964)
      	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1467)
      	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1346)
      	at java.lang.Thread.run(Thread.java:748)
      

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  brian.stansberry Brian Stansberry
                  Reporter:
                  dmlloyd David Lloyd
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: