Hangs in mixed domain testsuite

http://lightning.mw.lab.eng.bos.redhat.com/jenkins/job/wildfly-param-pull/7449/console

https://dl.dropboxusercontent.com/u/712508/mixed-hang-debug.zip contains details.

30805.dump is the client and shows the problem is occurring in the @After cleanup of MixedDomainDeploymentTest as it removes deployments from the domain.

"main" prio=10 tid=0xf7605c00 nid=0x7856 in Object.wait() [0xf776e000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

• waiting on <0xe50275f0> (a org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl)
  at java.lang.Object.wait(Object.java:503)
  at org.jboss.threads.AsyncFutureTask.await(AsyncFutureTask.java:192)
• locked <0xe50275f0> (a org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl)
  at org.jboss.threads.AsyncFutureTask.get(AsyncFutureTask.java:266)
• locked <0xe50275f0> (a org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl)
  at org.jboss.as.controller.client.impl.AbstractDelegatingAsyncFuture.get(AbstractDelegatingAsyncFuture.java:100)
  at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:127)
  at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:71)
  at org.jboss.as.controller.client.helpers.domain.impl.DomainClientImpl.execute(DomainClientImpl.java:81)
  at org.jboss.as.test.integration.domain.mixed.MixedDomainDeploymentTest.removeDeployment(MixedDomainDeploymentTest.java:441)
  at org.jboss.as.test.integration.domain.mixed.MixedDomainDeploymentTest.cleanDeployments(MixedDomainDeploymentTest.java:343)
  at org.jboss.as.test.integration.domain.mixed.MixedDomainDeploymentTest.confirmNoDeployments(MixedDomainDeploymentTest.java:163)

31894.dump shows the master HC. management-handler-thread is blocking on the way out (after sending the commit or rollback) waiting for a final result response from the slave.

"management-handler-thread - 1" prio=10 tid=0xca0efc00 nid=0x7cdd in Object.wait() [0xc7469000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

• waiting on <0xeb6d6558> (a org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl)
  at java.lang.Object.wait(Object.java:503)
  at org.jboss.threads.AsyncFutureTask.await(AsyncFutureTask.java:192)
• locked <0xeb6d6558> (a org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl)
  at org.jboss.threads.AsyncFutureTask.get(AsyncFutureTask.java:266)
• locked <0xeb6d6558> (a org.jboss.as.protocol.mgmt.ActiveOperationSupport$ActiveOperationImpl)
  at org.jboss.as.controller.client.impl.AbstractDelegatingAsyncFuture.get(AbstractDelegatingAsyncFuture.java:100)
  at org.jboss.as.domain.controller.operations.coordination.DomainSlaveHandler.finalizeOp(DomainSlaveHandler.java:215)
  at org.jboss.as.domain.controller.operations.coordination.DomainSlaveHandler.access$000(DomainSlaveHandler.java:58)
  at org.jboss.as.domain.controller.operations.coordination.DomainSlaveHandler$1.handleResult(DomainSlaveHandler.java:179)
  at org.jboss.as.controller.AbstractOperationContext$Step.handleRollback(AbstractOperationContext.java:805)
  at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:763)
  at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:738)

(Note that the "handleRollback" method name is a red-herring – the method should be renamed to a "handleResult" as it now is called no matter what the result.)

31986.dump is the slave HC. It shows it is blocking in stage DONE after having sent operationPrepared to the master, waiting for commit or rollback.

"domain-connection-threads - 1" prio=10 tid=0xc91a8400 nid=0x7d1a waiting on condition [0xc87fe000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0xea1fd410> (a java.util.concurrent.CountDownLatch$Sync)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
  at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:236)
  at org.jboss.as.controller.remote.TransactionalProtocolOperationHandler$ProxyOperationControlProxy.operationPrepared(TransactionalProtocolOperationHandler.java:173)
  at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:337)
  at org.jboss.as.controller.AbstractOperationContext.completeStep(AbstractOperationContext.java:211)
  at org.jboss.as.domain.controller.operations.coordination.ServerOperationsResolverHandler.execute(ServerOperationsResolverHandler.java:128)

Indication is the commit or rollback message is getting lost. There are no threads shown sending or receiving it.