When an unexpected error occurs in a remote client EJB invocation, the remoting system in both the client and the server hangs. An example of such unexpected error is that the client does not have access to a class used in the return value from the server.

An example stack trace of such invocation:

Caused by: java.lang.ClassNotFoundException: org.hibernate.collection.internal.PersistentSet
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.jboss.marshalling.AbstractClassResolver.loadClass(AbstractClassResolver.java:135)
at org.jboss.marshalling.AbstractClassResolver.resolveClass(AbstractClassResolver.java:116)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:892)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1204)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1677)
at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1593)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1235)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadCollectionObject(RiverUnmarshaller.java:180)
at org.jboss.marshalling.river.RiverUnmarshaller.readCollectionData(RiverUnmarshaller.java:771)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:649)
at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)
at org.jboss.ejb.client.remoting.MethodInvocationResponseHandler$MethodInvocationResultProducer.getResult(MethodInvocationResponseHandler.java:107)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:270)
at org.jboss.ejb.client.TransactionInterceptor.handleInvocationResult(TransactionInterceptor.java:47)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:272)
at org.jboss.ejb.client.ReceiverInterceptor.handleInvocationResult(ReceiverInterceptor.java:132)
at org.jboss.ejb.client.EJBClientInvocationContext.getResult(EJBClientInvocationContext.java:260)
at org.jboss.ejb.client.EJBClientInvocationContext.awaitResponse(EJBClientInvocationContext.java:399)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:140)
at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121)
at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104)

After that, relevant threads on the client look like:

"main" prio=10 tid=0x00007f6e5c007000 nid=0xefe in Object.wait() [0x00007f6e62be6000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

• waiting on <0x00000000fe1a4b08> (a java.lang.Thread)
  at java.lang.Thread.join(Thread.java:1186)
• locked <0x00000000fe1a4b08> (a java.lang.Thread)
  at java.lang.Thread.join(Thread.java:1239)
  at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
  at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
  at java.lang.Shutdown.runHooks(Shutdown.java:79)
  at java.lang.Shutdown.sequence(Shutdown.java:123)
  at java.lang.Shutdown.exit(Shutdown.java:168)
• locked <0x00000000f31219d0> (a java.lang.Class for java.lang.Shutdown)
  at java.lang.Runtime.exit(Runtime.java:90)
  at java.lang.System.exit(System.java:904)
  at ExampleApp(ExampleApp.java:133)

"Thread-1" prio=10 tid=0x00007f6e5c1f2000 nid=0xf12 in Object.wait() [0x00007f6e60846000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

• waiting on <0x00000000fe149190> (a java.lang.Object)
  at java.lang.Object.wait(Object.java:485)
  at org.jboss.remoting3.spi.AbstractHandleableCloseable.close(AbstractHandleableCloseable.java:177)
• locked <0x00000000fe149190> (a java.lang.Object)
  at org.jboss.ejb.client.remoting.AutoConnectionCloser.safeClose(AutoConnectionCloser.java:92)
  at org.jboss.ejb.client.remoting.AutoConnectionCloser.closeAll(AutoConnectionCloser.java:79)
• locked <0x00000000fe1a4bb0> (a java.util.ArrayList)
  at org.jboss.ejb.client.remoting.AutoConnectionCloser.run(AutoConnectionCloser.java:55)
  at java.lang.Thread.run(Thread.java:662)

"Remoting "config-based-ejb-client-endpoint" task-4" daemon prio=10 tid=0x00007f6e18052800 nid=0xf10 waiting on condition [0x00007f6e60947000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00000000fe130130> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  at org.xnio.LimitedBlockingQueue.take(LimitedBlockingQueue.java:95)
  at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  at java.lang.Thread.run(Thread.java:662)

"Remoting "config-based-ejb-client-endpoint" task-3" daemon prio=10 tid=0x00007f6e18050800 nid=0xf0f waiting on condition [0x00007f6e60a48000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00000000fe130130> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  at org.xnio.LimitedBlockingQueue.take(LimitedBlockingQueue.java:95)
  at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  at java.lang.Thread.run(Thread.java:662)

"Remoting "config-based-ejb-client-endpoint" task-2" daemon prio=10 tid=0x00007f6e1804d000 nid=0xf0e waiting on condition [0x00007f6e60b49000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00000000fe130130> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  at org.xnio.LimitedBlockingQueue.take(LimitedBlockingQueue.java:95)
  at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  at java.lang.Thread.run(Thread.java:662)

"Remoting "config-based-ejb-client-endpoint" task-1" daemon prio=10 tid=0x00007f6e1804b800 nid=0xf0d waiting on condition [0x00007f6e60c4a000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)

• parking to wait for <0x00000000fe130130> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
  at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
  at org.xnio.LimitedBlockingQueue.take(LimitedBlockingQueue.java:95)
  at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
  at java.lang.Thread.run(Thread.java:662)

"Remoting "config-based-ejb-client-endpoint" write-1" daemon prio=10 tid=0x00007f6e5c1e1800 nid=0xf0c runnable [0x00007f6e60d4b000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

• locked <0x00000000fe148330> (a sun.nio.ch.Util$2)
• locked <0x00000000fe148340> (a java.util.Collections$UnmodifiableSet)
• locked <0x00000000fe1482e8> (a sun.nio.ch.EPollSelectorImpl)
  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  at org.xnio.nio.WorkerThread.run(WorkerThread.java:153)

"Remoting "config-based-ejb-client-endpoint" read-1" daemon prio=10 tid=0x00007f6e5c1bc800 nid=0xf0b runnable [0x00007f6e60e4c000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

• locked <0x00000000fe130298> (a sun.nio.ch.Util$2)
• locked <0x00000000fe1302a8> (a java.util.Collections$UnmodifiableSet)
• locked <0x00000000fe130250> (a sun.nio.ch.EPollSelectorImpl)
  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
  at org.xnio.nio.WorkerThread.run(WorkerThread.java:153)

And on the server:

"EJB default - 6" prio=10 tid=0x00007fcacc0cd800 nid=0xe3c in Object.wait() [0x00007fcb119d8000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)

• waiting on <0x00000000f9c73080> (a org.jboss.remoting3.remote.OutboundMessage)
  at java.lang.Object.wait(Object.java:485)
  at org.jboss.remoting3.remote.OutboundMessage$1.accept(OutboundMessage.java:91)
• locked <0x00000000f9c73080> (a org.jboss.remoting3.remote.OutboundMessage)
  at org.xnio.streams.BufferPipeOutputStream.send(BufferPipeOutputStream.java:125)
  at org.xnio.streams.BufferPipeOutputStream.send(BufferPipeOutputStream.java:113)
  at org.xnio.streams.BufferPipeOutputStream.getBuffer(BufferPipeOutputStream.java:77)
  at org.xnio.streams.BufferPipeOutputStream.write(BufferPipeOutputStream.java:86)
• locked <0x00000000f9c730b8> (a org.xnio.streams.BufferPipeOutputStream)
  at org.jboss.remoting3.remote.OutboundMessage.write(OutboundMessage.java:168)
  at java.io.DataOutputStream.write(DataOutputStream.java:71)
• locked <0x00000000f9c73060> (a java.io.DataOutputStream)
  at org.jboss.as.ejb3.remote.protocol.versionone.AbstractMessageHandler$1.write(AbstractMessageHandler.java:188)
  at java.io.OutputStream.write(OutputStream.java:99)
  at org.jboss.marshalling.OutputStreamByteOutput.write(OutputStreamByteOutput.java:56)
  at org.jboss.marshalling.SimpleDataOutput.write(SimpleDataOutput.java:83)
  at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:302)
  at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063)
  at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019)
  at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)
  at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:585)
  at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
  at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
  at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.writeMethodInvocationResponse(MethodInvocationMessageHandler.java:338)
  at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$500(MethodInvocationMessageHandler.java:64)
  at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:226)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
  at java.util.concurrent.FutureTask.run(FutureTask.java:138)
  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
  at java.lang.Thread.run(Thread.java:662)
  at org.jboss.threads.JBossThread.run(JBossThread.java:122)

"Remoting "vlinux" read-1" prio=10 tid=0x00007fcadc08e800 nid=0xd33 waiting for monitor entry [0x00007fcb116cf000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.xnio.streams.BufferPipeOutputStream.close(BufferPipeOutputStream.java:156)

• waiting to lock <0x00000000f9c730b8> (a org.xnio.streams.BufferPipeOutputStream)
  at org.xnio.IoUtils.safeClose(IoUtils.java:137)
  at org.jboss.remoting3.remote.OutboundMessage.closeAsync(OutboundMessage.java:158)
  at org.jboss.remoting3.remote.RemoteConnectionChannel.handleAsyncClose(RemoteConnectionChannel.java:502)
  at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:259)
  at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:45)
  at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
  at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
  at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
  at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
  at org.xnio.nio.NioHandle.run(NioHandle.java:90)
  at org.xnio.nio.WorkerThread.run(WorkerThread.java:184)

When the server is in this state, new remoting invocations on the server can not be made. The remote client stalls for a short time, and then times out with "No EJB receiver available for handling".

The server and client are so wedged that the only way to shut them down on linux is kill -9.