Clustered stateful session bean removal of expired passivated instances causes deadlock

The background processing of expired stateful sessions that have been passivated can result in deadlock when clustering is used due to the fact that removal of the stateful session occurs with the cache lock held. This conflicts with removal requests sent to invalidate a cache entry. The deadlock is illustrated by the following 3.2.6 traces:

"Thread-25" daemon prio=1 tid=0x89648448 nid=0x3805 in Object.wait() [8aaed000..8aaed87c]
at java.lang.Object.wait(Native Method)
at org.jgroups.blocks.GroupRequest.doExecute(GroupRequest.java:505)
at org.jgroups.blocks.GroupRequest.execute(GroupRequest.java:183)

• locked <0x9b8ea9e8> (a java.lang.Object)
  at org.jgroups.blocks.MessageDispatcher.castMessage(MessageDispatcher.java:384)
  at org.jgroups.blocks.RpcDispatcher.callRemoteMethods(RpcDispatcher.java:134)
  at org.jboss.ha.framework.server.HAPartitionImpl.callMethodOnCluster(HAPartitionImpl.java:620)
  at org.jboss.ha.hasessionstate.server.HASessionStateImpl.removeSession(HASessionStateImpl.java:533)
  at org.jboss.ejb.plugins.StatefulHASessionPersistenceManager.removePassivated(StatefulHASessionPersistenceManager.java:219)
  at org.jboss.ejb.plugins.StatefulSessionInstanceCache.removePassivated(StatefulSessionInstanceCache.java:193)
  at org.jboss.ejb.plugins.LRUStatefulContextCachePolicy$RemoverTask.run(LRUStatefulContextCachePolicy.java:146)
• locked <0x9296b8c0> (a java.lang.Object)
  at java.util.TimerThread.mainLoop(Timer.java:432)
  at java.util.TimerThread.run(Timer.java:382)

"MessageDispatcher up processing thread" daemon prio=1 tid=0x086c06b0 nid=0x3805 waiting for monitor entry [8b788000..8b78887c]
at org.jboss.ejb.plugins.AbstractInstanceCache.remove(AbstractInstanceCache.java:206)

• waiting to lock <0x9296b8c0> (a java.lang.Object)
  at org.jboss.ejb.plugins.StatefulHASessionPersistenceManager.sessionExternallyModified(StatefulHASessionPersistenceManager.java:231)
  at org.jboss.ha.hasessionstate.server.HASessionStateImpl.ownedObjectExternallyModified(HASessionStateImpl.java:587)
  at org.jboss.ha.hasessionstate.server.HASessionStateImpl._setOwnership(HASessionStateImpl.java:504)
  at sun.reflect.GeneratedMethodAccessor400.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:324)
  at org.jgroups.blocks.MethodCall.invoke(MethodCall.java:236)
  at org.jboss.ha.framework.server.HAPartitionImpl.handle(HAPartitionImpl.java:828)
  at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:615)
  at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:512)
  at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:326)
  at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUp(MessageDispatcher.java:722)
  at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.access$300(MessageDispatcher.java:554)
  at org.jgroups.blocks.MessageDispatcher$1.run(MessageDispatcher.java:691)
  at java.lang.Thread.run(Thread.java:534)

The background removal of the expired passivated sessions should not be done with the cache lock held.