NPE on client invoking remote SFSB method during shutdown

Scenario encountered during clustered REPL_ASYNC sfsb clean failover testing:
Exception occurs just after shutdown is initiated on target node.

java.lang.NullPointerException
at org.jboss.cache.Fqn.<init>(Fqn.java:218)
at org.jboss.cache.Fqn.<init>(Fqn.java:184)
at org.jboss.ejb3.cache.tree.StatefulTreeCache.getFqn(StatefulTreeCache.java:579)
at org.jboss.ejb3.cache.tree.StatefulTreeCache.get(StatefulTreeCache.java:154)
at org.jboss.ejb3.cache.tree.StatefulTreeCache.get(StatefulTreeCache.java:148)
at org.jboss.ejb3.stateful.StatefulContainer.dynamicInvoke(StatefulContainer.java:556)
at org.jboss.ejb3.session.InvokableContextClassProxyHack._dynamicInvoke(InvokableContextClassProxyHack.java:53)
at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:91)
at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:891)
at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:744)
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:697)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:551)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232)
at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:211)
at org.jboss.remoting.Client.invoke(Client.java:1724)
at org.jboss.remoting.Client.invoke(Client.java:629)
at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:60)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:84)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:61)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.security.client.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:65)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.remoting.ClusteredIsLocalInterceptor.invoke(ClusteredIsLocalInterceptor.java:54)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.remoting.PojiProxy.invoke(PojiProxy.java:62)
at $Proxy4.invoke(Unknown Source)
at org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:207)
at org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:164)
at $Proxy3.getSerial(Unknown Source)
at org.jboss.smartfrog.ejb3.as5.SFSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(SFSBProcessorFactoryImpl.java:72)
at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:33)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:89)
at java.lang.Thread.run(Thread.java:619)
at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:72)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:84)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:61)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.security.client.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:65)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.remoting.ClusteredIsLocalInterceptor.invoke(ClusteredIsLocalInterceptor.java:54)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.remoting.PojiProxy.invoke(PojiProxy.java:62)
at $Proxy4.invoke(Unknown Source)
at org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:207)
at org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:164)
at $Proxy3.getSerial(Unknown Source)
at org.jboss.smartfrog.ejb3.as5.SFSBProcessorFactoryImpl$EJB3RequestProcessor.processRequest(SFSBProcessorFactoryImpl.java:72)
at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:33)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:89)
at java.lang.Thread.run(Thread.java:619)

The NPE occurs when the Fqn created in StatefulTreeCache.getFqn(...) tries to create an Fqn with a null cacheNode. The reference to cacheNode is created in StatefulTreeCache.start(), and is not dereferenced anywhere, so the NPE indicates that the Cache instance used in dynamicInvoke(...) was never started. Interestingly the Cache instances used in dynamicInvoke(...) are always obtained via StatefulContainer.getCache(), which lazily creates and starts the container's cache if it either does not yet exist or is not started. At first glance, it would appear that the cache returned by getCache() should always be started. The problem, I think, is a concurrency issue with getCache() that occurs when multiple remote requests on the container reach dynamicInvoke(...) when a shutdown is in progress.

So how can StatefulContainer.getCache() return a cache that was never started? Here's a possible scenario:

• Container shutdown initiated, cache is stopped.
• Asynchronous replication from a previous invocation kicks in calling StatefulContainer.getCache(). Due to lazy loading in getCache(), this causes a new cache to be created and started (BAD)
• A second remote invocation reaches StatefulContainer.dynamicInvoke(Invocation) and simultaneously calls StatefulContainer.getCache().
• Either thread may end up returning a new instance of a cache that has not yet started, since the cache validated as started by StatefulContainer.createAndStartCache() may not necessarily be the same instance that is returned by the getCache() method.