Failure in SFSB activation

While testing the DTM I ran into a problem that does not appear to be DTM-related.

Scenario:

• stateful session EJB1 deployed in JBoss server1
• entity EJB2 deployed on JBoss server2

What happens:

• Client creates an EJB1 session instance
• Client calls userTransaction.begin() to start transaction T1.
  (The remote calls that follow are all performed as part of T1.)
• Client calls a business method on the EJB1 instance
• EJB1 calls a business method on EJB2
• EJB2 calls entityContext.setRollbackOnly()
• Unaware that T1 has been marked to rollback, client does another
  business method call on EJB1
• EJB1 does another business method call on EJB2 and gets a
  TransactionRolledbackException, as expected.
• EJB1 throws an EJBException with the TransactionRolledBackException
  inside
• Client gets the TransactionRolledbackException
• Client calls userTransaction.rollback() to dissociate itself from T1
• Client calls userTransaction.begin() to start transaction T2.
• Client calls a business method on the EJB1 instance (the same
  instance it used before!)
• Server1 fails while attempting to activate the EJB1 instance:
  DEBUG [org.jboss.ejb.plugins.AbstractInstanceCache] Activation failure
  javax.ejb.EJBException: Could not activate; failed to restore state;
  CausedByException is:
  /home/reverbel/work/new/jboss-head/build/output/jboss-5.0.0alpha/server/dtm0/tmp/sessions/FrontEndEJB-e8z06kpz-9l/e8z06myu-c8.ser (No such file or directory)
  at org.jboss.ejb.plugins.StatefulSessionFilePersistenceManager.activateSession(StatefulSessionFilePersistenceManager.java:338)
  ...
• Client gets a TransactionRolledbackException with message
  "Could not activate; failed to restore state..."

--------------------------------------------------------------------------

This is the full stack trace of the EJB1 activation failure:

2005-05-20 18:46:22,173 DEBUG [org.jboss.ejb.plugins.AbstractInstanceCache] Activation failure
javax.ejb.EJBException: Could not activate; failed to restore state; CausedByException is:
/home/reverbel/work/new/jboss-head/build/output/jboss-5.0.0alpha/server/dtm0/tmp/sessions/FrontEndEJB-e8z06kpz-9l/e8z06myu-c8.ser (No such file or directory)
at org.jboss.ejb.plugins.StatefulSessionFilePersistenceManager.activateSession(StatefulSessionFilePersistenceManager.java:338)
at org.jboss.ejb.plugins.StatefulSessionInstanceCache.activate(StatefulSessionInstanceCache.java:99)
at org.jboss.ejb.plugins.AbstractInstanceCache.doActivate(AbstractInstanceCache.java:412)
at org.jboss.ejb.plugins.StatefulSessionInstanceCache.doActivate(StatefulSessionInstanceCache.java:115)
at org.jboss.ejb.plugins.AbstractInstanceCache.get(AbstractInstanceCache.java:109)
at org.jboss.ejb.plugins.StatefulSessionInstanceInterceptor.invoke(StatefulSessionInstanceInterceptor.java:220)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:105)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:420)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:622)
at org.jboss.ejb.Container.invoke(Container.java:873)
at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:143)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:82)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:247)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:653)
at org.jboss.invocation.unified.server.UnifiedInvoker.invoke(UnifiedInvoker.java:170)
at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:143)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:82)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:247)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:653)
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201)
at $Proxy11.invoke(Unknown Source)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:328)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:204)
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:259)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:327)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:147)
java.io.FileNotFoundException: /home/reverbel/work/new/jboss-head/build/output/jboss-5.0.0alpha/server/dtm0/tmp/sessions/FrontEndEJB-e8z06kpz-9l/e8z06myu-c8.ser (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:106)
at org.jboss.ejb.plugins.StatefulSessionFilePersistenceManager$FISAction.run(StatefulSessionFilePersistenceManager.java:505)
at java.security.AccessController.doPrivileged(Native Method)
at org.jboss.ejb.plugins.StatefulSessionFilePersistenceManager$FISAction.open(StatefulSessionFilePersistenceManager.java:514)
at org.jboss.ejb.plugins.StatefulSessionFilePersistenceManager.activateSession(StatefulSessionFilePersistenceManager.java:317)
at org.jboss.ejb.plugins.StatefulSessionInstanceCache.activate(StatefulSessionInstanceCache.java:99)
at org.jboss.ejb.plugins.AbstractInstanceCache.doActivate(AbstractInstanceCache.java:412)
at org.jboss.ejb.plugins.StatefulSessionInstanceCache.doActivate(StatefulSessionInstanceCache.java:115)
at org.jboss.ejb.plugins.AbstractInstanceCache.get(AbstractInstanceCache.java:109)
at org.jboss.ejb.plugins.StatefulSessionInstanceInterceptor.invoke(StatefulSessionInstanceInterceptor.java:220)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:105)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:420)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:622)
at org.jboss.ejb.Container.invoke(Container.java:873)
at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:143)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:82)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:247)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:653)
at org.jboss.invocation.unified.server.UnifiedInvoker.invoke(UnifiedInvoker.java:170)
at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:143)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:82)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:247)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:653)
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201)
at $Proxy11.invoke(Unknown Source)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:328)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:204)
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:259)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:327)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:147)
2005-05-20 18:46:22,175 ERROR [org.jboss.ejb.plugins.LogInterceptor] TransactionRolledbackException in method: public abstract int[] org.jboss.test.dtm.interfaces.FrontEnd.getBalances() throws java.rmi.RemoteException, causedBy:
java.rmi.NoSuchObjectException: Could not activate; failed to restore state; CausedByException is:
/home/reverbel/work/new/jboss-head/build/output/jboss-5.0.0alpha/server/dtm0/tmp/sessions/FrontEndEJB-e8z06kpz-9l/e8z06myu-c8.ser (No such file or directory)
at org.jboss.ejb.plugins.AbstractInstanceCache.get(AbstractInstanceCache.java:120)
at org.jboss.ejb.plugins.StatefulSessionInstanceInterceptor.invoke(StatefulSessionInstanceInterceptor.java:220)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:105)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:420)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:622)
at org.jboss.ejb.Container.invoke(Container.java:873)
at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:143)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:82)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:247)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:653)
at org.jboss.invocation.unified.server.UnifiedInvoker.invoke(UnifiedInvoker.java:170)
at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:143)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:82)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:247)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:653)
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201)
at $Proxy11.invoke(Unknown Source)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:328)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:204)
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:259)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:327)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:147)

--------------------------------------------------------------------------

This is the stack trace of the TransactionRolledbackException, which is correct and expected, but somehow makes the problem happen:

2005-05-20 18:46:22,142 ERROR [org.jboss.ejb.plugins.LogInterceptor] TransactionRolledbackException in method: public abstract void org.jboss.test.dtm.interfaces.FrontEnd.addToBalances(int) throws java.rmi.RemoteException, causedBy:
javax.transaction.TransactionRolledbackException: Transaction marked for rollback, possibly a timeout; nested exception is:
java.lang.RuntimeException: Transaction marked for rollback, possibly a timeout
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:244)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:420)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:166)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:139)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:192)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
at org.jboss.ejb.EntityContainer.internalInvoke(EntityContainer.java:514)
at org.jboss.ejb.Container.invoke(Container.java:873)
at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:143)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:82)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:247)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:653)
at org.jboss.invocation.unified.server.UnifiedInvoker.invoke(UnifiedInvoker.java:170)
at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:143)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:82)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:247)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:653)
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201)
at $Proxy11.invoke(Unknown Source)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:328)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:204)
at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:259)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:327)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:147)
Caused by: java.lang.RuntimeException: Transaction marked for rollback, possibly a timeout
at org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock.doSchedule(QueuedPessimisticEJBLock.java:228)
at org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock.schedule(QueuedPessimisticEJBLock.java:183)
at org.jboss.ejb.plugins.EntityLockInterceptor.invoke(EntityLockInterceptor.java:85)
at org.jboss.ejb.plugins.EntityCreationInterceptor.invoke(EntityCreationInterceptor.java:53)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:48)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:105)
... 31 more