JCA connected to remote live server with dedicated backup is not able to reconnect when fail-back occurs

Hi Clebert,
we have problem with test scenario with remote JCA. We have live HQ server with dedicated backup with deployed queues InputQueue and OutputQueue. And we have server with deployed MDB where JCA is configured to make remote connections to live HQ server. MDB reads messages from input queues and resends messages to output queue. Problem is that JCA is not able to make reconnection back to live HQ server when failback occurs and it is not able to continue to consume messages.

Test configuration:
Server hqa, production profile + queues InputQueue, OutputQueue
Server hqa-backup, dedicated backup for hqa
Server hqm, production profile + deployed MDB + JCA make connections to hqa via autodiscovery mechanism.

Step by step scenario:
1. Start servers hqa and hqa-backup
2. Run client and push 10 000 messages to InputQueue on hqa server
3. Start hqm and let MDB to consume several messages
4. Kill server hqa via -9 signal
5. Backup server takes its role, you will see lot of exceptions on the hqm server (See [1])
6. After huge amount of exceptions [1], hqm servers start to consume messages from backup server
7. Restart server hqa
8. When is hqa started, backup server stops to serving messages
9. JCA is not able to reconnect back to live server

We have tested this scenario with modified configuration for transactions, we have used unique node id for transactions for each node, and we have used configuration where we had registered recovery modules for each node hq (live + backup) for hqm server. Same issue, this additional configuration does not fix this issue.

Please could you check our configuration? Configuration is based on CR2. Have we missed some configuration? Or could you confirm that it is new issue?
I am attaching configuration and implementation with script for easier issue replication. We have tested this scenario on following builds: CR2, cannot be used, see issue JBPAPP-6017, NB r10280 and NB r10291.

This scenario was requested by Putnam for this reason I am marking it as blocker. This problem should be solved before CR3. Issue is very similar to postponed issues JBPAPP-5808 and JBPAPP-5805. Ivo has reported issues with standalone client.

[1] - following exceptions are hits only, they did not occur in one row

2011-03-08 12:30:42,230 ERROR [org.hornetq.core.client.impl.ClientSessionImpl] (Thread-25 (group:HornetQ-client-global-threads-1080281053)) Caught jmsexecptione
HornetQException[errorCode=5 message=Connection failure detected. Unblocking a blocking call that will never get a response]
at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:286)
at org.hornetq.core.client.impl.ClientSessionImpl.end(ClientSessionImpl.java:1246)
at org.hornetq.core.client.impl.DelegatingSession.end(DelegatingSession.java:332)
at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelPrepare(XAResourceRecord.java:259)
at com.arjuna.ats.arjuna.coordinator.BasicAction.doPrepare(BasicAction.java:2919)
at com.arjuna.ats.arjuna.coordinator.BasicAction.doPrepare(BasicAction.java:2876)
at com.arjuna.ats.arjuna.coordinator.BasicAction.prepare(BasicAction.java:2419)
at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1790)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:94)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:160)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1431)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:137)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.endTransaction(MessageInflowLocalProxy.java:464)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.finish(MessageInflowLocalProxy.java:343)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.after(MessageInflowLocalProxy.java:259)
at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:140)
at $Proxy149.afterDelivery(Unknown Source)
at org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:282)
at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:841)
at org.hornetq.core.client.impl.ClientConsumerImpl.access$100(ClientConsumerImpl.java:45)
at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:958)
at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
2011-03-08 12:30:42,637 ERROR [org.hornetq.ra.HornetQRASessionFactoryImpl] (Thread-5 (group:HornetQ-client-global-threads-1080281053)) Could not create session
javax.resource.ResourceException: Transaction is not active: tx=TransactionImple < ac, BasicAction: a221bd5:c789:4d76132f:fe5 status: ActionStatus.ABORT_ONLY >
at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:413)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)
at org.hornetq.ra.HornetQRASessionFactoryImpl.allocateConnection(HornetQRASessionFactoryImpl.java:837)
at org.hornetq.ra.HornetQRASessionFactoryImpl.createSession(HornetQRASessionFactoryImpl.java:470)
at org.jboss.qa.hornetq.MessageDrivenBeanToQueueMDB.onMessage(MessageDrivenBeanToQueueMDB.java:82)
at sun.reflect.GeneratedMethodAccessor288.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.jboss.aop.joinpoint.MethodInvocation.invokeTarget(MethodInvocation.java:122)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
at org.jboss.ejb3.interceptors.container.ContainerMethodInvocationWrapper.invokeNext(ContainerMethodInvocationWrapper.java:72)
at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.invoke(InterceptorSequencer.java:76)
at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.aroundInvoke(InterceptorSequencer.java:62)
at sun.reflect.GeneratedMethodAccessor287.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.jboss.aop.advice.PerJoinpointAdvice.invoke(PerJoinpointAdvice.java:174)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.fillMethod(InvocationContextInterceptor.java:72)
at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_fillMethod_721725057.invoke(InvocationContextInterceptor_z_fillMethod_721725057.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.setup(InvocationContextInterceptor.java:88)
at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_setup_721725057.invoke(InvocationContextInterceptor_z_setup_721725057.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:62)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:56)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:68)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126)
at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:194)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

2011-03-08 12:30:42,613 ERROR [org.hornetq.core.client.impl.ClientSessionImpl] (Thread-30 (group:HornetQ-client-global-threads-1080281053)) XA operation failed Cannot start, session is already doing work in a transaction null code:-6
2011-03-08 12:30:42,613 WARN [com.arjuna.ats.jta.logging.loggerI18N] (Thread-30 (group:HornetQ-client-global-threads-1080281053)) [com.arjuna.ats.internal.jta.transaction.arjunacore.enliststarterror] [com.arjuna.ats.internal.jta.transaction.arjunacore.enliststarterror] TransactionImple.enlistResource - XAResource.start returned: XAException.XAER_PROTO for < 131075, 27, 25, 49459750504998100535899555657585210055544951501025810210110097505049981005358995556575852100555449515010258102101101 >
2011-03-08 12:30:42,613 ERROR [STDERR] (Thread-30 (group:HornetQ-client-global-threads-1080281053)) javax.transaction.xa.XAException
2011-03-08 12:30:42,614 ERROR [STDERR] (Thread-30 (group:HornetQ-client-global-threads-1080281053)) at org.hornetq.core.client.impl.ClientSessionImpl.start(ClientSessionImpl.java:1489)
2011-03-08 12:30:42,614 ERROR [STDERR] (Thread-30 (group:HornetQ-client-global-threads-1080281053)) at org.hornetq.core.client.impl.DelegatingSession.start(DelegatingSession.java:502)
2011-03-08 12:30:42,614 ERROR [STDERR] (Thread-30 (group:HornetQ-client-global-threads-1080281053)) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:799)
2011-03-08 12:30:42,614 ERROR [STDERR] (Thread-30 (group:HornetQ-client-global-threads-1080281053)) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.enlistResource(TransactionImple.java:510)
2011-03-08 12:30:42,614 ERROR [STDERR] (Thread-30 (group:HornetQ-client-global-threads-1080281053)) at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.startTransaction(MessageInflowLocalProxy.java:399)
2011-03-08 12:30:42,614 ERROR [STDERR] (Thread-30 (group:HornetQ-client-global-threads-1080281053)) at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.before(MessageInflowLocalProxy.java:235)
2011-03-08 12:30:42,614 ERROR [STDERR] (Thread-30 (group:HornetQ-client-global-threads-1080281053)) at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:135)
2011-03-08 12:30:42,614 ERROR [STDERR] (Thread-30 (group:HornetQ-client-global-threads-1080281053)) at $Proxy149.beforeDelivery(Unknown Source)