CLIENT: start invocation (157) // // Invocation is made from EJBClient to forwarding server // EJB INVOCATION HANDLER: send request with possible retries EJBCLIENT INVOCATION CONTEXT: sendRequest() EJBCLIENT INVOCATION CONTEXT: calling interceptor org.jboss.ejb.client.ReceiverInterceptor RECEIVER INTERCEPTOR: transaction node = null, transaction context class = org.jboss.ejb.client.EJBClientUserTransactionContext EJBCLIENT INVOCATION CONTEXT: sendRequest() EJBCLIENT INVOCATION CONTEXT: calling interceptor org.jboss.ejb.client.TransactionInterceptor TRANSACTION INTERCEPTOR (unknown): EJBClientTransactionContext class = org.jboss.ejb.client.EJBClientUserTransactionContext EJBCLIENT INVOCATION CONTEXT: sendRequest() EJBCLIENT INVOCATION CONTEXT: calling interceptor org.jboss.ejb.client.EJBHomeInterceptor EJBCLIENT INVOCATION CONTEXT: sendRequest() EJBCLIENT INVOCATION CONTEXT: calling interceptor org.jboss.ejb.client.EJBObjectInterceptor EJBCLIENT INVOCATION CONTEXT: sendRequest() EJBCLIENT INVOCATION CONTEXT: calling process invocation 15:51:43,993 INFO [stdout] (EJB default - 7) CMTTxINTERCEPTOR(clusterA-node0): calling invokeInOutTx: txn = TransactionImple < ac, BasicAction: 0:ffffc0a80065:7cf5852:556e093b:157 status: ActionStatus.RUNNING > 15:51:43,993 INFO [stdout] (EJB default - 7) DISTRIBUTABLE CACHE (clusterA-node0) : calling get( UnknownSessionID [6965664849555365687052565256556965556866656754514951565154506655]) 15:51:43,993 INFO [org.jboss.as.test.clustering.twoclusters.bean.forwarding.AbstractForwardingStatefulSBImpl] (EJB default - 7) getSerialAndIncrement() called on forwarding node clusterA-node0 // // Invocation is made from forwarding server-client to non-forwarding server, in a managed context // 15:51:43,993 INFO [stdout] (EJB default - 7) EJB INVOCATION HANDLER: send request with possible retries 15:51:43,993 INFO [stdout] (EJB default - 7) EJBCLIENT INVOCATION CONTEXT: sendRequest() 15:51:43,994 INFO [stdout] (EJB default - 7) EJBCLIENT INVOCATION CONTEXT: calling interceptor org.jboss.ejb.client.ReceiverInterceptor 15:51:43,994 INFO [stdout] (EJB default - 7) RECEIVER INTERCEPTOR: transaction node = null, transaction context class = org.jboss.ejb.client.EJBClientManagedTransactionContext 15:51:43,994 INFO [stdout] (EJB default - 7) EJBCLIENT INVOCATION CONTEXT: sendRequest() 15:51:43,994 INFO [stdout] (EJB default - 7) EJBCLIENT INVOCATION CONTEXT: calling interceptor org.jboss.ejb.client.TransactionInterceptor 15:51:43,994 INFO [stdout] (EJB default - 7) TRANSACTION INTERCEPTOR (clusterA-node0): EJBClientTransactionContext class = org.jboss.ejb.client.EJBClientManagedTransactionContext 15:51:43,994 INFO [stdout] (EJB default - 7) MANAGED TXN CONTEXT (clusterA-node0): calling start: xid = [B@4b896148 15:51:43,994 INFO [stdout] (EJB default - 7) EJB CLIENT MANAGED TXN CONTEXT: enlisting resource with txn id: [B@6a7b1e47 15:51:43,994 INFO [stdout] (EJB default - 7) TRANSACTION INTERCEPTOR (clusterA-node0): attaching txn id to request: xid = XidTransactionID [485048484850484848534968484848484848484848484848484848484848484870707070674865564848545348556770535653505353546948575166484848484849535551495052484848484848484848484848484848484848484870707070674865564848545348556770535653505353546948575166484848484849535748484848484848484848484848484848] 15:51:43,994 INFO [stdout] (EJB default - 7) EJBCLIENT INVOCATION CONTEXT: sendRequest() 15:51:43,994 INFO [stdout] (EJB default - 7) EJBCLIENT INVOCATION CONTEXT: calling interceptor org.jboss.ejb.client.EJBHomeInterceptor 15:51:43,995 INFO [stdout] (EJB default - 7) EJBCLIENT INVOCATION CONTEXT: sendRequest() 15:51:43,995 INFO [stdout] (EJB default - 7) EJBCLIENT INVOCATION CONTEXT: calling interceptor org.jboss.ejb.client.EJBObjectInterceptor 15:51:43,995 INFO [stdout] (EJB default - 7) EJBCLIENT INVOCATION CONTEXT: sendRequest() 15:51:43,995 INFO [stdout] (EJB default - 7) EJBCLIENT INVOCATION CONTEXT: calling process invocation 15:51:43,998 INFO [org.wildfly.extension.undertow] (MSC service thread 1-13) WFLYUT0019: Host default-host stopping 15:51:44,000 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 67) MODCLUSTER000002: Initiating mod_cluster shutdown // // Invocation arrives arrives at non-forwarding server and is refused as server is shutting down // 15:51:43,996 ERROR [org.jboss.as.ejb3.invocation] (EJB default - 6) WFLYEJB0034: EJB Invocation failed on component RemoteStatefulSBImpl for method public abstract int org.jboss.as.test.clustering.twoclusters.bean.common.CommonStatefulSB.getSerialAndIncrement(): org.jboss.as.ejb3.component.EJBComponentUnavailableException: WFLYEJB0467: The request was rejected as the container is suspended at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:635) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195) at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:331) at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:69) at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:202) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at org.jboss.threads.JBossThread.run(JBossThread.java:320) // // Exceptional response (NoSuchEJBException) message arrives back at server-client client on a different thread // 15:51:44,003 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (default task-120) RETRY: Retrying invocation which failed on node clusterB-node0 with exception:: javax.ejb.NoSuchEJBException: No such EJB[appname=,modulename=clusterbench-ee6-ejb,distinctname=,beanname=RemoteStatefulSBImpl,viewclassname=org.jboss.as.test.clustering.twoclusters.bean.stateful.RemoteStatefulSB] at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64) at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:386) at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:498) at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:463) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) // // the retry request gets processed on the server-client, but on a different thread, which does not have the original txn associated with it // 15:51:44,004 INFO [stdout] (default task-120) EJBCLIENT INVOCATION CONTEXT: retry request() 15:51:44,004 INFO [stdout] (default task-120) EJBCLIENT INVOCATION CONTEXT: sendRequest() 15:51:44,004 INFO [stdout] (default task-120) EJBCLIENT INVOCATION CONTEXT: calling interceptor org.jboss.ejb.client.ReceiverInterceptor 15:51:44,004 INFO [stdout] (default task-120) RECEIVER INTERCEPTOR: transaction node = null, transaction context class = org.jboss.ejb.client.EJBClientManagedTransactionContext 15:51:44,005 INFO [stdout] (default task-120) EJBCLIENT INVOCATION CONTEXT: sendRequest() 15:51:44,005 INFO [stdout] (default task-120) EJBCLIENT INVOCATION CONTEXT: calling interceptor org.jboss.ejb.client.TransactionInterceptor 15:51:44,005 INFO [stdout] (default task-120) TRANSACTION INTERCEPTOR (clusterA-node0): EJBClientTransactionContext class = org.jboss.ejb.client.EJBClientManagedTransactionContext // there is no enlsiing of resource here 15:51:44,005 INFO [stdout] (default task-120) EJBCLIENT INVOCATION CONTEXT: sendRequest() 15:51:44,005 INFO [stdout] (default task-120) EJBCLIENT INVOCATION CONTEXT: calling interceptor org.jboss.ejb.client.EJBHomeInterceptor 15:51:44,005 INFO [stdout] (default task-120) EJBCLIENT INVOCATION CONTEXT: sendRequest() 15:51:44,005 INFO [stdout] (default task-120) EJBCLIENT INVOCATION CONTEXT: calling interceptor org.jboss.ejb.client.EJBObjectInterceptor 15:51:44,005 INFO [stdout] (default task-120) EJBCLIENT INVOCATION CONTEXT: sendRequest() 15:51:44,005 INFO [stdout] (default task-120) EJBCLIENT INVOCATION CONTEXT: calling process invocation // // Retry of invocation has succeeded on server-client // 15:51:44,005 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (default task-120) RETRY: Retry of invocation which failed on node clusterB-node0 succeeded; removing stale invocation id: 132 15:51:44,008 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 71) ISPN000029: Passivating all entries to disk 15:51:44,024 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 71) ISPN000030: Passivated 2 entries in 15 milliseconds 15:51:44,028 INFO [org.jboss.weld.deployer] (MSC service thread 1-4) WFLYWELD0010: Stopping weld service for deployment clusterbench-ee6-ejb.jar 15:51:44,030 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-9) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS] 15:51:44,031 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-15) WFLYJCA0019: Stopped Driver service with driver-name = h2 15:51:44,049 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 71) WFLYCLINF0003: Stopped clusterbench-ee6-ejb.jar cache from ejb container 15:51:44,057 INFO [stdout] (EJB default - 1) DISTRIBUTABLE CACHE (clusterB-node1) : calling get( UnknownSessionID [5566665350525251546668675251694865524949554951545069505365546767]) 15:51:44,072 INFO [org.jboss.as.server.deployment] (MSC service thread 1-13) WFLYSRV0028: Stopped deployment clusterbench-ee6-ejb.jar (runtime-name: clusterbench-ee6-ejb.jar) in 86ms 15:51:44,072 INFO [org.wildfly.extension.undertow] (MSC service thread 1-11) WFLYUT0008: Undertow AJP listener ajp suspending 15:51:44,076 INFO [org.wildfly.extension.undertow] (MSC service thread 1-11) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to /127.0.0.1:8209 15:51:44,087 INFO [org.wildfly.extension.undertow] (MSC service thread 1-9) WFLYUT0008: Undertow HTTP listener default suspending 15:51:44,086 INFO [org.jboss.ejb.client.remoting] (default task-125) EJBCLIENT000016: Channel Channel ID fce73619 (outbound) of Remoting connection 66e65bdd to /127.0.0.1:8280 can no longer process messages 15:51:44,088 INFO [org.wildfly.extension.undertow] (MSC service thread 1-9) WFLYUT0007: Undertow HTTP listener default stopped, was bound to /127.0.0.1:8280 15:51:44,088 INFO [org.wildfly.extension.undertow] (MSC service thread 1-9) WFLYUT0004: Undertow 1.2.6.Final stopping 15:51:44,090 INFO [org.jboss.as.test.clustering.twoclusters.bean.common.CommonStatefulSBImpl] (EJB default - 1) getSerialAndIncrement() called on non-forwarding node clusterB-node1 15:51:44,103 INFO [stdout] (EJB default - 7) MANAGED TXN CONTEXT (clusterA-node0): calling end: xid = [B@67638495 15:51:44,103 INFO [stdout] (EJB default - 7) MANAGED TXN CONTEXT (clusterA-node0): calling commit: xid = [B@49510a64 15:51:44,106 INFO [org.jboss.ejb.client.remoting] (default task-37) EJBCLIENT000016: Channel Channel ID b8b4cd56 (outbound) of Remoting connection 22e4a9ae to /127.0.0.1:8280 can no longer process messages 15:51:44,110 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 71) ISPN000029: Passivating all entries to disk 15:51:44,118 INFO [org.infinispan.eviction.impl.PassivationManagerImpl] (ServerService Thread Pool -- 71) ISPN000030: Passivated 1 entries in 8 milliseconds 15:51:44,121 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 71) WFLYCLINF0003: Stopped dist cache from ejb container 15:51:44,124 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-13) ISPN000080: Disconnecting JGroups channel ejb 15:51:44,125 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-13) ISPN000082: Stopping the RpcDispatcher for channel ejb 15:51:44,145 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-3,ee,clusterB-node1) ISPN000094: Received new cluster view for channel ejb: [clusterB-node1|2] (1) [clusterB-node1] 15:51:44,158 INFO [org.jboss.as] (MSC service thread 1-10) WFLYSRV0050: WildFly Full 10.0.0.Alpha1-SNAPSHOT (WildFly Core 2.0.0.Alpha2) stopped in 161ms 15:51:44,234 INFO [org.jboss.as.test.clustering.NodeUtil] (main) Stopped container=container-2 // // server-client tries to commit the transaction by sending transaction invocation, but goes to old node // 15:51:49,107 WARN [com.arjuna.ats.jta] (EJB default - 7) ARJUNA016039: onePhaseCommit on < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffffc0a80065:7cf5852:556e093b:157, node_name=1, branch_uid=0:ffffc0a80065:7cf5852:556e093b:159, subordinatenodename=null, eis_name=unknown eis name > (ResourceImpl{transactionKey=0:ffffc0a80065:7cf5852:556e093b:157, ejbClientContext=org.jboss.ejb.client.EJBClientContext@4c370249, nodeName='clusterB-node0', state=null}) failed with exception -: java.lang.IllegalStateException: EJBCLIENT000027: No EJBReceiver available for node name clusterB-node0 at org.jboss.ejb.client.EJBClientContext.requireNodeEJBReceiver(EJBClientContext.java:869) at org.jboss.ejb.client.EJBClientContext.requireNodeEJBReceiverContext(EJBClientContext.java:910) at org.jboss.ejb.client.EJBClientManagedTransactionContext$ResourceImpl.commit(EJBClientManagedTransactionContext.java:246) at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelOnePhaseCommit(XAResourceRecord.java:703) at com.arjuna.ats.arjuna.coordinator.BasicAction.onePhaseCommit(BasicAction.java:2386) at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1520) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96) at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1200) at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126) at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:89) at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:91) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:281) at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:329) at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:79) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.deployment.processors.EjbSuspendInterceptor.processInvocation(EjbSuspendInterceptor.java:53) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:635) at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356) at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340) at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195) at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:331) at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:69) at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:202) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at org.jboss.threads.JBossThread.run(JBossThread.java:320) 15:51:49,135 INFO [stdout] (EJB default - 7) DISTRIBUTABLE CACHE (clusterA-node0) : calling release( UnknownSessionID [6965664849555365687052565256556965556866656754514951565154506655]) CLIENT: made invocation (157) on bean, count = 157, value = 157