Details
-
Bug
-
Resolution: Done
-
Major
-
None
-
2.0.3.Final
-
None
Description
Situation: 4-node cluster, one node is failed at a time, during the whole test, clients are accessing stateless clustered EJB3. The failure type in this particular case is simulating server crash: JVM is killed using kill -9.
When the server was starting, being brought back, we saw this error:
16:05:29,966 ERROR [org.jboss.ejb.client.ClusterContext] (ejb-client-cluster-node-connection-creation-2-thread-17) Cannot create EJBReceiver since no cluster node manager found for node perf20 in cluster context for cluster ejb
Here is the log output again with context, the server was starting:
16:05:29,966 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-2) Retrying invocation which failed on node perf20 with exception: javax.ejb.NoSuchEJBException: No such EJB[appname=clusterbench-ee6,modulename=clusterbench-ee6-ejb,distinctname=,beanname=RemoteStatelessSBImpl] at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64) at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:366) at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:458) at org.jboss.remoting3.remote.RemoteConnectionChannel$4.run(RemoteConnectionChannel.java:373) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 16:05:29,966 ERROR [org.jboss.ejb.client.ClusterContext] (ejb-client-cluster-node-connection-creation-2-thread-17) Cannot create EJBReceiver since no cluster node manager found for node perf20 in cluster context for cluster ejb 16:05:29,974 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-2) Retrying invocation which failed on node perf20 with exception: javax.ejb.NoSuchEJBException: No such EJB[appname=clusterbench-ee6,modulename=clusterbench-ee6-ejb,distinctname=,beanname=RemoteStatelessSBImpl] at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64) at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:366) at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:458) at org.jboss.remoting3.remote.RemoteConnectionChannel$4.run(RemoteConnectionChannel.java:373) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 16:05:29,974 ERROR [org.jboss.ejb.client.ClusterContext] (ejb-client-cluster-node-connection-creation-2-thread-18) Cannot create EJBReceiver since no cluster node manager found for node perf20 in cluster context for cluster ejb 16:05:29,975 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-2) Retrying invocation which failed on node perf20 with exception: javax.ejb.NoSuchEJBException: No such EJB[appname=clusterbench-ee6,modulename=clusterbench-ee6-ejb,distinctname=,beanname=RemoteStatelessSBImpl] at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64) at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:366) at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:458) at org.jboss.remoting3.remote.RemoteConnectionChannel$4.run(RemoteConnectionChannel.java:373) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) 16:05:29,975 INFO [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-2) Retrying invocation which failed on node perf20 with exception: javax.ejb.NoSuchEJBException: No such EJB[appname=clusterbench-ee6,modulename=clusterbench-ee6-ejb,distinctname=,beanname=RemoteStatelessSBImpl] at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64) at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:366) at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:458) at org.jboss.remoting3.remote.RemoteConnectionChannel$4.run(RemoteConnectionChannel.java:373) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
During the whole test, there were 4 server crashes and startups (1 for each node), but only 3 occurences of the above mentioned error. These cluster nodes are perf18, perf19, perf20, perf21, but this error was seen only for perf20 (two occurences) and perf18 (one occurence).
I did not find anything suspicious in the server.log of perf20.
Cache: REPL_ASYNC
Versions: EAP 6.1.0.ER6, ejb-client 1.0.19.Final
Link to hudson job:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbstateless-jvmkill-repl-async/15/
