Uploaded image for project: 'EJB Client Library (AS7+)'
  1. EJB Client Library (AS7+)
  2. EJBCLIENT-133

EjbClient: No cluster node manager found for node XY during server restart

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 2.0.3.Final
    • Fix Version/s: None
    • Labels:
      None

      Description

      Situation: 4-node cluster, one node is failed at a time, during the whole test, clients are accessing stateless clustered EJB3. The failure type in this particular case is simulating server crash: JVM is killed using kill -9.
      When the server was starting, being brought back, we saw this error:
      16:05:29,966 ERROR [org.jboss.ejb.client.ClusterContext] (ejb-client-cluster-node-connection-creation-2-thread-17) Cannot create EJBReceiver since no cluster node manager found for node perf20 in cluster context for cluster ejb

      Here is the log output again with context, the server was starting:

      16:05:29,966 INFO  [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-2) Retrying invocation which failed on node perf20 with exception:
      javax.ejb.NoSuchEJBException: No such EJB[appname=clusterbench-ee6,modulename=clusterbench-ee6-ejb,distinctname=,beanname=RemoteStatelessSBImpl]
      	at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64)
      	at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:366)
      	at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:458)
      	at org.jboss.remoting3.remote.RemoteConnectionChannel$4.run(RemoteConnectionChannel.java:373)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
      	at java.lang.Thread.run(Thread.java:662)
      16:05:29,966 ERROR [org.jboss.ejb.client.ClusterContext] (ejb-client-cluster-node-connection-creation-2-thread-17) Cannot create EJBReceiver since no cluster node manager found for node perf20 in cluster context for cluster ejb
      16:05:29,974 INFO  [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-2) Retrying invocation which failed on node perf20 with exception:
      javax.ejb.NoSuchEJBException: No such EJB[appname=clusterbench-ee6,modulename=clusterbench-ee6-ejb,distinctname=,beanname=RemoteStatelessSBImpl]
      	at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64)
      	at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:366)
      	at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:458)
      	at org.jboss.remoting3.remote.RemoteConnectionChannel$4.run(RemoteConnectionChannel.java:373)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
      	at java.lang.Thread.run(Thread.java:662)
      16:05:29,974 ERROR [org.jboss.ejb.client.ClusterContext] (ejb-client-cluster-node-connection-creation-2-thread-18) Cannot create EJBReceiver since no cluster node manager found for node perf20 in cluster context for cluster ejb
      16:05:29,975 INFO  [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-2) Retrying invocation which failed on node perf20 with exception:
      javax.ejb.NoSuchEJBException: No such EJB[appname=clusterbench-ee6,modulename=clusterbench-ee6-ejb,distinctname=,beanname=RemoteStatelessSBImpl]
      	at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64)
      	at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:366)
      	at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:458)
      	at org.jboss.remoting3.remote.RemoteConnectionChannel$4.run(RemoteConnectionChannel.java:373)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
      	at java.lang.Thread.run(Thread.java:662)
      16:05:29,975 INFO  [org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler] (Remoting "config-based-ejb-client-endpoint" task-2) Retrying invocation which failed on node perf20 with exception:
      javax.ejb.NoSuchEJBException: No such EJB[appname=clusterbench-ee6,modulename=clusterbench-ee6-ejb,distinctname=,beanname=RemoteStatelessSBImpl]
      	at org.jboss.ejb.client.remoting.NoSuchEJBExceptionResponseHandler.processMessage(NoSuchEJBExceptionResponseHandler.java:64)
      	at org.jboss.ejb.client.remoting.ChannelAssociation.processResponse(ChannelAssociation.java:366)
      	at org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver.handleMessage(ChannelAssociation.java:458)
      	at org.jboss.remoting3.remote.RemoteConnectionChannel$4.run(RemoteConnectionChannel.java:373)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
      

      During the whole test, there were 4 server crashes and startups (1 for each node), but only 3 occurences of the above mentioned error. These cluster nodes are perf18, perf19, perf20, perf21, but this error was seen only for perf20 (two occurences) and perf18 (one occurence).
      I did not find anything suspicious in the server.log of perf20.

      Cache: REPL_ASYNC

      Versions: EAP 6.1.0.ER6, ejb-client 1.0.19.Final

      Link to hudson job:
      https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbstateless-jvmkill-repl-async/15/

      Server log:
      https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-failover-ejb-ejbstateless-jvmkill-repl-async/15/artifact/report/config/jboss-perf20/server.log

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                thofman Tomas Hofman
                Reporter:
                dpospisil Dominik Pospisil
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: