Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-16341

[GSS](7.2.z) WFLY-11682 - Clustered SLSB membership anomalies when all cluster members removed

    Details

    • Type: Bug
    • Status: Verified (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 7.2.0.GA
    • Fix Version/s: 7.2.2.CR1, 7.2.2.GA
    • Component/s: Clustering, EJB
    • Labels:
    • Environment:
      • 2 WildFly server running the same application (not necessarily clustered), but using the same database persistence for timers
    • Target Release:
    • Steps to Reproduce:
      Hide
      1. Install a plain WildFly server
      2. Build the playground.zip project attached to this Jira using Maven
      3. Copy the build playground.jar to $JBOSS_HOME/standalone/deployments
      4. Multiply the standalone folder into node1, node2 and node3
      5. Start the servers:
        $JBOSS_HOME/bin/standalone.sh -c standalone-ha.xml -Djboss.node.name=node1 -Djboss.server.base.dir=$JBOSS_HOME/node1
        $JBOSS_HOME/bin/standalone.sh -c standalone-ha.xml -Djboss.node.name=node2 -Djboss.server.base.dir=$JBOSS_HOME/node2 -Djboss.socket.binding.port-offset=300
        $JBOSS_HOME/bin/standalone.sh -c standalone-ha.xml -Djboss.node.name=node3 -Djboss.server.base.dir=$JBOSS_HOME/node3 -Djboss.socket.binding.port-offset=600
      6. start the client (300 = 5 minutes):
        mvn -f playground-jar/pom.xml exec:exec -Druntime=300
      7. Stop node1
      8. Stop node2
      9. Stop node3
      10. Start node1 (see above) again
      11. Observe node3 in the list of available node, see:
        INFO (ThreadPoolTaskExecutor-5) [com.jboss.examples.ejb.CustomClusterNodeSelector] connectedNodes(1) '[node1]', availableNodes(2) '[node3, node1]'
      Show
      Install a plain WildFly server Build the playground.zip project attached to this Jira using Maven Copy the build playground.jar to $JBOSS_HOME/standalone/deployments Multiply the standalone folder into node1 , node2 and node3 Start the servers: $JBOSS_HOME/bin/standalone.sh -c standalone-ha.xml -Djboss.node.name=node1 -Djboss.server.base.dir=$JBOSS_HOME/node1 $JBOSS_HOME/bin/standalone.sh -c standalone-ha.xml -Djboss.node.name=node2 -Djboss.server.base.dir=$JBOSS_HOME/node2 -Djboss.socket.binding.port-offset=300 $JBOSS_HOME/bin/standalone.sh -c standalone-ha.xml -Djboss.node.name=node3 -Djboss.server.base.dir=$JBOSS_HOME/node3 -Djboss.socket.binding.port-offset=600 start the client (300 = 5 minutes): mvn -f playground-jar/pom.xml exec:exec -Druntime=300 Stop node1 Stop node2 Stop node3 Start node1 (see above) again Observe node3 in the list of available node, see: INFO (ThreadPoolTaskExecutor-5) [com.jboss.examples.ejb.CustomClusterNodeSelector] connectedNodes(1) ' [node1] ', availableNodes(2) ' [node3, node1] '
    • QE Test Coverage:
      +

      Description

      This description will be based on a 3 node cluster. Cluster node 1 and 2 are configured in the PROVIDER_URL, node 3 is not.

      The client has a custom ClusterNodeSelector implementation that is printing the connectedNodes and the availableNodes and doing a random balancing.

      As long as all nodes are up and running the client is calling EJBs in a balanced way.

      When node1 is shut down, the client get the notification below:

      ...
      DEBUG (XNIO-1 task-1) [org.jboss.ejb.client.invocation] Received MODULE_UNAVAILABLE(9) message for module /playground
      DEBUG (XNIO-1 task-4) [org.jboss.ejb.client.invocation] Received MODULE_UNAVAILABLE(9) message for module /playground
      DEBUG (XNIO-1 task-4) [org.jboss.ejb.client.invocation] Received MODULE_UNAVAILABLE(9) message for module /playground
      DEBUG (XNIO-1 task-1) [org.jboss.ejb.client.invocation] Received MODULE_UNAVAILABLE(9) message for module /playground
      DEBUG (XNIO-1 task-1) [org.jboss.ejb.client.invocation] Received CLUSTER_TOPOLOGY_NODE_REMOVAL(18) message for (cluster, node) = (ejb, node1)
      DEBUG (XNIO-1 task-1) [org.jboss.ejb.client.invocation] Received CLUSTER_TOPOLOGY_NODE_REMOVAL(18) message for (cluster, node) = (ejb, node1)
      ...
      

      Then node2 is shut down. Again the client get the information, see:

      ...
      DEBUG (XNIO-1 task-1) [org.jboss.ejb.client.invocation] Received MODULE_UNAVAILABLE(9) message for module /playground
      DEBUG (XNIO-1 task-1) [org.jboss.ejb.client.invocation] Received MODULE_UNAVAILABLE(9) message for module /playground
      DEBUG (XNIO-1 task-1) [org.jboss.ejb.client.invocation] Received CLUSTER_TOPOLOGY_NODE_REMOVAL(18) message for (cluster, node) = (ejb, node2)
      ...
      

      Finally node3 is being shut down. Now the client only get the following information:

      ...
      DEBUG (XNIO-1 task-1) [org.jboss.ejb.client.invocation] Received MODULE_UNAVAILABLE(9) message for module /playground
      DEBUG (XNIO-1 task-1) [org.jboss.ejb.client.invocation] Received MODULE_UNAVAILABLE(9) message for module /playground
      ...
      

      This mean the node3 is not being informed about the fact that the last node of the cluster has been stopped.

      From this point on the client is always getting Caused by: java.net.ConnectException: Connection refused

      Now node1 is started again, resulting in the following output for connectedNodes and the availableNodes:

      ...
      INFO  (ThreadPoolTaskExecutor-1) [com.jboss.examples.ejb.CustomClusterNodeSelector] connectedNodes(1) '[node1]', availableNodes(2) '[node3, node1]'
      ...
      

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  rachmato Richard Achmatowicz
                  Reporter:
                  jbaesner Joerg Baesner
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  6 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: