(7.4.z) OOM Error after node restart, in a 4 nodes cluster

Download and unzip JMeter Run the following on e.g. your laptop or some node you will be using as client: echo '====================================================' echo 'prepare jmeter...' echo '====================================================' wget https://dlcdn.apache.org//jmeter/binaries/apache-jmeter-5.6.2.zip unzip -q apache-jmeter-5.6.2.zip Prepare a 4 nodes cluster You will need another 4 hosts: one for each EAP instance; We used 4 VMs with the following properties: RAM: 4GB VCPUs: 2 VCPU Disk: 40GB Configure each node as in the following: echo '====================================================' echo 'prepare eap...' echo '====================================================' export WILDFLY_ZIP=jboss-eap-8.0.0.GA-CR2.2.zip export WILDFLY_DIR=jboss-eap-8.0 rm -rdf $WILDFLY_DIR unzip -q $WILDFLY_ZIP create user for the ejb clients ./jboss-eap-8.0/bin/add-user.sh -u joe -p secret-Passw0rd -a Deploy clusterbench on every node of the cluster Use the attached [^clusterbench-ee10.ear] (or build it from https://github.com/clusterbench/clusterbench): cp clusterbench-ee10.ear ./jboss-eap-8.0/standalone/deployments/ Start the 4 WF instances Node1: NODE_IP=$(ip -o route get to 8.8.8.8 | sed -n 's/.*src \([0-9.]\+\).*/\1/p') ./jboss-eap-8.0/bin/standalone.sh --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.190 -b=$NODE_IP -bprivate=$NODE_IP -Djboss.node.name=wildfly1 Node2: NODE_IP=$(ip -o route get to 8.8.8.8 | sed -n 's/.*src \([0-9.]\+\).*/\1/p') ./jboss-eap-8.0/bin/standalone.sh --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.190 -b=$NODE_IP -bprivate=$NODE_IP -Djboss.node.name=wildfly2 Node3: NODE_IP=$(ip -o route get to 8.8.8.8 | sed -n 's/.*src \([0-9.]\+\).*/\1/p') ./jboss-eap-8.0/bin/standalone.sh --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.190 -b=$NODE_IP -bprivate=$NODE_IP -Djboss.node.name=wildfly3 Node4: NODE_IP=$(ip -o route get to 8.8.8.8 | sed -n 's/.*src \([0-9.]\+\).*/\1/p') ./jboss-eap-8.0/bin/standalone.sh --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.190 -b=$NODE_IP -bprivate=$NODE_IP -Djboss.node.name=wildfly4 Start the JMeter client Find the IPs of the nodes where you run WF and store them in the following shell variables: NODE1_IP=... NODE2_IP=... NODE3_IP=... NODE4_IP=... Use the attached clustering-jmeter-samplers-jar-with-dependencies.jar which contains the Java class performing the remote EJB invocation (you can also build it yourself from this repo: https://github.com/tommaso-borgato/clustering-jmeter-samplers ) and the TestPlanEJB.jmx JMeter test plan: ./apache-jmeter-5.6.2/bin/jmeter -n \ -t TestPlanEJB.jmx \ -Jjmeter.save.saveservice.output_format=csv \ -Jjmeter.save.saveservice.default_delimiter="," \ -Jjmeter.save.saveservice.autoflush=true \ -l jmeter_results-perf.csv \ -Jhost=$NODE1_IP,$NODE2_IP,$NODE3_IP,$NODE4_IP \ -Jport=8080,8080,8080,8080 \ -Jpath=/clusterbench/session \ -Jusername=joe -Jpassword=secret-Passw0rd -Jusers=4000 -Jrampup=60 -Jremote.prog=0 \ -Jjmeter.save.saveservice.timestamp_format='yyyy/M/dd HH:mm:ss' \ -Lorg.jboss.eapqe.clustering.jmeter=DEBUG \ -Juser.classpath=clustering-jmeter-samplers-jar-with-dependencies.jar Restart the first node After the 4 nodes are up and the jmeter client successfully started the 4000 clients, stop (CTRL+C from terminal) and restart the first node (the coordinator) then wait some time and, eventually, you will be able to observe the error: memory grows to approximately 4073012kB and then the java process is oom killed; NOTE: the error doesn't show up if you run the test locally (4 nodes and client on the same machine)