Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-18700

java.lang.OutOfMemoryError: Direct buffer memory

    XMLWordPrintable

Details

    • Hide

      Download and unzip JMeter

      Run the following on e.g. your laptop or some node you will be using as client:

      echo '===================================================='
      echo 'prepare jmeter...'
      echo '===================================================='
      wget https://dlcdn.apache.org//jmeter/binaries/apache-jmeter-5.6.2.zip
      unzip -q apache-jmeter-5.6.2.zip
      

      Prepare a 4 nodes cluster

      You will need another 4 hosts for WF instances;

      We used 4 VMs with the following properties:

      • RAM: 4GB
      • VCPUs: 2 VCPU
      • Disk: 40GB

      Configure each node as in the following:

      echo '===================================================='
      echo 'prepare wildlfy...'
      echo '===================================================='
      wget https://github.com/wildfly/wildfly/releases/download/30.0.0.Final/wildfly-30.0.0.Final.zip
      export WILDFLY_ZIP=wildfly-30.0.0.Final.zip
      export WILDFLY_DIR=wildfly-30.0.0.Final
      
      rm -rdf wildfly
      unzip -q $WILDFLY_ZIP
      mv $WILDFLY_DIR wildfly
      
      # create user for the ejb clients
      ./wildfly/bin/add-user.sh -u joe -p secret-Passw0rd -a
      
      

      Deploy clusterbench on every node of the cluster

      Use the attached clusterbench-ee10.ear (or build it from https://github.com/clusterbench/clusterbench):

      cp clusterbench-ee10.ear ./wildfly/standalone/deployments/
      

      Start the 4 WF instances

      Node1:

      NODE_IP=$(ip -o route get to 8.8.8.8 | sed -n 's/.*src \([0-9.]\+\).*/\1/p')
      ./wildfly/bin/standalone.sh --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.190 -b=$NODE_IP -bprivate=$NODE_IP -Djboss.node.name=wildfly1
      

      Node2:

      NODE_IP=$(ip -o route get to 8.8.8.8 | sed -n 's/.*src \([0-9.]\+\).*/\1/p')
      ./wildfly/bin/standalone.sh --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.190 -b=$NODE_IP -bprivate=$NODE_IP -Djboss.node.name=wildfly2
      

      Node3:

      NODE_IP=$(ip -o route get to 8.8.8.8 | sed -n 's/.*src \([0-9.]\+\).*/\1/p')
      ./wildfly/bin/standalone.sh --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.190 -b=$NODE_IP -bprivate=$NODE_IP -Djboss.node.name=wildfly3
      

      Node4:

      NODE_IP=$(ip -o route get to 8.8.8.8 | sed -n 's/.*src \([0-9.]\+\).*/\1/p')
      ./wildfly/bin/standalone.sh --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.190 -b=$NODE_IP -bprivate=$NODE_IP -Djboss.node.name=wildfly4
      

      Start the JMeter client

      Find the IPs of the nodes where you run WF and store them in the following shell variables:

      NODE1_IP=...
      NODE2_IP=...
      NODE3_IP=...
      NODE4_IP=...
      

      Use the attached clustering-jmeter-samplers-jar-with-dependencies.jar which contains the Java class performing the remote EJB invocation (you can also build it yourself from this repo: https://github.com/tommaso-borgato/clustering-jmeter-samplers) and the TestPlanEJB.jmx JMeter test plan:

      ./apache-jmeter-5.6.2/bin/jmeter -n \
      -t TestPlanEJB.jmx \
      -Jjmeter.save.saveservice.output_format=csv \
      -Jjmeter.save.saveservice.default_delimiter="," \
      -Jjmeter.save.saveservice.autoflush=true \
      -l jmeter_results-perf.csv \
      -Jhost=$NODE1_IP,$NODE2_IP,$NODE3_IP,$NODE4_IP \
      -Jport=8080,8080,8080,8080 \
      -Jpath=/clusterbench/session \
      -Jusername=joe -Jpassword=secret-Passw0rd -Jusers=4000 -Jrampup=60 -Jremote.prog=0 \
      -Jjmeter.save.saveservice.timestamp_format='yyyy/M/dd HH:mm:ss' \
      -Lorg.jboss.eapqe.clustering.jmeter=DEBUG \
      -Juser.classpath=clustering-jmeter-samplers-jar-with-dependencies.jar
      

      Restart all nodes at least once

      After the 4 nodes are up and the jmeter client successfully started the 4000 clients, stop (CTRL+C from terminal) and restart each node with a 3 minutes interval: this means you leave the cluster running for 5 minutes, the you stop one node, wait 3 minutes, restart the node, wait 3 minutes, stop another node... then wait some time and, eventually, you will be able to observe the error;

      NOTE: the error doesn't show up if you run the test locally (4 nodes and client on the same machine)

      NOTE: this error hasn't surfaced in EAP yet

      Show
      Download and unzip JMeter Run the following on e.g. your laptop or some node you will be using as client: echo '====================================================' echo 'prepare jmeter...' echo '====================================================' wget https://dlcdn.apache.org//jmeter/binaries/apache-jmeter-5.6.2.zip unzip -q apache-jmeter-5.6.2.zip Prepare a 4 nodes cluster You will need another 4 hosts for WF instances; We used 4 VMs with the following properties: RAM: 4GB VCPUs: 2 VCPU Disk: 40GB Configure each node as in the following: echo '====================================================' echo 'prepare wildlfy...' echo '====================================================' wget https://github.com/wildfly/wildfly/releases/download/30.0.0.Final/wildfly-30.0.0.Final.zip export WILDFLY_ZIP=wildfly-30.0.0.Final.zip export WILDFLY_DIR=wildfly-30.0.0.Final rm -rdf wildfly unzip -q $WILDFLY_ZIP mv $WILDFLY_DIR wildfly # create user for the ejb clients ./wildfly/bin/add-user.sh -u joe -p secret-Passw0rd -a Deploy clusterbench on every node of the cluster Use the attached clusterbench-ee10.ear (or build it from https://github.com/clusterbench/clusterbench): cp clusterbench-ee10.ear ./wildfly/standalone/deployments/ Start the 4 WF instances Node1: NODE_IP=$(ip -o route get to 8.8.8.8 | sed -n 's/.*src \([0-9.]\+\).*/\1/p') ./wildfly/bin/standalone.sh --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.190 -b=$NODE_IP -bprivate=$NODE_IP -Djboss.node.name=wildfly1 Node2: NODE_IP=$(ip -o route get to 8.8.8.8 | sed -n 's/.*src \([0-9.]\+\).*/\1/p') ./wildfly/bin/standalone.sh --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.190 -b=$NODE_IP -bprivate=$NODE_IP -Djboss.node.name=wildfly2 Node3: NODE_IP=$(ip -o route get to 8.8.8.8 | sed -n 's/.*src \([0-9.]\+\).*/\1/p') ./wildfly/bin/standalone.sh --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.190 -b=$NODE_IP -bprivate=$NODE_IP -Djboss.node.name=wildfly3 Node4: NODE_IP=$(ip -o route get to 8.8.8.8 | sed -n 's/.*src \([0-9.]\+\).*/\1/p') ./wildfly/bin/standalone.sh --server-config=standalone-ha.xml -Djboss.default.multicast.address=230.0.0.190 -b=$NODE_IP -bprivate=$NODE_IP -Djboss.node.name=wildfly4 Start the JMeter client Find the IPs of the nodes where you run WF and store them in the following shell variables: NODE1_IP=... NODE2_IP=... NODE3_IP=... NODE4_IP=... Use the attached clustering-jmeter-samplers-jar-with-dependencies.jar which contains the Java class performing the remote EJB invocation (you can also build it yourself from this repo: https://github.com/tommaso-borgato/clustering-jmeter-samplers ) and the TestPlanEJB.jmx JMeter test plan: ./apache-jmeter-5.6.2/bin/jmeter -n \ -t TestPlanEJB.jmx \ -Jjmeter.save.saveservice.output_format=csv \ -Jjmeter.save.saveservice.default_delimiter="," \ -Jjmeter.save.saveservice.autoflush=true \ -l jmeter_results-perf.csv \ -Jhost=$NODE1_IP,$NODE2_IP,$NODE3_IP,$NODE4_IP \ -Jport=8080,8080,8080,8080 \ -Jpath=/clusterbench/session \ -Jusername=joe -Jpassword=secret-Passw0rd -Jusers=4000 -Jrampup=60 -Jremote.prog=0 \ -Jjmeter.save.saveservice.timestamp_format='yyyy/M/dd HH:mm:ss' \ -Lorg.jboss.eapqe.clustering.jmeter=DEBUG \ -Juser.classpath=clustering-jmeter-samplers-jar-with-dependencies.jar Restart all nodes at least once After the 4 nodes are up and the jmeter client successfully started the 4000 clients, stop (CTRL+C from terminal) and restart each node with a 3 minutes interval: this means you leave the cluster running for 5 minutes, the you stop one node, wait 3 minutes, restart the node, wait 3 minutes, stop another node... then wait some time and, eventually, you will be able to observe the error; NOTE: the error doesn't show up if you run the test locally (4 nodes and client on the same machine) NOTE: this error hasn't surfaced in EAP yet
    • Regression
    • ---
    • Reported by QE

    Description

      We hit this error when running a 4 nodes cluster where we deploy an application containing a stateful session bean; data is replicated across the cluster;
      We start the cluster and then run ~4000 ejb clients from yet another node;
      After some time and after having restarted all nodes at least once, the following error pops out on one or more nodes:

      2023-10-27 11:15:37,894 ERROR [org.xnio.listener] (default I/O-2) XNIO001007: A channel event listener threw an exception: java.lang.OutOfMemoryError: Direct buffer memory
              at java.base/java.nio.Bits.reserveMemory(Bits.java:175)
              at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118)
              at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:318)
              at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool$1.createBuffer(ByteBufferPool.java:514)
              at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool.allocateMaster(ByteBufferPool.java:503)
              at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool.access$000(ByteBufferPool.java:42)
              at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool$DefaultCache.allocate(ByteBufferPool.java:780)
              at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool.allocate(ByteBufferPool.java:161)
              at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool$2.createBuffer(ByteBufferPool.java:535)
              at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool.allocateMaster(ByteBufferPool.java:503)
              at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool.access$000(ByteBufferPool.java:42)
              at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool$DefaultCache.allocate(ByteBufferPool.java:780)
              at org.jboss.xnio@3.8.11.Final//org.xnio.ByteBufferPool.allocate(ByteBufferPool.java:161)
              at org.jboss.remoting@5.0.27.Final//org.jboss.remoting3.remote.RemoteConnection.allocate(RemoteConnection.java:80)
              at org.jboss.remoting@5.0.27.Final//org.jboss.remoting3.remote.ServerConnectionOpenListener.handleEvent(ServerConnectionOpenListener.java:95)
              at org.jboss.remoting@5.0.27.Final//org.jboss.remoting3.remote.HttpUpgradeConnectionProvider$ConnectionAdaptorImpl.accept(HttpUpgradeConnectionProvider.java:270)
              at org.jboss.remoting@5.0.27.Final//org.jboss.remoting3.remote.HttpUpgradeConnectionProvider$ConnectionAdaptorImpl.accept(HttpUpgradeConnectionProvider.java:244)
              at org.jboss.as.remoting@22.0.1.Final//org.jboss.as.remoting.RemotingHttpUpgradeService$1.handleEvent(RemotingHttpUpgradeService.java:163)
              at org.jboss.as.remoting@22.0.1.Final//org.jboss.as.remoting.RemotingHttpUpgradeService$1.handleEvent(RemotingHttpUpgradeService.java:160)
              at org.jboss.xnio@3.8.11.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
              at io.undertow.core@2.3.10.Final//io.undertow.server.handlers.ChannelUpgradeHandler$1.handleUpgrade(ChannelUpgradeHandler.java:80)
              at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpReadListener.exchangeComplete(HttpReadListener.java:390)
              at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpServerConnection.exchangeComplete(HttpServerConnection.java:244)
              at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.invokeExchangeCompleteListeners(HttpServerExchange.java:1361)
              at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.terminateResponse(HttpServerExchange.java:1646)
              at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.closeAndFlushResponse(HttpServerExchange.java:1794)
              at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1783)
              at io.undertow.core@2.3.10.Final//io.undertow.server.AbstractServerConnection$CloseSetter.handleEvent(AbstractServerConnection.java:316)
              at io.undertow.core@2.3.10.Final//io.undertow.server.AbstractServerConnection$CloseSetter.handleEvent(AbstractServerConnection.java:296)
      at org.jboss.xnio@3.8.11.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
              at io.undertow.core@2.3.10.Final//io.undertow.server.handlers.ChannelUpgradeHandler$1.handleUpgrade(ChannelUpgradeHandler.java:80)
              at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpReadListener.exchangeComplete(HttpReadListener.java:390)
              at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpServerConnection.exchangeComplete(HttpServerConnection.java:244)
              at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.invokeExchangeCompleteListeners(HttpServerExchange.java:1361)
              at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.terminateResponse(HttpServerExchange.java:1646)
              at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.closeAndFlushResponse(HttpServerExchange.java:1794)
              at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1783)
              at io.undertow.core@2.3.10.Final//io.undertow.server.AbstractServerConnection$CloseSetter.handleEvent(AbstractServerConnection.java:316)
              at io.undertow.core@2.3.10.Final//io.undertow.server.AbstractServerConnection$CloseSetter.handleEvent(AbstractServerConnection.java:296)
              at org.jboss.xnio@3.8.11.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
              at org.jboss.xnio@3.8.11.Final//org.xnio.StreamConnection.invokeCloseListener(StreamConnection.java:132)
              at org.jboss.xnio@3.8.11.Final//org.xnio.Connection.close(Connection.java:142)
              at io.undertow.core@2.3.10.Final//io.undertow.server.AbstractServerConnection.close(AbstractServerConnection.java:159)
              at org.jboss.xnio@3.8.11.Final//org.xnio.IoUtils.safeClose(IoUtils.java:152)
              at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpResponseConduit.flush(HttpResponseConduit.java:772)
              at io.undertow.core@2.3.10.Final//io.undertow.conduits.HeadStreamSinkConduit.flush(HeadStreamSinkConduit.java:131)
              at org.jboss.xnio@3.8.11.Final//org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162)
              at io.undertow.core@2.3.10.Final//io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:119)
              at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.closeAndFlushResponse(HttpServerExchange.java:1813)
              at io.undertow.core@2.3.10.Final//io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1783)
              at io.undertow.core@2.3.10.Final//io.undertow.server.handlers.ChannelUpgradeHandler.handleRequest(ChannelUpgradeHandler.java:206)
              at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http2.Http2UpgradeHandler.handleRequest(Http2UpgradeHandler.java:101)
              at io.undertow.core@2.3.10.Final//io.undertow.server.handlers.DisallowedMethodsHandler.handleRequest(DisallowedMethodsHandler.java:62)
              at io.undertow.core@2.3.10.Final//io.undertow.server.Connectors.executeRootHandler(Connectors.java:393)
              at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:256)
              at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
              at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:162)
              at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:100)
              at io.undertow.core@2.3.10.Final//io.undertow.server.protocol.http.HttpOpenListener.handleEvent(HttpOpenListener.java:57)
              at org.jboss.xnio@3.8.11.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
              at org.jboss.xnio@3.8.11.Final//org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:291)
              at org.jboss.xnio@3.8.11.Final//org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286)
              at org.jboss.xnio@3.8.11.Final//org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
              at org.jboss.xnio.nio@3.8.11.Final//org.xnio.nio.QueuedNioTcpServer2.acceptTask(QueuedNioTcpServer2.java:178)
              at org.jboss.xnio.nio@3.8.11.Final//org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:624)
              at org.jboss.xnio.nio@3.8.11.Final//org.xnio.nio.WorkerThread.run(WorkerThread.java:491)
      

      heap info of the WF instance emitting the error:

      [hudson@tborgato-clustering-a-2 tmp]$ jps -lv
      98419 /tmp/wildfly/jboss-modules.jar -D[Standalone] -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true --add-exports=java.desktop/sun.awt=ALL-UNNAMED --add-exports=java.naming/com.sun.jndi.ldap=ALL-UNNAMED --add-exports=java.naming/com.sun.jndi.url.ldap=ALL-UNNAMED --add-exports=java.naming/com.sun.jndi.url.ldaps=ALL-UNNAMED --add-exports=jdk.naming.dns/com.sun.jndi.dns=ALL-UNNAMED --add-opens=java.base/com.sun.net.ssl.internal.ssl=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.security=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.management/javax.management=ALL-UNNAMED --add-opens=java.naming/javax.naming=ALL-UNNAMED -Dorg.jboss
      99151 jdk.jcmd/sun.tools.jps.Jps -Dapplication.home=/qa/tools/opt/x86_64/jdk-11.0.19 -Xms8m -Djdk.module.main=jdk.jcmd
      98926 check-leak-0.11.jar
      [hudson@tborgato-clustering-a-2 tmp]$ jcmd 98419 GC.heap_info
      98419:
       garbage-first heap   total 462848K, used 276038K [0x00000000e0000000, 0x0000000100000000)
        region size 1024K, 0 young (0K), 0 survivors (0K)
       Metaspace       used 117302K, capacity 130315K, committed 132608K, reserved 366592K
        class space    used 15524K, capacity 19945K, committed 20736K, reserved 253952K
      [hudson@tborgato-clustering-a-2 tmp]$ jstat -gc 98419
       S0C    S1C    S0U    S1U      EC       EU        OC         OU       MC     MU    CCSC   CCSU   YGC     YGCT    FGC    FGCT    CGC    CGCT     GCT   
       0.0    0.0    0.0    0.0   114688.0  6144.0   356352.0   277806.4  132864.0 117345.6 20736.0 15525.4   1506   10.827  333   185.297  320     5.613  201.737
      

      Find attached:

      • server.zip : logs of the WF instance emitting the error
      • check-leak.zip : report created with check-leak of the WF instance emitting the error (io.undertow.conduits.ReadTimeoutStreamSourceConduit$1 is one of the first domain classes shown: )

      Attachments

        1. check-leak.zip
          34.13 MB
        2. clusterbench-ee10.ear
          75 kB
        3. clustering-jmeter-samplers-jar-with-dependencies.jar
          19.45 MB
        4. screenshot-1.png
          screenshot-1.png
          101 kB
        5. server.zip
          466 kB
        6. TestPlanEJB.jmx
          4 kB
        7. thread-dump.txt
          110 kB
        8. thread-dump-1.txt
          110 kB
        9. wildfly-29.0.1.Final-with-new-deps-memory-dump.zip
          105.97 MB
        10. wildfly-30.0.0.Final-with-old-deps-memory-dump.zip
          98.29 MB

        Issue Links

          Activity

            People

              flaviarnn Flavia Rainone
              tborgato@redhat.com Tommaso Borgato
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: