Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-14234

EJB bean method direct memory buffer leaking

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • None
    • 19.0.0.Final
    • EJB, IO
    • None
    • Hide
      bug is solved after wildfly update
      19.0.1.Final -> 22.0.0.Final
      Show
      bug is solved after wildfly update 19.0.1.Final -> 22.0.0.Final
    • ---
    • ---

    Description

      Wildfly.19.0.1.Final standalone

      We're using haproxy to balance ejb calls between wildfly servers.
      That's very useful model, but there are problems with it.

      All connections are declared as remote-outbound-connection and it works.

          <remote-outbound-connection name="{{ c.name }}" outbound-socket-binding-ref="{{ c.socket_binding }}" protocol="http-remoting" security-realm="{{ c.security_realm }}" username="{{ c.user }}">
              <properties>
                  <property name="SASL_POLICY_NOANONYMOUS" value="false"/>
                  <property name="SSL_ENABLED" value="false"/>
              </properties>
          </remote-outbound-connection>
      

      But there is the problem when some from servers is offline. haproxy send response by http and sends RST by tcp, but WF can't make sense of this response.

      Problem can be reprouced when servers listening port, but there is no repsonse for ejb call(in our case it means haproxy doesn't have suitable backend online)
      But everything is clear, when port closed or server don't response for a calls.
      Haproxy doesn't have feature to don't listen port if backed is offline.

      At OS problems shows as huge count(equal to thread count) of connections in state CLOSE_WAIT. That means server had send packet TCP_FIN and client accepted it, but there were no respond on it. In other words WF and it's subsystems didn't agree with closing connection and it becomes to CLOSE_WAIT. At OS side closing connection in sometime and took it as dead(it's normal case)

      At WF memory it appears as growing count of objects org.xnio.ByteBufferSlicePool$PooledByteBuffer, that have attribute buffer that holds about 2mb each.
      Take a look:

      Problem allocated in XNIO library. There were found a mistake and looks very similar to our problem. XNIO-254
      The same time we'd found XNIO-380 and we tried it with last version of this library(snapshot 3.9), no results were there.

      We tried get undrestanding ow DMB clean, but did nothing due to too complicated code design.
      As per investigations DMB used for an SSL connection, but we don't it.
      So we were to patch jboss-remoting class RemoteConnectionProvider.

      after that we'd solved issue with leaking of DMB.
      Our patched library: src_main_java_org_jboss_remoting3_remote_RemoteConnectionProvider.java
      Please take look how this bug can be fixed.

      Attachments

        Activity

          People

            flaviarnn Flavia Rainone
            zharymv Mike Zhary (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: