Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-6686

JGroups ForkChannel can hold references to Hibernate SessionFactoryImpl which cause memory leak

    XMLWordPrintable

    Details

    • Steps to Reproduce:
      Hide

      1. deploy an application using JPA with Hibernate second level cache enabled
      2. use the second level cache (I'm note sure if this step is really needed)
      3. undeploy the application
      4. take a heap dump
      5. search for incoming references to SessionFactoryImpl

      Show
      1. deploy an application using JPA with Hibernate second level cache enabled 2. use the second level cache (I'm note sure if this step is really needed) 3. undeploy the application 4. take a heap dump 5. search for incoming references to SessionFactoryImpl

      Description

      We are using Hibernate second level cache through JPA configured as such:

          <properties>
              <property name="hibernate.cache.use_query_cache" value="true"/>
              <property name="hibernate.cache.use_second_level_cache" value="true"/>
              <property name="hibernate.cache.region.factory_class" value="org.jboss.as.jpa.hibernate5.infinispan.SharedInfinispanRegionFactory"/>
          </properties>
      

      After heap dump inspection, it seems that the JGroups ForkChannel identified by "hibernate" can hold a listener that hold the SessionFactoryImpl which then hold the whole application classloader.

      When undeploying the application, this can lead to classloader leak.
      I took an heap dump of such scenario and analysed it using eclipse memory analyzer (MAT) and here is the result:

      Class Name                                                                                                  | Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap
      ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
      channel org.jgroups.JChannel @ 0xc0eac6a0                                                                   |            1 |          112 |               136 |           448
      '- channel_listeners java.util.concurrent.CopyOnWriteArraySet @ 0xc0ecd260                                  |            1 |           16 |               136 |           112
         '- al java.util.concurrent.CopyOnWriteArrayList @ 0xc0ecd270                                             |            1 |           24 |               136 |            96
            '- array java.lang.Object[2] @ 0xc0ecd2b8                                                             |            1 |           24 |               136 |            24
               '- [0] org.jgroups.fork.ForkChannel @ 0xc103d250                                                   |            1 |          120 |               136 |         1 328
                  '- channel_listeners java.util.concurrent.CopyOnWriteArraySet @ 0xc103d350                      |            1 |           16 |               136 |           968
                     '- al java.util.concurrent.CopyOnWriteArrayList @ 0xc103d360                                 |            1 |           24 |               136 |           952
                        '- array java.lang.Object[1] @ 0xc103d3a8                                                 |            1 |           24 |               136 |           880
                           '- [0] org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher @ 0xc103d1f0|            1 |           96 |               136 |           856
      ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
      

      To remove that leak I ugily patched JGroups ForkChannel close method as such:

          /** Closes the fork-channel, essentially setting its state to CLOSED. Note that - contrary to a regular channel -
           * a closed fork-channel can be connected again: this means re-attaching the fork-channel to the main-channel*/
          @Override
          public void close() {
              ((ForkProtocolStack)prot_stack).remove(fork_channel_id);
              if(state == State.CLOSED)
                  return;
              disconnect();                     // leave group if connected
              prot_stack.destroy();
              state=State.CLOSED;
              notifyChannelClosed(this);
      	this.clearChannelListeners(); // <-- this is the line I added
          }
      

      With that change in place, the memory leak is gone. I highly doubt though this is an acceptable fix. Though it does confirm my theory.

      I doubt that JGroups is really the culprit – I'm more in the thinking that the "thing managing" JGroups is the culprit.

      Since I'm not an expert around that field I've opened the issue against the Wildfly project. Feel free to move it to the proper project.

      If you need any other informations let me know.

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  pferraro Paul Ferraro
                  Reporter:
                  matlach Mathieu Lachance
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  5 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: