Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-6376

Class loader leak due to Infinispan ExpirationManagerImpl

    XMLWordPrintable

Details

    Description

      Scenario

      Given a WAR containing a persistence unit with second level cache and query cache enabled, I'm consistently hitting a Metaspace OutOfMemoryError after redeploying the unchanged application a couple of times.

      Analyzing the situation with Eclipse Memory Analyzer, I found one cause to be WFLY-6348, but even after applying that fix locally, I'm still having a Classloader leak which I can trace to a thread used by Infinispan referencing an obsolete web app classloader as context classloader.

      Just to rule out that this might be related to WFLY-6283, WFLY-6285, I repeated my experiments with a local build of WildFly master (2f11a59aee0dbdd52b65c5c684eafa83c3f418da), with Hibernate locally upgraded to 5.0.9.

      I'm still getting a classloader leak with that build.

      Analysis

      org.infinispan.expiration.impl.ExpirationManagerImpl uses a LazyInitializingScheduledExecutorService. Due to lazy initialization, the ExecutorService and the underlying thread pool is not created until my web app is deployed. Thus, when the ExecutorService is created, the context class loader is set to the web app class loader, and this appears to propagate to the threads of the executor thread pool.

      When the application is undeployed, ExpirationManagerImpl.stop() gets invoked to cancel any running expiration task. However, the ExecutorService is not shut down, the threads remain alive and still keep a reference to the now obsolete context classloader.

      Remarks

      I'm not sure if this analysis is correct, at least I hope there's a clue for WildFly and Infinispan experts to identify the real cause.

      By the way, it would be helpful if all threads created by Infinispan had meaningful names, rather than default names like pool-5-thread-1.

      Attachments

        Issue Links

          Activity

            People

              pferraro@redhat.com Paul Ferraro
              hwellmann.de Harald Wellmann (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: