Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-10206

Cache fails to stop due to unreleased persistence manager semaphore permits when using early terminal stream operations

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 10.0.0.Beta3, 9.4.13.Final
    • Fix Version/s: 9.4.14.Final, 10.0.0.Beta4
    • Component/s: Core, Loaders and Stores
    • Labels:
      None
    • Environment:

      Windows 10 x64
      java version "1.8.0_212"
      Java(TM) SE Runtime Environment (build 1.8.0_212-b10)
      Java HotSpot(TM) 64-Bit Server VM (build 25.212-b10, mixed mode)
      Wildfly 15.0.1

    • Steps to Reproduce:
      Hide

      1. start server and access web application (http://localhost:8080/wildfly-helloworld/hello) so that it fills the caches.
      2. shutdown Wildfly server using jboss-cli.bat --connect --controller=localhost:9990 --command=:shutdown (this time it stops without problems)
      3. start server and access web application (http://localhost:8080/wildfly-helloworld/hello) so that it calls method that accesses the keySets and entrySets from caches
      4. shutdown Wildfly server using jboss-cli.bat --connect --controller=localhost:9990 --command=:shutdown (this time it doesn't stops)

      Show
      1. start server and access web application ( http://localhost:8080/wildfly-helloworld/hello ) so that it fills the caches. 2. shutdown Wildfly server using jboss-cli.bat --connect --controller=localhost:9990 --command=:shutdown (this time it stops without problems) 3. start server and access web application ( http://localhost:8080/wildfly-helloworld/hello ) so that it calls method that accesses the keySets and entrySets from caches 4. shutdown Wildfly server using jboss-cli.bat --connect --controller=localhost:9990 --command=:shutdown (this time it doesn't stops)
    • Affects:
      User Experience
    • Workaround:
      Workaround Exists
    • Workaround Description:
      Hide

      Either kill process if it takes to long or create a ShutdownHook that checks if a cache stop thread stays to much in waiting and force to release a permit:

      ((AbstractQueuedSynchronizer) LockSupport.getBlocker(thread)).releaseShared(1);
      
      Show
      Either kill process if it takes to long or create a ShutdownHook that checks if a cache stop thread stays to much in waiting and force to release a permit: ((AbstractQueuedSynchronizer) LockSupport.getBlocker(thread)).releaseShared(1);

      Description

      Wildfly 15.0.1 fails to stop on shutdown command due to Infinispan subsystem failing to stop it's persisted caches successfully due to a possible bug in Infinispan's PersistenceManagerImpl.publishEntries not releasing Semaphore permit after finishing, so that the stop method waits forever because it lacks one more permit.

      To clarify this issue only happens when invoking an operation that uses an early terminating stream operation that doesn't fully iterate over the store contents. Examples would be `Cache.isEmpty`


      Use following application to reproduce the issue:
      [^web-java-wildfly.zip]

      standalone Infinispan caches:

                  <cache-container name="testMetaData">
                      <local-cache name="testCache">
                          <locking concurrency-level="3"/>
                          <transaction locking="OPTIMISTIC" mode="FULL_XA"/>
                          <object-memory size="1000000"/>
                          <file-store path="testCH/testCache" passivation="false" preload="false" purge="false"/>
                      </local-cache>
                      <local-cache name="testComponentCache">
                          <locking concurrency-level="3"/>
                          <transaction locking="OPTIMISTIC" mode="FULL_XA"/>
                          <object-memory size="1000000"/>
                          <file-store path="testCH/testComponentCache" passivation="false" preload="false" purge="false"/>
                      </local-cache>
                  </cache-container>
      
      11:02:00,103 INFO  [stdout] (default task-1) bla blanull
      11:02:00,103 INFO  [stdout] (default task-1) bla blanull
      11:03:17,748 INFO  [org.jboss.as.server] (management-handler-thread - 1) WFLYSRV0236: Suspending server with no timeout.
      11:03:17,750 INFO  [org.jboss.as.ejb3] (management-handler-thread - 1) WFLYEJB0493: EJB subsystem suspension complete
      11:03:17,752 INFO  [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
      11:03:17,785 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-4) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb1
      11:03:17,785 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-7) WFLYMAIL0002: Unbound mail session [java:/mail/myproject]
      11:03:17,785 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-3) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb2
      11:03:17,787 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb3
      11:03:17,789 INFO  [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 88) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/ConnectionFactory
      11:03:17,790 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-3) WFLYMAIL0002: Unbound mail session [java:jboss/mail/Default]
      11:03:17,790 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-8) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb4
      11:03:17,797 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-4) WFLYJCA0019: Stopped Driver service with driver-name = ojdbc.jar
      11:03:17,800 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 101) WFLYUT0022: Unregistered web context: '/wildfly-helloworld' from server 'default-server'
      11:03:17,821 INFO  [org.jboss.modcluster] (ServerService Thread Pool -- 101) MODCLUSTER000002: Initiating mod_cluster shutdown
      11:03:17,823 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0019: Host default-host stopping
      11:03:17,840 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-3) WFLYJCA0119: Unbinding connection factory named java:/JmsXA to alias java:jboss/DefaultJMSConnectionFactory
      11:03:17,841 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-4) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
      11:03:17,842 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
      11:03:17,842 INFO  [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0019: Stopped Driver service with driver-name = h2
      11:03:17,846 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 101) WFLYCLINF0003: Stopped client-mappings cache from ejb container
      11:03:17,864 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0008: Undertow HTTP listener default suspending
      11:03:17,864 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0008: Undertow AJP listener ajp suspending
      11:03:17,870 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0028: Stopped deployment ojdbc.jar (runtime-name: ojdbc.jar) in 108ms
      11:03:17,883 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0028: Stopped deployment wildfly-helloworld.war (runtime-name: wildfly-helloworld.war) in 120ms
      11:03:17,875 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to 192.168.1.2:8009
      11:03:17,874 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0007: Undertow HTTP listener default stopped, was bound to 192.168.1.2:8080
      11:03:17,872 INFO  [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 104) AMQ151003: resource adaptor stopped
      11:03:17,932 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 104) AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.6.3.jbossorg-00014 [7b985d04-761e-11e9-bb4e-0a0027000008] stopped, uptime 1 minute
      11:03:17,933 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0008: Undertow HTTPS listener default-ssl suspending
      11:03:17,934 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0007: Undertow HTTPS listener default-ssl stopped, was bound to 192.168.1.2:8443
      11:03:17,935 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0004: Undertow 2.0.15.Final stopping
      

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  william.burns Will Burns
                  Reporter:
                  pferraro Paul Ferraro
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: