Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-15333

Passivation with OffHeap can hang forever with concurrent write

    XMLWordPrintable

Details

    Description

      I have the following configuration on the server:

         <cache-container name="default" statistics="true">
            <jmx enabled="true"/>
            <metrics accurate-size="true"/>
            <transport cluster="${infinispan.cluster.name:cluster}" />
            <replicated-cache mode="SYNC" statistics="true" name="myConf">
             <memory storage="OFF_HEAP" max-count="10" when-full="REMOVE"/>
             <persistence passivation="true">
                <file-store shared="false" preload="false" purge="true"/>
             </persistence>
            </replicated-cache>
         </cache-container>
      

      I have a simple application which puts 600k entries into the cache in parallel threads, i.e. 6 threads each is putting it's portion of entries, each puts 100k entries into the cache over Hot Rod client. In 4 runs from 5, I am starting to get the following exception on the server and on the client:

      Server exception:

      2023-11-22 11:58:34,387 ERROR (non-blocking-thread--p2-t21) [org.infinispan.interceptors.impl.InvocationContextInterceptor] ISPN000136: Error executing command PutKeyValueCommand on Cache 'myConf', writing keys [WrappedByteArray[\J09\k\e\y\5\0\2\5\7\6 (11 bytes)]] org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key WrappedByteArray[\J09\k\e\y\5\0\2\5\7\6 (11 bytes)] and requestor CommandInvocation:fedora-20723:15519. Lock is held by CommandInvocation:fedora-20723:15515
      	at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.get(DefaultLockManager.java:299)
      	at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.get(DefaultLockManager.java:229)
      	at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.checkState(InfinispanLock.java:440)
      	at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.lambda$toInvocationStage$3(InfinispanLock.java:416)
      	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
      	at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
      	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
      	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
      	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
      	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413)
      	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
      	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
      	at java.base/java.lang.Thread.run(Thread.java:833)
      

      Client exception:

      WARN: ISPN004005: Error received from the server: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key WrappedByteArray[\J09\k\e\y\1\0\0\5\8\0 (11 bytes)] and requestor CommandInvocation:fedora-34014:3514. Lock is held by CommandInvocation:fedora-34014:3512
      [WARNING] 
      org.infinispan.client.hotrod.exceptions.HotRodClientException: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key WrappedByteArray[\J09\k\e\y\1\0\0\5\8\0 (11 bytes)] and requestor CommandInvocation:fedora-34014:3514. Lock is held by CommandInvocation:fedora-34014:3512
          at java.util.concurrent.CompletableFuture.reportGet (CompletableFuture.java:396)
          at java.util.concurrent.CompletableFuture.get (CompletableFuture.java:2096)
          at org.infinispan.client.hotrod.impl.Util.await (Util.java:52)
          at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put (RemoteCacheSupport.java:196)
          at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put (RemoteCacheSupport.java:186)
          at test.Main2$Loader.run (Main2.java:50)
      

      The issue appears for both OFF_HEAP and HEAP memory config. The issue doesn't appear if `memory` tag is removed.
      The reproducer java file is attached. Also attaching the thread dump for one of the similar runs.

      Attachments

        Activity

          People

            wburns@redhat.com Will Burns
            amanukya@redhat.com Anna Manukyan
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: