Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-16832

[GSS](7.4.z) "ISPN000299: Unable to acquire lock" happens because of the deadlock between SFSB expiration and passivation

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • None
    • 7.2.0.GA, 7.2.1.CR2, 7.4.6.GA
    • Clustering, EJB
    • None
    • +
    • Hide

      Disable SFSB passivation

      Show
      Disable SFSB passivation
    • Hide

      You can reproduce with the following byteman rule, configuration and applicaiton:

      • Use the following byteman rule to trigger a race condition easily:
      RULE org.wildfly.clustering.ejb.infinispan.group.InfinispanBeanGroupFactory_remove_sleep
CLASS org.wildfly.clustering.ejb.infinispan.group.InfinispanBeanGroupFactory
METHOD remove
AT ENTRY
IF TRUE
DO
    # traceStack("### org.wildfly.clustering.ejb.infinispan.group.InfinispanBeanGroupFactory#remove() is invoked: " + $1.toString() + "\n");
    traceln("### org.wildfly.clustering.ejb.infinispan.group.InfinispanBeanGroupFactory#remove() - sleeping....");
    Thread.sleep(5000);
ENDRULE
      • Modify standalone-full-ha.xml and set a smaller value to the "max-size" attribute of passivation-store in ejb3 subsystem in order to passivate SFSB quickly:
              <subsystem xmlns="urn:jboss:domain:ejb3:5.0">
            ...
            <passivation-stores>
                <passivation-store name="infinispan" cache-container="ejb" max-size="1"/> <!-- change from 10000 to 1 -->
            </passivation-stores>
      • Use the quick start application ejb-in-war with specifying a short expiration to GreeterEJB (10 seconds expiration like the following):
      package org.jboss.as.quickstarts.ejbinwar.ejb;

import javax.ejb.Stateful;
import javax.ejb.StatefulTimeout;
import java.util.concurrent.TimeUnit;

...

@Stateful
@StatefulTimeout(value=10, unit=TimeUnit.SECONDS)
public class GreeterEJB {
    ...

      Steps to reproduce:

      1. Send a request to one node:

      curl -v http://node1:8080/ejb-in-war/greeter.jsf

      2. Wait 10 seconds for the expiration of the SFSB and the following log message:

      INFO  [stdout] (InfinispanBeanManager - 1) ### org.wildfly.clustering.ejb.infinispan.group.InfinispanBeanGroupFactory#remove() - sleeping....

      3. Send the next request during the above sleep duration:

      curl -v http://node1:8080/ejb-in-war/greeter.jsf

      or

      curl -v http://node2:8080/ejb-in-war/greeter.jsf

      4. You will see ISPN000299 TimeoutException in server.log.
      If you send a request to node1 in step 3, you will see a slow response due to ISPN000299.
      If you send a request to node2 in step 3, you will face a slow response and get an error response finally.

      Show
      You can reproduce with the following byteman rule, configuration and applicaiton: Use the following byteman rule to trigger a race condition easily: RULE org.wildfly.clustering.ejb.infinispan.group.InfinispanBeanGroupFactory_remove_sleep CLASS org.wildfly.clustering.ejb.infinispan.group.InfinispanBeanGroupFactory METHOD remove AT ENTRY IF TRUE DO # traceStack( "### org.wildfly.clustering.ejb.infinispan.group.InfinispanBeanGroupFactory#remove() is invoked: " + $1.toString() + "\n" ); traceln( "### org.wildfly.clustering.ejb.infinispan.group.InfinispanBeanGroupFactory#remove() - sleeping...." ); Thread .sleep(5000); ENDRULE Modify standalone-full-ha.xml and set a smaller value to the "max-size" attribute of passivation-store in ejb3 subsystem in order to passivate SFSB quickly: <subsystem xmlns= "urn:jboss:domain:ejb3:5.0" > ... <passivation-stores> <passivation-store name= "infinispan" cache-container= "ejb" max-size= "1" /> <!-- change from 10000 to 1 --> </passivation-stores> Use the quick start application ejb-in-war with specifying a short expiration to GreeterEJB (10 seconds expiration like the following): package org.jboss.as.quickstarts.ejbinwar.ejb; import javax.ejb.Stateful; import javax.ejb.StatefulTimeout; import java.util.concurrent.TimeUnit; ... @Stateful @StatefulTimeout(value=10, unit=TimeUnit.SECONDS) public class GreeterEJB { ... Steps to reproduce: 1. Send a request to one node: curl -v http: //node1:8080/ejb-in-war/greeter.jsf 2. Wait 10 seconds for the expiration of the SFSB and the following log message: INFO [stdout] (InfinispanBeanManager - 1) ### org.wildfly.clustering.ejb.infinispan.group.InfinispanBeanGroupFactory#remove() - sleeping.... 3. Send the next request during the above sleep duration: curl -v http: //node1:8080/ejb-in-war/greeter.jsf or curl -v http: //node2:8080/ejb-in-war/greeter.jsf 4. You will see ISPN000299 TimeoutException in server.log. If you send a request to node1 in step 3, you will see a slow response due to ISPN000299. If you send a request to node2 in step 3, you will face a slow response and get an error response finally.

    Description

      A deadlock can happen between SFSB expiration and passivation.

      • One thread holds the infinispan level lock and waits for the java level lock in Caffeine
      • Another thread holds the java level lock in Caffeine and waits for the infinispan level lock

      As the dead lock is not released until the ISPN000299 TimeoutException, this can cause a slow response (on a single node) or an error response (on the clustered nodes).

      The following exception happens at the time of the issue:

      12:43:19,795 ERROR [org.infinispan.interceptors.impl.InvocationContextInterceptor] (async-thread--p21-t2) ISPN000136: Error executing command RemoveCommand, writing keys [InfinispanBeanGroupKey(UUIDSessionID [958f33e3-5642-4100-984f-4e9357edecbe])]: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key InfinispanBeanGroupKey(UUIDSessionID [958f33e3-5642-4100-984f-4e9357edecbe]) and requestor GlobalTx:local:11. Lock is held by GlobalTx:local:9
	at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.get(DefaultLockManager.java:288)
	at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.get(DefaultLockManager.java:218)
	at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.checkState(InfinispanLock.java:436)
	at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.lambda$toInvocationStage$3(InfinispanLock.java:412)
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
	at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
	at java.lang.Thread.run(Thread.java:748)

12:43:19,797 WARN  [org.wildfly.clustering.ejb.infinispan] (default task-1) ISPN000299: Unable to acquire lock after 15 seconds for key InfinispanBeanGroupKey(UUIDSessionID [958f33e3-5642-4100-984f-4e9357edecbe]) and requestor GlobalTx:local:11. Lock is held by GlobalTx:local:9: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key InfinispanBeanGroupKey(UUIDSessionID [958f33e3-5642-4100-984f-4e9357edecbe]) and requestor GlobalTx:local:11. Lock is held by GlobalTx:local:9
	at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:259)
	at org.infinispan.cache.impl.CacheImpl.executeCommandWithInjectedTx(CacheImpl.java:1729)
	at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1707)
	at org.infinispan.cache.impl.CacheImpl.remove(CacheImpl.java:632)
	at org.infinispan.cache.impl.DecoratedCache.remove(DecoratedCache.java:549)
	at org.infinispan.cache.impl.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:363)
	at org.infinispan.cache.impl.EncoderCache.remove(EncoderCache.java:683)
	at org.infinispan.cache.impl.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:363)
	at org.wildfly.clustering.ejb.infinispan.group.InfinispanBeanGroupFactory.remove(InfinispanBeanGroupFactory.java:119)
	at org.wildfly.clustering.ejb.infinispan.group.InfinispanBeanGroup.close(InfinispanBeanGroup.java:122)
	at org.wildfly.clustering.ejb.infinispan.group.InfinispanBeanGroupFactory.passivated(InfinispanBeanGroupFactory.java:148)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.lambda$invoke$1(AbstractListenerImpl.java:382)
	at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:20)
	at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:404)
	at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1699)
	at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1695)
	at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invokeNoChecks(CacheNotifierImpl.java:1690)
	at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1647)
	at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryPassivated(CacheNotifierImpl.java:641)
	at org.infinispan.eviction.impl.PassivationManagerImpl.passivate(PassivationManagerImpl.java:71)
	at org.infinispan.container.impl.AbstractInternalDataContainer$DefaultEvictionListener.onEntryChosenForEviction(AbstractInternalDataContainer.java:488)
	at org.infinispan.container.impl.AbstractInternalDataContainer$1.delete(AbstractInternalDataContainer.java:469)
	at org.infinispan.container.impl.AbstractInternalDataContainer$1.delete(AbstractInternalDataContainer.java:455)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$evictEntry$2(BoundedLocalCache.java:786)
	at java.util.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1769)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.evictEntry(BoundedLocalCache.java:756)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.evictFromMain(BoundedLocalCache.java:638)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.evictEntries(BoundedLocalCache.java:504)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.maintenance(BoundedLocalCache.java:1137)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.performCleanUp(BoundedLocalCache.java:1108)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache$PerformCleanupTask.run(BoundedLocalCache.java:2979)
	at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:20)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.scheduleDrainBuffers(BoundedLocalCache.java:1080)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.scheduleAfterWrite(BoundedLocalCache.java:1047)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.afterWrite(BoundedLocalCache.java:1017)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.remap(BoundedLocalCache.java:2272)
	at com.github.benmanes.caffeine.cache.BoundedLocalCache.compute(BoundedLocalCache.java:2146)
	at com.github.benmanes.caffeine.cache.LocalCache.compute(LocalCache.java:100)
	at org.infinispan.container.impl.AbstractInternalDataContainer.put(AbstractInternalDataContainer.java:140)
	at org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:145)
	at org.infinispan.statetransfer.CommitManager.commitEntry(CommitManager.java:139)
	at org.infinispan.statetransfer.CommitManager.commit(CommitManager.java:97)
	at org.infinispan.interceptors.locking.ClusteringDependentLogic$LocalLogic.commitSingleEntry(ClusteringDependentLogic.java:352)
	at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.commitEntry(ClusteringDependentLogic.java:189)
	at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:584)
	at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:813)
	at org.infinispan.interceptors.impl.EntryWrappingInterceptor.lambda$commitContextEntries$7(EntryWrappingInterceptor.java:570)
	at java.util.HashMap.forEach(HashMap.java:1289)
	at org.infinispan.context.EntryLookup.forEachEntry(EntryLookup.java:60)
	at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:569)
	at org.infinispan.interceptors.impl.EntryWrappingInterceptor.lambda$new$1(EntryWrappingInterceptor.java:140)
	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:105)
	at org.infinispan.interceptors.impl.EntryWrappingInterceptor.prepareHandler(EntryWrappingInterceptor.java:178)
	at org.infinispan.interceptors.impl.EntryWrappingInterceptor.wrapEntriesForPrepareAndApply(EntryWrappingInterceptor.java:837)
	at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:173)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185)
	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
	at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
	at org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:132)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185)
	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:98)
	at org.infinispan.interceptors.impl.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:41)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185)
	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:98)
	at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:121)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185)
	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
	at org.infinispan.interceptors.impl.TxInterceptor.handlePrepareCommand(TxInterceptor.java:157)
	at org.infinispan.interceptors.impl.TxInterceptor.visitPrepareCommand(TxInterceptor.java:126)
	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185)
	at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:123)
	at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90)
	at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:248)
	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
	at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:155)
	at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:127)
	at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68)
	at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:419)
	at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:466)
	at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335)
	at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110)
	at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
	at org.jboss.as.ejb3.cache.distributable.DistributableCache.create(DistributableCache.java:101)
	at org.jboss.as.ejb3.component.stateful.StatefulComponentSessionIdGeneratingInterceptor.processInvocation(StatefulComponentSessionIdGeneratingInterceptor.java:57)
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
	...(snip)...
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key InfinispanBeanGroupKey(UUIDSessionID [958f33e3-5642-4100-984f-4e9357edecbe]) and requestor GlobalTx:local:11. Lock is held by GlobalTx:local:9
	at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.get(DefaultLockManager.java:288)
	at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.get(DefaultLockManager.java:218)
	at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.checkState(InfinispanLock.java:436)
	at org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.lambda$toInvocationStage$3(InfinispanLock.java:412)
	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
	at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:442)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
	... 1 more
	Suppressed: org.infinispan.util.logging.TraceException
		at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:41)
		at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:250)
		at org.infinispan.cache.impl.CacheImpl.executeCommandWithInjectedTx(CacheImpl.java:1729)
		at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1707)
		at org.infinispan.cache.impl.CacheImpl.remove(CacheImpl.java:632)
		at org.infinispan.cache.impl.DecoratedCache.remove(DecoratedCache.java:549)
		at org.infinispan.cache.impl.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:363)
		at org.infinispan.cache.impl.EncoderCache.remove(EncoderCache.java:683)
		at org.infinispan.cache.impl.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:363)
		at org.wildfly.clustering.ejb.infinispan.group.InfinispanBeanGroupFactory.remove(InfinispanBeanGroupFactory.java:119)
		at org.wildfly.clustering.ejb.infinispan.group.InfinispanBeanGroup.close(InfinispanBeanGroup.java:122)
		at org.wildfly.clustering.ejb.infinispan.group.InfinispanBeanGroupFactory.passivated(InfinispanBeanGroupFactory.java:148)
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.lambda$invoke$1(AbstractListenerImpl.java:382)
		at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:20)
		at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:404)
		at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1699)
		at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1695)
		at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invokeNoChecks(CacheNotifierImpl.java:1690)
		at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1647)
		at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryPassivated(CacheNotifierImpl.java:641)
		at org.infinispan.eviction.impl.PassivationManagerImpl.passivate(PassivationManagerImpl.java:71)
		at org.infinispan.container.impl.AbstractInternalDataContainer$DefaultEvictionListener.onEntryChosenForEviction(AbstractInternalDataContainer.java:488)
		at org.infinispan.container.impl.AbstractInternalDataContainer$1.delete(AbstractInternalDataContainer.java:469)
		at org.infinispan.container.impl.AbstractInternalDataContainer$1.delete(AbstractInternalDataContainer.java:455)
		at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$evictEntry$2(BoundedLocalCache.java:786)
		at java.util.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1769)
		at com.github.benmanes.caffeine.cache.BoundedLocalCache.evictEntry(BoundedLocalCache.java:756)
		at com.github.benmanes.caffeine.cache.BoundedLocalCache.evictFromMain(BoundedLocalCache.java:638)
		at com.github.benmanes.caffeine.cache.BoundedLocalCache.evictEntries(BoundedLocalCache.java:504)
		at com.github.benmanes.caffeine.cache.BoundedLocalCache.maintenance(BoundedLocalCache.java:1137)
		at com.github.benmanes.caffeine.cache.BoundedLocalCache.performCleanUp(BoundedLocalCache.java:1108)
		at com.github.benmanes.caffeine.cache.BoundedLocalCache$PerformCleanupTask.run(BoundedLocalCache.java:2979)
		at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:20)
		at com.github.benmanes.caffeine.cache.BoundedLocalCache.scheduleDrainBuffers(BoundedLocalCache.java:1080)
		at com.github.benmanes.caffeine.cache.BoundedLocalCache.scheduleAfterWrite(BoundedLocalCache.java:1047)
		at com.github.benmanes.caffeine.cache.BoundedLocalCache.afterWrite(BoundedLocalCache.java:1017)
		at com.github.benmanes.caffeine.cache.BoundedLocalCache.remap(BoundedLocalCache.java:2272)
		at com.github.benmanes.caffeine.cache.BoundedLocalCache.compute(BoundedLocalCache.java:2146)
		at com.github.benmanes.caffeine.cache.LocalCache.compute(LocalCache.java:100)
		at org.infinispan.container.impl.AbstractInternalDataContainer.put(AbstractInternalDataContainer.java:140)
		at org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:145)
		at org.infinispan.statetransfer.CommitManager.commitEntry(CommitManager.java:139)
		at org.infinispan.statetransfer.CommitManager.commit(CommitManager.java:97)
		at org.infinispan.interceptors.locking.ClusteringDependentLogic$LocalLogic.commitSingleEntry(ClusteringDependentLogic.java:352)
		at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.commitEntry(ClusteringDependentLogic.java:189)
		at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:584)
		at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:813)
		at org.infinispan.interceptors.impl.EntryWrappingInterceptor.lambda$commitContextEntries$7(EntryWrappingInterceptor.java:570)
		at java.util.HashMap.forEach(HashMap.java:1289)
		at org.infinispan.context.EntryLookup.forEachEntry(EntryLookup.java:60)
		at org.infinispan.interceptors.impl.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:569)
		at org.infinispan.interceptors.impl.EntryWrappingInterceptor.lambda$new$1(EntryWrappingInterceptor.java:140)
		at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:105)
		at org.infinispan.interceptors.impl.EntryWrappingInterceptor.prepareHandler(EntryWrappingInterceptor.java:178)
		at org.infinispan.interceptors.impl.EntryWrappingInterceptor.wrapEntriesForPrepareAndApply(EntryWrappingInterceptor.java:837)
		at org.infinispan.interceptors.impl.EntryWrappingInterceptor.visitPrepareCommand(EntryWrappingInterceptor.java:173)
		at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185)
		at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
		at org.infinispan.interceptors.DDAsyncInterceptor.handleDefault(DDAsyncInterceptor.java:54)
		at org.infinispan.interceptors.DDAsyncInterceptor.visitPrepareCommand(DDAsyncInterceptor.java:132)
		at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185)
		at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:98)
		at org.infinispan.interceptors.impl.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:41)
		at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185)
		at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextThenAccept(BaseAsyncInterceptor.java:98)
		at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitPrepareCommand(PessimisticLockingInterceptor.java:121)
		at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185)
		at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNext(BaseAsyncInterceptor.java:54)
		at org.infinispan.interceptors.impl.TxInterceptor.handlePrepareCommand(TxInterceptor.java:157)
		at org.infinispan.interceptors.impl.TxInterceptor.visitPrepareCommand(TxInterceptor.java:126)
		at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:185)
		at org.infinispan.interceptors.BaseAsyncInterceptor.invokeNextAndExceptionally(BaseAsyncInterceptor.java:123)
		at org.infinispan.interceptors.impl.InvocationContextInterceptor.visitCommand(InvocationContextInterceptor.java:90)
		at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:248)
		at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:137)
		at org.infinispan.transaction.impl.TransactionCoordinator.commit(TransactionCoordinator.java:155)
		at org.infinispan.transaction.xa.XaTransactionTable.commit(XaTransactionTable.java:127)
		at org.infinispan.transaction.xa.TransactionXaAdapter.commit(TransactionXaAdapter.java:68)
		at org.infinispan.commons.tx.TransactionImpl.finishResource(TransactionImpl.java:419)
		at org.infinispan.commons.tx.TransactionImpl.commitResources(TransactionImpl.java:466)
		at org.infinispan.commons.tx.TransactionImpl.runCommit(TransactionImpl.java:335)
		at org.infinispan.commons.tx.TransactionImpl.commit(TransactionImpl.java:110)
		at org.wildfly.clustering.ee.infinispan.InfinispanBatch.close(InfinispanBatch.java:97)
		at org.jboss.as.ejb3.cache.distributable.DistributableCache.create(DistributableCache.java:101)
		at org.jboss.as.ejb3.component.stateful.StatefulComponentSessionIdGeneratingInterceptor.processInvocation(StatefulComponentSessionIdGeneratingInterceptor.java:57)
		at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:422)
		at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:53)
		at org.jboss.as.ee.component.ViewService$DefaultViewInstanceFactory.createViewInstance(ViewService.java:305)
		at org.jboss.as.ee.component.ViewService$View.createInstance(ViewService.java:184)
		at org.jboss.as.ee.component.ViewService$View.createInstance(ViewService.java:174)
		...(snip)...
		at javax.faces.webapp.FacesServlet.service(FacesServlet.java:671)
		at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
		at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
		at io.opentracing.contrib.jaxrs2.server.SpanFinishingFilter.doFilter(SpanFinishingFilter.java:55)
		at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
		at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
		at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
		at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
		at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
		at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
		at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
		at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
		at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
		at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
		at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
		at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
		at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
		at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
		at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
		at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
		at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
		at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
		at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
		at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
		at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
		at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
		at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
		at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
		at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
		at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
		at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
		at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
		at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
		at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
		at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
		at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
		at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
		at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
		at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
		at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
		at io.undertow.server.Connectors.executeRootHandler(Connectors.java:360)
		at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
		at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
		at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
		at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
		at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
		... 1 more

      The threads can vary, but the same issue can happen both with SFSB replication (two cluster nodes using standalone-ha.xml) and without SFSB replication (a single node using standalone-ha.xml, or using <local-cache> instead of <distributed-cache> for ejb cache).

      Attachments

        Issue Links

          links to

          Web Link KCS#4111201

          Activity

            People

              pferraro@redhat.com Paul Ferraro
              rhn-support-mmiura Masafumi Miura
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: