Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-5123

Execution error: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 10.0.0.CR1
    • 10.0.0.Alpha6, 10.0.0.Beta1
    • Clustering
    • None

    Description

      Seen regularly in basically every failover test (no matter what failover type/cache used).

      Setup: 4 node cluster, one node at time is shutdown, while standalone clients keep calling the application.
      After failing one node(failover type: jvmkill/shutdown/undeploy) - perf20 in this case, perf21 logged the following error message:

      [JBossINF] [0m[31m12:57:35,029 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (transport-thread--p7-t10) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for topology 20
      [JBossINF] 	at org.infinispan.statetransfer.StateTransferLockImpl.waitForTransactionData(StateTransferLockImpl.java:92)
      [JBossINF] 	at org.infinispan.interceptors.base.BaseStateTransferInterceptor.waitForTransactionData(BaseStateTransferInterceptor.java:96)
      [JBossINF] 	at org.infinispan.statetransfer.StateTransferInterceptor.handleTxWriteCommand(StateTransferInterceptor.java:274)
      [JBossINF] 	at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:243)
      [JBossINF] 	at org.infinispan.statetransfer.StateTransferInterceptor.visitRemoveCommand(StateTransferInterceptor.java:119)
      [JBossINF] 	at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
      [JBossINF] 	at org.infinispan.interceptors.CacheMgmtInterceptor.visitRemoveCommand(CacheMgmtInterceptor.java:209)
      [JBossINF] 	at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)
      [JBossINF] 	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)
      [JBossINF] 	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)
      [JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:49)
      [JBossINF] 	at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:58)
      [JBossINF] 	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)
      [JBossINF] 	at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1613)
      [JBossINF] 	at org.infinispan.cache.impl.CacheImpl.removeInternal(CacheImpl.java:579)
      [JBossINF] 	at org.infinispan.cache.impl.CacheImpl.remove(CacheImpl.java:572)
      [JBossINF] 	at org.infinispan.cache.impl.DecoratedCache.remove(DecoratedCache.java:442)
      [JBossINF] 	at org.infinispan.cache.impl.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:297)
      [JBossINF] 	at org.wildfly.clustering.server.registry.CacheRegistry.topologyChanged(CacheRegistry.java:152)
      [JBossINF] 	at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
      [JBossINF] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      [JBossINF] 	at java.lang.reflect.Method.invoke(Method.java:497)
      [JBossINF] 	at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl$1.run(AbstractListenerImpl.java:286)
      [JBossINF] 	at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:22)
      [JBossINF] 	at org.infinispan.notifications.impl.AbstractListenerImpl$ListenerInvocationImpl.invoke(AbstractListenerImpl.java:309)
      [JBossINF] 	at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.doRealInvocation(CacheNotifierImpl.java:1212)
      [JBossINF] 	at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1170)
      [JBossINF] 	at org.infinispan.notifications.cachelistener.CacheNotifierImpl$BaseCacheEntryListenerInvocation.invoke(CacheNotifierImpl.java:1135)
      [JBossINF] 	at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyTopologyChanged(CacheNotifierImpl.java:590)
      [JBossINF] 	at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:203)
      [JBossINF] 	at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:47)
      [JBossINF] 	at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:115)
      [JBossINF] 	at org.infinispan.topology.LocalTopologyManagerImpl.doHandleTopologyUpdate(LocalTopologyManagerImpl.java:282)
      [JBossINF] 	at org.infinispan.topology.LocalTopologyManagerImpl$1.run(LocalTopologyManagerImpl.java:217)
      [JBossINF] 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      [JBossINF] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      [JBossINF] 	at org.infinispan.executors.SemaphoreCompletionService$QueueingTask.runInternal(SemaphoreCompletionService.java:173)
      [JBossINF] 	at org.infinispan.executors.SemaphoreCompletionService$QueueingTask.run(SemaphoreCompletionService.java:151)
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      [JBossINF] 	at java.lang.Thread.run(Thread.java:745)
      [JBossINF] 
      

      Server log:
      http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-ejbremote-shutdown-dist-async/3/console-perf21/

      This error sometimes occur just before the server shuts down, as can be seen here:
      http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-ejbremote-shutdown-dist-async/3/console-perf19/

      See also: JBEAP-505#comment-13098103

      This issue is also related to: WFLY-4697#comment-13083024

      Attachments

        Issue Links

          Activity

            People

              pferraro@redhat.com Paul Ferraro
              mvinkler1@redhat.com Michal Vinkler
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: