Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-5307

CacheException: Initial state transfer timed out for cache

    XMLWordPrintable

Details

    Description

      EAP 7.0.0.DR9
      Scenario: failover-ejb-ejbremote-undeploy-dist-async

      During application deployment (after failover), perf18 logged this error (besides the Replication timeout WARN message):

      [JBossINF] [0m[0m15:29:55,511 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 85) WFLYCLINF0002: Started dist cache from ejb container
      [JBossINF] [0m[0m15:29:55,513 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 86) WFLYCLINF0002: Started clusterbench-ee7.ear.clusterbench-ee7-web-default.war cache from web container
      [JBossINF] [0m[0m15:29:55,516 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 87) WFLYCLINF0002: Started clusterbench-ee7.ear.clusterbench-ee7-web-passivating.war cache from web container
      [JBossINF] [0m[0m15:29:55,538 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 87) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
      [JBossINF] [0m[0m15:29:55,539 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 87) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
      [JBossINF] [0m[0m15:29:56,499 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 87) WFLYCLINF0002: Started clusterbench-ee7.ear/clusterbench-ee7-ejb.jar cache from ejb container
      [JBossINF] [0m[0m15:30:00,507 INFO  [org.jboss.test.clusterbench.common.ejb.CommonStatefulSBImpl] (EJB default - 1) New SFSB created: org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl@5293c097.
      [JBossINF] [0m[0m15:30:01,323 INFO  [org.jboss.test.clusterbench.common.ejb.CommonStatefulSBImpl] (EJB default - 2) New SFSB created: org.jboss.test.clusterbench.ejb.stateful.RemoteStatefulSBImpl@4dfb0a03.
      [JBossINF] [0m[33m15:30:55,461 WARN  [org.infinispan.statetransfer.StateConsumerImpl] (ServerService Thread Pool -- 84) ISPN000286: Issue when retrieving cluster listeners from perf19: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf19
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:755)
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$79(JGroupsTransport.java:589)
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport$$Lambda$22/1199733093.apply(Unknown Source)
      [JBossINF] 	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
      [JBossINF] 	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
      [JBossINF] 	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      [JBossINF] 	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1954)
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
      [JBossINF] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      [JBossINF] 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      [JBossINF] 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      [JBossINF] 	at java.lang.Thread.run(Thread.java:745)
      [JBossINF] 
      [JBossINF] [0m[31m15:31:55,467 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 84) MSC000001: Failed to start service jboss.infinispan.web.dist: org.jboss.msc.service.StartException in service jboss.infinispan.web.dist: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
      [JBossINF] 	at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:107)
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      [JBossINF] 	at java.lang.Thread.run(Thread.java:745)
      [JBossINF] 	at org.jboss.threads.JBossThread.run(JBossThread.java:320)
      [JBossINF] Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.Exception on object of type StateTransferManagerImpl
      [JBossINF] 	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:172)
      [JBossINF] 	at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
      [JBossINF] 	at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
      [JBossINF] 	at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
      [JBossINF] 	at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
      [JBossINF] 	at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:218)
      [JBossINF] 	at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:839)
      [JBossINF] 	at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:629)
      [JBossINF] 	at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:580)
      [JBossINF] 	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:445)
      [JBossINF] 	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:464)
      [JBossINF] 	at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:455)
      [JBossINF] 	at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:120)
      [JBossINF] 	at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:111)
      [JBossINF] 	at org.wildfly.clustering.infinispan.spi.service.CacheBuilder.start(CacheBuilder.java:80)
      [JBossINF] 	at org.wildfly.clustering.service.AsynchronousServiceBuilder$1.run(AsynchronousServiceBuilder.java:102)
      [JBossINF] 	... 4 more
      [JBossINF] Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache dist on perf18
      [JBossINF] 	at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:225)
      [JBossINF] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      [JBossINF] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      [JBossINF] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      [JBossINF] 	at java.lang.reflect.Method.invoke(Method.java:497)
      [JBossINF] 	at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
      [JBossINF] 	... 19 more
      [JBossINF] 
      [JBossINF] [0m[31m15:31:55,476 ERROR [org.jboss.as.server] (management-handler-thread - 2) WFLYSRV0021: Deploy of deployment "clusterbench-ee7.ear" was rolled back with the following failure message: undefined
      

      Which ends up with WFLYSRV0021: Deploy of deployment "clusterbench-ee7.ear" was rolled back.

      In this very same testrun, there was another issue with application deployment on perf20, may be related: JBEAP-1043

      Server log:
      http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-ejbremote-undeploy-dist-async/5/console-perf18/

      This issue is similar to this EAP6 bug: BZ927948

      Attachments

        Issue Links

          Activity

            People

              pferraro@redhat.com Paul Ferraro
              mvinkler1@redhat.com Michal Vinkler
              Votes:
              2 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: