Details
-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
None
-
None
-
False
-
None
-
False
Description
Cluster does not start properly. Putting non-coordinator members to not Ready state. On coordinator side there is error
[0m[33m14:24:34,632 WARN [org.infinispan.CLUSTER] (timeout-thread--p16-t1) ISPN000071: Caught exception when handling command TopologyJoinCommand\{cacheName='org.infinispan.PERMISSIONS', origin=hsc-1-vzb6c, joinInfo=CacheJoinInfo{consistentHashFactory=org.infinispan.distribution.ch.impl.SyncReplicatedConsistentHashFactory@2a6be5, numSegments=256, numOwners=2, timeout=240000, cacheMode=REPL_SYNC, persistentUUID=df33edc7-cd15-480a-9118-04b745c20ee5, persistentStateChecksum=Optional.empty}, viewId=2}: org.infinispan.util.concurrent.TimeoutException: ISPN000402: Timeout waiting for view 2. Current view is 1, current status is COORDINATOR
at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.topology.ClusterTopologyManagerImpl.lambda$handleJoin$3(ClusterTopologyManagerImpl.java:221)
at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.util.concurrent.ConditionFuture.lambda$newConditionStage$0(ConditionFuture.java:60)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
On non-coordinator side there is error
[0m[31m14:24:35,074 ERROR [org.infinispan.CONFIG] (ServerService Thread Pool -- 80) ISPN000660: Cache org.infinispan.PERMISSIONS start failed, stopping any running components: org.infinispan.commons.CacheConfigurationException: ISPN000659: Component ComponentWrapper\{name=org.infinispan.statetransfer.StateTransferManager, instance=StateTransferManagerImpl [org.infinispan.PERMISSIONS@hsc-1-nj9vq], status=FAILED} failed to start at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:584) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:807) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:379) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:252) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:222) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1009) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:504) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:727) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:673) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:562) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:525) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.security.mappers.ClusterPermissionMapper.start(ClusterPermissionMapper.java:44) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.security.mappers.CorePackageImpl$2.start(CorePackageImpl.java:65) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.security.mappers.CorePackageImpl$2.start(CorePackageImpl.java:58) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:616) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:607) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:576) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:807) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:634) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:598) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:576) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:807) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:379) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:252) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:779) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:747) at org.jboss.as.clustering.infinispan@8.0.0.GA-redhat-00011//org.jboss.as.clustering.infinispan.subsystem.CacheContainerServiceConfigurator.get(CacheContainerServiceConfigurator.java:119) at org.jboss.as.clustering.infinispan@8.0.0.GA-redhat-00011//org.jboss.as.clustering.infinispan.subsystem.CacheContainerServiceConfigurator.get(CacheContainerServiceConfigurator.java:77) at org.wildfly.clustering.service@8.0.0.GA-redhat-00011//org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:63) at org.wildfly.clustering.service@8.0.0.GA-redhat-00011//org.wildfly.clustering.service.AsyncServiceConfigurator$AsyncService.lambda$start$0(AsyncServiceConfigurator.java:117) at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990) at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1348) at java.base/java.lang.Thread.run(Thread.java:840) at org.jboss.threads@2.4.0.Final-redhat-00001//org.jboss.threads.JBossThread.run(JBossThread.java:513) Caused by: java.util.concurrent.CompletionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 2 from hsc-1-ldnxd after 240 seconds at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.util.concurrent.CompletionStages.join(CompletionStages.java:87) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.statetransfer.StateTransferManagerImpl.start(StateTransferManagerImpl.java:123) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.statetransfer.CorePackageImpl$2.start(CorePackageImpl.java:104) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.statetransfer.CorePackageImpl$2.start(CorePackageImpl.java:83) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:616) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:607) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:576) ... 35 more Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 2 from hsc-1-ldnxd after 240 seconds at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:86) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:88) at org.infinispan.core@14.0.17.Final-redhat-00002//org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840)
With CR3 there pop up this exception for the first what I am looking into EAP8 results. When it happened it occured with openshift.KUBE_PING. I do not know if it is so rare, that it was coincidence or since CR3 we will see this more often. I will observe it.
Im past there was similar report https://issues.redhat.com/browse/JBEAP-17079, which did not go nowhere. Paul asked there "Does this test case use stateful sets?". And answer is no it does not. It use DeploymentConfig. Seems test is here scaling up from 0 directly to 3. Which I know is not recommended approach because of coordinator competition. Could that be caus of this?