Details
-
Bug
-
Resolution: Cannot Reproduce
-
Critical
-
None
-
7.3.0.CD16
-
None
Description
The issue affects Clusters of EAP nodes where nodes a failed un-deploying and then re-deploying the test application (clusterbench); this is worth noting because the issue is not present when nodes are failed shutting down and then re-starting EAP;
The scenario is the ejb-servlet scenario where both the web and the ejb cache are involved (eap-7.x-clustering-ejb-ejbservlet-undeploy-repl);
Server configuration is:
embed-server --server-config=standalone-ha.xml /subsystem=jgroups/stack=udp/transport=UDP/property=ip_ttl:add(value=32) /subsystem=jgroups:write-attribute(name=default-stack,value=tcp) /subsystem=jgroups/channel=ee:write-attribute(name=stack,value=tcp) /subsystem=infinispan/cache-container=web/replicated-cache=testRepl:add() /subsystem=infinispan/cache-container=web/replicated-cache=testRepl/component=locking:write-attribute(name=isolation, value=REPEATABLE_READ) /subsystem=infinispan/cache-container=web/replicated-cache=testRepl/component=transaction:write-attribute(name=mode, value=BATCH) /subsystem=infinispan/cache-container=web:write-attribute(name=default-cache, value=testRepl)
The cluster startup is OK: all the 4 nodes forming the cluster are started and the test application is deployed successfully;
Then the test application is un-deployed and then re-deployed on node 1 and it works;
Then the test application is un-deployed and then re-deployed on node 2 and we get the error:
2019-04-17 15:02:03,044 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 123) MSC000001: Failed to start service org.wildfly.clustering.infinispan.cache.ejb.clusterbench-ee8.ear/clusterbench-ee8-ejb.jar/infinispan: org.jboss.msc.service.StartException in service org.wildfly.clustering.infinispan.cache.ejb.clusterbench-ee8.ear/clusterbench-ee8-ejb.jar/infinispan: org.infinispan.commons.CacheException: Initial state transfer timed out for cache clusterbench-ee8.ear/clusterbench-ee8-ejb.jar/infinispan on wildfly2 at org.wildfly.clustering.service@7.3.0.CD16-redhat-00001//org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:70) at org.wildfly.clustering.service@7.3.0.CD16-redhat-00001//org.wildfly.clustering.service.AsyncServiceConfigurator$AsyncService.lambda$start$0(AsyncServiceConfigurator.java:117) at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1348) at java.base/java.lang.Thread.run(Thread.java:834) at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.JBossThread.run(JBossThread.java:485) Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache clusterbench-ee8.ear/clusterbench-ee8-ejb.jar/infinispan on wildfly2 at org.infinispan@9.4.11.Final-redhat-00001//org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:238) at org.infinispan@9.4.11.Final-redhat-00001//org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1113) at org.infinispan@9.4.11.Final-redhat-00001//org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:511) at org.infinispan@9.4.11.Final-redhat-00001//org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:657) at org.infinispan@9.4.11.Final-redhat-00001//org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:601) at org.infinispan@9.4.11.Final-redhat-00001//org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:484) at org.infinispan@9.4.11.Final-redhat-00001//org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:468) at org.infinispan@9.4.11.Final-redhat-00001//org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:454) at org.jboss.as.clustering.infinispan@7.3.0.CD16-redhat-00001//org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:83) at org.wildfly.clustering.infinispan.spi@7.3.0.CD16-redhat-00001//org.wildfly.clustering.infinispan.spi.service.CacheServiceConfigurator.get(CacheServiceConfigurator.java:77) at org.wildfly.clustering.infinispan.spi@7.3.0.CD16-redhat-00001//org.wildfly.clustering.infinispan.spi.service.CacheServiceConfigurator.get(CacheServiceConfigurator.java:55) at org.wildfly.clustering.service@7.3.0.CD16-redhat-00001//org.wildfly.clustering.service.FunctionalService.start(FunctionalService.java:67) ... 7 more
Note that just before we get the timeout we see a lot (more that 100) of the following errors on the same node:
2019-04-17 14:58:08,670 ERROR [io.undertow.request] (default task-18) UT005023: Exception handling request to /clusterbench/ejbservlet: org.jboss.weld.exceptions.IllegalStateException: BeanIdentifier index not built! at org.jboss.weld.core@3.1.0.Final-redhat-00001//org.jboss.weld.serialization.BeanIdentifierIndex.checkIsBuilt(BeanIdentifierIndex.java:169) at org.jboss.weld.core@3.1.0.Final-redhat-00001//org.jboss.weld.serialization.BeanIdentifierIndex.getIndex(BeanIdentifierIndex.java:65) at org.jboss.weld.core@3.1.0.Final-redhat-00001//org.jboss.weld.contexts.beanstore.BeanIdentifierIndexNamingScheme.prefix(BeanIdentifierIndexNamingScheme.java:62) at org.jboss.weld.core@3.1.0.Final-redhat-00001//org.jboss.weld.contexts.beanstore.AttributeBeanStore.get(AttributeBeanStore.java:138) at org.jboss.weld.core@3.1.0.Final-redhat-00001//org.jboss.weld.module.web.context.beanstore.http.AbstractSessionBeanStore.get(AbstractSessionBeanStore.java:85) at org.jboss.weld.core@3.1.0.Final-redhat-00001//org.jboss.weld.contexts.AbstractContext.get(AbstractContext.java:83) at org.jboss.weld.core@3.1.0.Final-redhat-00001//org.jboss.weld.contexts.AbstractContext.get(AbstractContext.java:114) at org.jboss.weld.core@3.1.0.Final-redhat-00001//org.jboss.weld.contexts.PassivatingContextWrapper$AbstractPassivatingContextWrapper.get(PassivatingContextWrapper.java:78) at org.jboss.weld.core@3.1.0.Final-redhat-00001//org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.getIfExists(ContextualInstanceStrategy.java:89) at org.jboss.weld.core@3.1.0.Final-redhat-00001//org.jboss.weld.bean.ContextualInstanceStrategy$CachingContextualInstanceStrategy.getIfExists(ContextualInstanceStrategy.java:164) at org.jboss.weld.core@3.1.0.Final-redhat-00001//org.jboss.weld.bean.ContextualInstance.getIfExists(ContextualInstance.java:63) at org.jboss.weld.core@3.1.0.Final-redhat-00001//org.jboss.weld.bean.proxy.ContextBeanInstance.getInstance(ContextBeanInstance.java:87) at org.jboss.weld.core@3.1.0.Final-redhat-00001//org.jboss.weld.bean.proxy.ProxyMethodHandler.getInstance(ProxyMethodHandler.java:131) at deployment.clusterbench-ee8.ear.clusterbench-ee8-ejb.jar//org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_WeldClientProxy.getSerialAndIncrement(Unknown Source) at deployment.clusterbench-ee8.ear.clusterbench-ee8-web.war//org.jboss.test.clusterbench.web.ejb.LocalEjbServlet.doGet(LocalEjbServlet.java:41) at javax.servlet.api@1.0.0.Final-redhat-1//javax.servlet.http.HttpServlet.service(HttpServlet.java:686) at javax.servlet.api@1.0.0.Final-redhat-1//javax.servlet.http.HttpServlet.service(HttpServlet.java:791) at io.undertow.servlet@2.0.19.Final-redhat-00001//io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) at io.undertow.servlet@2.0.19.Final-redhat-00001//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) at io.opentracing.contrib.opentracing-jaxrs2@0.1.7.redhat-00001//io.opentracing.contrib.jaxrs2.server.SpanFinishingFilter.doFilter(SpanFinishingFilter.java:55) at io.undertow.servlet@2.0.19.Final-redhat-00001//io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) at io.undertow.servlet@2.0.19.Final-redhat-00001//io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) at io.undertow.servlet@2.0.19.Final-redhat-00001//io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) at io.undertow.servlet@2.0.19.Final-redhat-00001//io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.servlet@2.0.19.Final-redhat-00001//io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) at io.undertow.servlet@2.0.19.Final-redhat-00001//io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow@7.3.0.CD16-redhat-00001//org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.core@2.0.19.Final-redhat-00001//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet@2.0.19.Final-redhat-00001//io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) at io.undertow.servlet@2.0.19.Final-redhat-00001//io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at io.undertow.core@2.0.19.Final-redhat-00001//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.core@2.0.19.Final-redhat-00001//io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet@2.0.19.Final-redhat-00001//io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at io.undertow.core@2.0.19.Final-redhat-00001//io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) at io.undertow.servlet@2.0.19.Final-redhat-00001//io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) at io.undertow.core@2.0.19.Final-redhat-00001//io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) at io.undertow.core@2.0.19.Final-redhat-00001//io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) at io.undertow.core@2.0.19.Final-redhat-00001//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow@7.3.0.CD16-redhat-00001//org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.core@2.0.19.Final-redhat-00001//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow@7.3.0.CD16-redhat-00001//org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68) at io.undertow.core@2.0.19.Final-redhat-00001//io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet@2.0.19.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) at io.undertow.servlet@2.0.19.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) at io.undertow.servlet@2.0.19.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) at io.undertow.servlet@2.0.19.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) at io.undertow.servlet@2.0.19.Final-redhat-00001//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) at io.undertow.servlet@2.0.19.Final-redhat-00001//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) at org.wildfly.extension.undertow@7.3.0.CD16-redhat-00001//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105) at org.wildfly.extension.undertow@7.3.0.CD16-redhat-00001//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) at org.wildfly.extension.undertow@7.3.0.CD16-redhat-00001//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) at org.wildfly.extension.undertow@7.3.0.CD16-redhat-00001//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) at org.wildfly.extension.undertow@7.3.0.CD16-redhat-00001//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502) at io.undertow.servlet@2.0.19.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) at io.undertow.servlet@2.0.19.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) at io.undertow.servlet@2.0.19.Final-redhat-00001//io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) at io.undertow.core@2.0.19.Final-redhat-00001//io.undertow.server.Connectors.executeRootHandler(Connectors.java:364) at io.undertow.core@2.0.19.Final-redhat-00001//io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) at org.jboss.threads@2.3.3.Final-redhat-00001//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) at java.base/java.lang.Thread.run(Thread.java:834)
complete log here