Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-6428

EJB Invocation failed on component LocalStatefulSB: TimeoutException: Replication timeout for X

    XMLWordPrintable

Details

    • Bug
    • Resolution: Obsolete
    • Major
    • None
    • 10.0.0.Final
    • Clustering
    • None

    Description

      Seen in ejbservlet scenarios, no matter what failover type (jvmkill/shutdown/undeploy) is used, with replicated cache used (both SYNC and ASYNC replication).

      With no obvious connection to a failover of any node, all the servers logged EJB Invocation failed on component LocalStatefulSB: TimeoutException: Replication timeout for X:

      [JBossINF] [0m[31m11:20:36,590 ERROR [org.jboss.as.ejb3.invocation] (default task-31) WFLYEJB0034: EJB Invocation failed on component LocalStatefulSB for method public int org.jboss.test.clusterbench.common.ejb.CommonStatefulSBImpl.getSerialAndIncrement(): javax.ejb.EJBException: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf20
[JBossINF] 	at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
[JBossINF] 	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
[JBossINF] 	at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
[JBossINF] 	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
[JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
[JBossINF] 	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
[JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
[JBossINF] 	at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
[JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
[JBossINF] 	at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
[JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
[JBossINF] 	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
[JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
[JBossINF] 	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
[JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
[JBossINF] 	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
[JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
[JBossINF] 	at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
[JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
[JBossINF] 	at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
[JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
[JBossINF] 	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
[JBossINF] 	at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
[JBossINF] 	at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
[JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
[JBossINF] 	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
[JBossINF] 	at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
[JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
[JBossINF] 	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
[JBossINF] 	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
[JBossINF] 	at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
[JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
[JBossINF] 	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
[JBossINF] 	at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73)
[JBossINF] 	at org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$$$view3.getSerialAndIncrement(Unknown Source)
[JBossINF] 	at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
[JBossINF] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[JBossINF] 	at java.lang.reflect.Method.invoke(Method.java:497)
[JBossINF] 	at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:433)
[JBossINF] 	at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:128)
[JBossINF] 	at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
[JBossINF] 	at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100)
[JBossINF] 	at org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_Weld$EnterpriseProxy$.getSerialAndIncrement(Unknown Source)
[JBossINF] 	at org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_WeldClientProxy.getSerialAndIncrement(Unknown Source)
[JBossINF] 	at org.jboss.test.clusterbench.web.ejb.LocalEjbServlet.doGet(LocalEjbServlet.java:38)
[JBossINF] 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
[JBossINF] 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
[JBossINF] 	at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
[JBossINF] 	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
[JBossINF] 	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
[JBossINF] 	at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
[JBossINF] 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
[JBossINF] 	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
[JBossINF] 	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
[JBossINF] 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
[JBossINF] 	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
[JBossINF] 	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
[JBossINF] 	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
[JBossINF] 	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
[JBossINF] 	at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
[JBossINF] 	at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
[JBossINF] 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
[JBossINF] 	at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
[JBossINF] 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
[JBossINF] 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
[JBossINF] 	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:285)
[JBossINF] 	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:264)
[JBossINF] 	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
[JBossINF] 	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:175)
[JBossINF] 	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
[JBossINF] 	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:792)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[JBossINF] 	at java.lang.Thread.run(Thread.java:745)
[JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf20
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$175(JGroupsTransport.java:612)
[JBossINF] 	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
[JBossINF] 	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
[JBossINF] 	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
[JBossINF] 	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
[JBossINF] 	at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
[JBossINF] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[JBossINF] 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
[JBossINF] 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
[JBossINF] 	... 3 more

      The number of occurrences is ~500 in this test run, each error causes client getting 500:

      Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>>
        org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>
	at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:163)
	at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
	at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
	at java.lang.Thread.run(Thread.java:745)

      Server link:
      http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-ejbservlet-shutdown-repl-sync/20/console-perf21/

      Client link:
      http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-ejbservlet-shutdown-repl-sync/20/console-perf17/

      Attachments

        Issue Links

          clones

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-3954 EJB Invocation failed on component LocalStatefulSB: TimeoutException: Replication timeout for X

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Verified

          Activity

            People

              pferraro@redhat.com Paul Ferraro
              mvinkler1@redhat.com Michal Vinkler
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: