Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-3954

EJB Invocation failed on component LocalStatefulSB: TimeoutException: Replication timeout for X

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Verified (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 7.0.0.ER7
    • Fix Version/s: 7.1.0.DR6
    • Component/s: Clustering
    • Labels:
      None
    • Target Release:
    • Affects:
      Release Notes
    • Release Notes Docs Status:
      Documented as Known Issue

      Description

      Seen in ejbservlet scenarios, no matter what failover type (jvmkill/shutdown/undeploy) is used, with replicated cache used (both SYNC and ASYNC replication).

      With no obvious connection to a failover of any node, all the servers logged EJB Invocation failed on component LocalStatefulSB: TimeoutException: Replication timeout for X:

      [JBossINF] [0m[31m11:20:36,590 ERROR [org.jboss.as.ejb3.invocation] (default task-31) WFLYEJB0034: EJB Invocation failed on component LocalStatefulSB for method public int org.jboss.test.clusterbench.common.ejb.CommonStatefulSBImpl.getSerialAndIncrement(): javax.ejb.EJBException: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf20
      		 
      [JBossINF] 	at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleExceptionInOurTx(CMTTxInterceptor.java:187)
      		 
      [JBossINF] 	at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
      		 
      [JBossINF] 	at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:327)
      		 
      [JBossINF] 	at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:239)
      		 
      [JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      		 
      [JBossINF] 	at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
      		 
      [JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      		 
      [JBossINF] 	at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
      		 
      [JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      		 
      [JBossINF] 	at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100)
      		 
      [JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      		 
      [JBossINF] 	at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64)
      		 
      [JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      		 
      [JBossINF] 	at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:66)
      		 
      [JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      		 
      [JBossINF] 	at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50)
      		 
      [JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      		 
      [JBossINF] 	at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:54)
      		 
      [JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      		 
      [JBossINF] 	at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
      		 
      [JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      		 
      [JBossINF] 	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
      		 
      [JBossINF] 	at org.wildfly.security.manager.WildFlySecurityManager.doChecked(WildFlySecurityManager.java:636)
      		 
      [JBossINF] 	at org.jboss.invocation.AccessCheckingInterceptor.processInvocation(AccessCheckingInterceptor.java:61)
      		 
      [JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      		 
      [JBossINF] 	at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
      		 
      [JBossINF] 	at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
      		 
      [JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      		 
      [JBossINF] 	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
      		 
      [JBossINF] 	at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:195)
      		 
      [JBossINF] 	at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:185)
      		 
      [JBossINF] 	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
      		 
      [JBossINF] 	at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
      		 
      [JBossINF] 	at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:73)
      		 
      [JBossINF] 	at org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$$$view3.getSerialAndIncrement(Unknown Source)
      		 
      [JBossINF] 	at sun.reflect.GeneratedMethodAccessor35.invoke(Unknown Source)
      		 
      [JBossINF] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      		 
      [JBossINF] 	at java.lang.reflect.Method.invoke(Method.java:497)
      		 
      [JBossINF] 	at org.jboss.weld.util.reflection.Reflections.invokeAndUnwrap(Reflections.java:433)
      		 
      [JBossINF] 	at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.invoke(EnterpriseBeanProxyMethodHandler.java:128)
      		 
      [JBossINF] 	at org.jboss.weld.bean.proxy.EnterpriseTargetBeanInstance.invoke(EnterpriseTargetBeanInstance.java:56)
      		 
      [JBossINF] 	at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:100)
      		 
      [JBossINF] 	at org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_Weld$EnterpriseProxy$.getSerialAndIncrement(Unknown Source)
      		 
      [JBossINF] 	at org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_WeldClientProxy.getSerialAndIncrement(Unknown Source)
      		 
      [JBossINF] 	at org.jboss.test.clusterbench.web.ejb.LocalEjbServlet.doGet(LocalEjbServlet.java:38)
      		 
      [JBossINF] 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
      		 
      [JBossINF] 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
      		 
      [JBossINF] 	at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
      		 
      [JBossINF] 	at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
      		 
      [JBossINF] 	at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
      		 
      [JBossINF] 	at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
      		 
      [JBossINF] 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      		 
      [JBossINF] 	at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
      		 
      [JBossINF] 	at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
      		 
      [JBossINF] 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      		 
      [JBossINF] 	at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
      		 
      [JBossINF] 	at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
      		 
      [JBossINF] 	at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
      		 
      [JBossINF] 	at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
      		 
      [JBossINF] 	at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
      		 
      [JBossINF] 	at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
      		 
      [JBossINF] 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      		 
      [JBossINF] 	at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
      		 
      [JBossINF] 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      		 
      [JBossINF] 	at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
      		 
      [JBossINF] 	at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:285)
      		 
      [JBossINF] 	at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:264)
      		 
      [JBossINF] 	at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
      		 
      [JBossINF] 	at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:175)
      		 
      [JBossINF] 	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
      		 
      [JBossINF] 	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:792)
      		 
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      		 
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      		 
      [JBossINF] 	at java.lang.Thread.run(Thread.java:745)
      		 
      [JBossINF] Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout for perf20
      		 
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:765)
      		 
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$175(JGroupsTransport.java:612)
      		 
      [JBossINF] 	at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
      		 
      [JBossINF] 	at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
      		 
      [JBossINF] 	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
      		 
      [JBossINF] 	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
      		 
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:47)
      		 
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16)
      		 
      [JBossINF] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      		 
      [JBossINF] 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      		 
      [JBossINF] 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      		 
      [JBossINF] 	... 3 more

      The number of occurrences is ~500 in this test run, each occurrence causes client getting 500:

      2016/03/22 11:20:36:599 EDT [WARN ][Runner - 524] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Error sampling data:  <org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>>
      		 
              org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>
      		 
      	at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:163)
      		 
      	at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
      		 
      	at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
      		 
      	at java.lang.Thread.run(Thread.java:745)

      Server link:
      http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-ejbservlet-shutdown-repl-sync/20/console-perf21/

      Client link:
      http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-failover-ejb-ejbservlet-shutdown-repl-sync/20/console-perf17/

        Gliffy Diagrams

          Attachments

            Issue Links

            cloned to

            Bug - A problem which impairs or prevents the functions of the product. WFLY-6428 EJB Invocation failed on component LocalStatefulSB: TimeoutException: Replication timeout for X

            • Major - A request that should be considered seriously but is not a show stopper.
            • Open

              Activity

                People

                • Assignee:
                  pferraro Paul Ferraro
                  Reporter:
                  mvinkler Michal Vinkler
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: