Details
-
Bug
-
Resolution: Won't Do
-
Major
-
None
-
7.1.0.DR18
Description
We saw this error on server during failover scenario ejb-ejbservlet-undeploy-dist-sync.
The exception was logged after new cluster view was received.
There are 8 occurences on two of the four nodes.
variant 1:
09:38:53,524 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-2) ISPN000094: Received new cluster view for channel hibernate: [perf20|4] (3) [perf20, perf21, perf19] 09:39:07,626 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-113) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: Replication timeout [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:639) [JBossINF] at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) [JBossINF] at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) [JBossINF] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) [JBossINF] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) [JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:61) [JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [JBossINF] at java.lang.Thread.run(Thread.java:745)
followed by one occurence of the second variant of the exception:
09:39:17,643 ERROR [io.undertow.request] (default task-113) UT005071: Undertow request failed HttpServerExchange{ GET /clusterbench/ejbservlet request {Cookie=[$Version=0; JSESSIONID=aZheHLnU4YKkDqAKFdDob2NgZ1xZyP70Ysbt4uCD.perf21; $Path=/clusterbench], Content-Length=[0], User-Agent=[Jakarta Commons-HttpClient/3.1], Host=[perf17:8080]} response {Set-Cookie=[JSESSIONID=aZheHLnU4YKkDqAKFdDob2NgZ1xZyP70Ysbt4uCD.perf20; path=/clusterbench], Content-Type=[text/html;charset=UTF-8], Content-Length=[80], Date=[Fri, 12 May 2017 13:39:07 GMT]}}: org.infinispan.util.concurrent.TimeoutException: Replication timeout [JBossINF] at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$1(JGroupsTransport.java:639) [JBossINF] at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602) [JBossINF] at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) [JBossINF] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) [JBossINF] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962) [JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:61) [JBossINF] at org.infinispan.remoting.transport.jgroups.RspListFuture.call(RspListFuture.java:16) [JBossINF] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [JBossINF] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [JBossINF] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [JBossINF] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [JBossINF] at java.lang.Thread.run(Thread.java:745)
There are two WARN messages logged at the same time on client:
Client log stacktrace:
2017/05/12 09:39:07:634 EDT [WARN ][Runner - 1453] HOST perf17.mw.lab.eng.bos.redhat.com:rootProcess:c - Error sampling data: <org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>>
org.jboss.smartfrog.loaddriver.RequestProcessingException: Invalid response code: 500 Content: <html><head><title>Error</title></head><body>Internal Server Error</body></html>
at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:164)
at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
at java.lang.Thread.run(Thread.java:745)
Link to server log:
http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failover-ejb-ejbservlet-undeploy-dist-sync/15/console-perf21/
Link to client log:
http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/perflab_eap-7x-failover-ejb-ejbservlet-undeploy-dist-sync/15/console-perf17/