Uploaded image for project: 'WildFly'
  1. WildFly
  2. WFLY-9958

CacheException: ISPN000482: Cannot create remote transaction GlobalTx:X:Y, already completed

    XMLWordPrintable

Details

    Description

      Seen in scenario: stress-granular-repl-sync

      While the test continues fine for 30 minutes (see graph-throughput.png, number of clients: 4400), after that servers start logging:

      [JBossINF] [0m[33m14:46:11,468 WARN  [org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler] (remote-thread--p6-t109) ISPN000071: Caught exception when handling command PrepareCommand {modifications=[PutKeyValueCommand{key=SessionAttributeKey(ncgjLiWY7FMFf-_Ckh5ono75wTkovkRoyrFFw4er[1]), value=[B@45b6e38f, flags=[FORCE_SYNCHRONOUS], commandInvocationId=CommandInvocation:local:0, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=13}, PutKeyValueCommand{key=SessionAccessMetaDataKey(ncgjLiWY7FMFf-_Ckh5ono75wTkovkRoyrFFw4er), value=org.wildfly.clustering.web.infinispan.session.SimpleSessionAccessMetaData@4b711549, flags=[IGNORE_RETURN_VALUES], commandInvocationId=CommandInvocation:local:0, putIfAbsent=false, valueMatcher=MATCH_ALWAYS, metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, successful=true, topologyId=13}], onePhaseCommit=true, retried=false, gtx=GlobalTx:dev215:17647090, cacheName='clusterbench-ee7.ear.clusterbench-ee7-web-granular.war', topologyId=13}: org.infinispan.commons.CacheException: ISPN000482: Cannot create remote transaction GlobalTx:dev215:17647090, already completed
      [JBossINF] 	at org.infinispan.transaction.impl.TransactionTable.lambda$getOrCreateRemoteTransaction$1(TransactionTable.java:387)
      [JBossINF] 	at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
      [JBossINF] 	at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:380)
      [JBossINF] 	at org.infinispan.transaction.impl.TransactionTable.getOrCreateRemoteTransaction(TransactionTable.java:360)
      [JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.createContext(PrepareCommand.java:120)
      [JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.invokeAsync(PrepareCommand.java:101)
      [JBossINF] 	at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:102)
      [JBossINF] 	at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.invoke(BaseBlockingRunnable.java:99)
      [JBossINF] 	at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.runAsync(BaseBlockingRunnable.java:71)
      [JBossINF] 	at org.infinispan.remoting.inboundhandler.BaseBlockingRunnable.run(BaseBlockingRunnable.java:40)
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      [JBossINF] 	at org.wildfly.clustering.service.concurrent.ClassLoaderThreadFactory.lambda$newThread$0(ClassLoaderThreadFactory.java:47)
      [JBossINF] 	at java.lang.Thread.run(Thread.java:748)
      

      Clients start logging Read timeout:

      2018/03/04 14:45:56:846 EST [WARN ][Runner - 21] HOST dev220.mw.lab.eng.bos.redhat.com:rootProcess:c - Error sampling data:  <org.jboss.smartfrog.loaddriver.RequestProcessingException: IO error: java.net.SocketTimeoutException: Read timed out>
              
      2018/03/04 14:45:56:846 EST [WARN ][Runner - 21] SFCORE_LOG - Error sampling data:  <org.jboss.smartfrog.loaddriver.RequestProcessingException: IO error: java.net.SocketTimeoutException: Read timed out>
              org.jboss.smartfrog.loaddriver.RequestProcessingException: IO error: java.net.SocketTimeoutException: Read timed out
      	at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:266)
      	at org.jboss.smartfrog.loaddriver.CompoundRequestProcessorFactoryImpl$CompoundRequestProcessor.processRequest(CompoundRequestProcessorFactoryImpl.java:52)
      	at org.jboss.smartfrog.loaddriver.Runner.run(Runner.java:103)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: java.net.SocketTimeoutException: Read timed out
      	at java.net.SocketInputStream.socketRead0(Native Method)
      	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      	at java.net.SocketInputStream.read(SocketInputStream.java:171)
      	at java.net.SocketInputStream.read(SocketInputStream.java:141)
      	at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
      	at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
      	at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
      	at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
      	at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
      	at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
      	at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
      	at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
      	at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
      	at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
      	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
      	at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
      	at org.jboss.smartfrog.loaddriver.http.HttpRequestProcessorFactoryImpl$HttpRequestProcessor.processRequest(HttpRequestProcessorFactoryImpl.java:147)
      	... 3 more
      

      The test recovers after some time and then the issue occurs again in the cycles.

      Links
      client: http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-stress-granular-repl-sync/42/console-dev221/
      server: http://jenkins.hosts.mwqe.eng.bos.redhat.com/hudson/job/eap-7x-stress-granular-repl-sync/42/console-dev212/

      Attachments

        Activity

          People

            pferraro@redhat.com Paul Ferraro
            mvinkler1@redhat.com Michal Vinkler
            Tommaso Borgato Tommaso Borgato
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: