Uploaded image for project: 'Application Server 7'
  1. Application Server 7
  2. AS7-4277

JBPAPP-8740 - SYNC replication results in TimeoutException; Lock held by [GlobalTransaction:<perf21/web>:5473619:remote] (Regression introduced in AS7-4205)

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 7.1.2.Final (EAP)
    • No Release
    • Clustering
    • None

    Description

      After https://github.com/jbossas/jboss-as/pull/1869 we are seeing read time-outs on the clients and TimeoutException on the server.

      [JBossINF] 12:46:55,011 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (OOB-5717,null) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [15 seconds] on key [OKaB26plMWYsWpHQS+UmsJ7R] for requestor [GlobalTransaction:<perf21/web>:3592818:remote]! Lock held by [GlobalTransaction:<perf21/web>:3591864:remote]
      [JBossINF] 	at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:204)
      [JBossINF] 	at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLock(LockManagerImpl.java:178)
      [JBossINF] 	at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLock(LockManagerImpl.java:168)
      [JBossINF] 	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockKeyAndCheckOwnership(AbstractTxLockingInterceptor.java:209)
      [JBossINF] 	at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockAndRegisterBackupLock(AbstractTxLockingInterceptor.java:136)
      [JBossINF] 	at org.infinispan.interceptors.locking.OptimisticLockingInterceptor$LockAcquisitionVisitor.visitPutKeyValueCommand(OptimisticLockingInterceptor.java:202)
      [JBossINF] 	at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)
      [JBossINF] 	at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.acquireLocksVisitingCommands(OptimisticLockingInterceptor.java:262)
      [JBossINF] 	at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPrepareCommand(OptimisticLockingInterceptor.java:92)
      [JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
      [JBossINF] 	at org.infinispan.interceptors.NotificationInterceptor.visitPrepareCommand(NotificationInterceptor.java:58)
      [JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)
      [JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)
      [JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
      [JBossINF] 	at org.infinispan.interceptors.TxInterceptor.visitPrepareCommand(TxInterceptor.java:106)
      [JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
      [JBossINF] 	at org.infinispan.interceptors.StateTransferLockInterceptor.handleWithRetries(StateTransferLockInterceptor.java:208)
      [JBossINF] 	at org.infinispan.interceptors.StateTransferLockInterceptor.visitPrepareCommand(StateTransferLockInterceptor.java:85)
      [JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)
      [JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)
      [JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
      [JBossINF] 	at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:130)
      [JBossINF] 	at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:89)
      [JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)
      [JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)
      [JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)
      [JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)
      [JBossINF] 	at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)
      [JBossINF] 	at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:70)
      [JBossINF] 	at org.infinispan.commands.AbstractVisitor.visitPrepareCommand(AbstractVisitor.java:113)
      [JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.acceptVisitor(PrepareCommand.java:131)
      [JBossINF] 	at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345)
      [JBossINF] 	at org.infinispan.commands.tx.PrepareCommand.perform(PrepareCommand.java:127)
      [JBossINF] 	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:127)
      [JBossINF] 	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:136)
      [JBossINF] 	at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithRetry(InboundInvocationHandlerImpl.java:162)
      [JBossINF] 	at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:114)
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:221)
      [JBossINF] 	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:201)
      [JBossINF] 	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.protocols.RSVP.up(RSVP.java:181) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.protocols.FlowControl.up(FlowControl.java:400) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.protocols.pbcast.GMS.up(GMS.java:882) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:595) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.protocols.BARRIER.up(BARRIER.java:102) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.protocols.Discovery.up(Discovery.java:359) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.stack.Protocol.up(Protocol.java:358) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.protocols.TP.passMessageUp(TP.java:1174) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704) [jgroups-3.0.8.Final.jar:3.0.8.Final]
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_30]
      [JBossINF] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_30]
      [JBossINF] 	at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_30]
      

      Attachments

        Issue Links

          Activity

            People

              pferraro@redhat.com Paul Ferraro
              rhn-engineering-rhusar Radoslav Husar
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: