Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-9087

Timeout during put operation when a node is blocked

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 9.3.0.CR1
    • None
    • None
    • None
    • Sprint 9.3.0.Beta1, Sprint 9.3.0.CR1
    • Hide

      Create a cluster with 2 nodes.
      Start putting data into the nodeA
      Stop the nodeB ( kill -STOP $PID )
      nodeA start returning timeout during the put operation

      Show
      Create a cluster with 2 nodes. Start putting data into the nodeA Stop the nodeB ( kill -STOP $PID ) nodeA start returning timeout during the put operation
    • Hide

      Change the protocol to:

         <FD_ALL timeout="10000"
           interval="2000"
           timeout_check_interval="2000"
   />
      Show
      Change the protocol to: <FD_ALL timeout= "10000" interval= "2000" timeout_check_interval= "2000" />

    Description

          2018-04-17 13:30:02.782 ERROR 14932 --- [timeout-thread--p3-t1] o.i.i.impl.InvocationContextInterceptor  : ISPN000136: Error executing command PutKeyValueCommand, writing keys [5db796a3-3f65-468a-b86a-6d5ef8b4b330]
     
    org.infinispan.util.concurrent.TimeoutException: ISPN000427: Timeout after 15 seconds waiting for acks. Id=100000
            at org.infinispan.util.concurrent.CommandAckCollector.createTimeoutException(CommandAckCollector.java:188) ~[infinispan-embedded-8.5.0.Final-redhat-6.jar:8.5.0.Final-redhat-6]
            at org.infinispan.util.concurrent.CommandAckCollector.access$300(CommandAckCollector.java:51) ~[infinispan-embedded-8.5.0.Final-redhat-6.jar:8.5.0.Final-redhat-6]
            at org.infinispan.util.concurrent.CommandAckCollector$BaseCollector.call(CommandAckCollector.java:214) [infinispan-embedded-8.5.0.Final-redhat-6.jar:8.5.0.Final-redhat-6]
            at org.infinispan.util.concurrent.CommandAckCollector$BaseCollector.call(CommandAckCollector.java:191) [infinispan-embedded-8.5.0.Final-redhat-6.jar:8.5.0.Final-redhat-6]
            at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_161]
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_161]
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.8.0_161]
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161]
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161]
            at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]

      After some investigation together with dberinde@redhat.com, we found that the FD_ALL is just too slow.

             <FD_ALL timeout="60000"
               interval="15000"
               timeout_check_interval="5000"
       />

      Attachments

        Activity

          People

            dberinde@redhat.com Dan Berindei (Inactive)
            dlovison@redhat.com Diego Lovison
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: