Uploaded image for project: 'JGroups'
  1. JGroups
  2. JGRP-690

Thread interrupt status is not always cleared by default

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 2.6.2, 2.7
    • 2.6, 2.7
    • None

      If Thread#interrupt() is invoked on a thread that is blocked in an invocation of the wait(), wait(long), or wait(long, int) methods of the Object class, or of the join(), join(long), join(long, int), sleep(long), or sleep(long, int), methods of this class, then its interrupt status will be cleared and it will receive an InterruptedException. (citing Thread#interrupt javadoc)

      Conventional wisdom is that interrupt signal was always cleared. However, in cases except the above interrupt signal is actually not cleared! JGroups code that is throwing InterruptedException should make sure that it also clears the interrupt status. Otherwise, if we wrongly assume that interrupt status is always cleared, we can have inconsistencies that can lead to hard-to-trace bugs (See below). We should verify that throughout our code that we always clear interrupt status.

      Gray said on javagroups-list:
      We have been seeing a large percentage of our production servers (on 2.6.1)
      spinning above 100% of CPU. After some investigation, we seem to have found
      an infinite loop in Scheduler.java. If sched_thread gets interrupted, its
      interrupt flag never gets reset so it spins forever suspending and resuming
      the current_thread and throwing and catching InterruptedException(s).
      According to jconsole, one of our boxes has done this 690 million times –
      nice effort. When this is happening, the memory space is alternating
      between 200 and 300 mb which makes sense given the exception bandwidth.

      We are running our RPC dispatchers wth deadlock_detection and
      concurrent_processing set to true and use_concurrent_stack="true".

      Here are http://www.nabble.com/file/p15395455/stack_dumps.txt some stack
      snapshots of the spinning thread . As you can see they are all in
      Scheduler.run. Here is the section from
      http://www.nabble.com/file/p15395455/Scheduler.java.txt Scheduler.java with
      the offending part of the code including where sched_thread is being
      interrupted by a call to Scheduler.addPrio(). The pluses show where we
      think a missing Thread.interrupt() call should be.

      Here's the http://www.nabble.com/file/p15395455/Scheduler.java.patch.txt
      patch for Scheduler.java (from 2.6.1) which adds in the interrupt call.

      This is all with Jgroups 2.6.1. Anyone else see this behavior? We're going
      to roll this patch out to see if it stops the spinning.

      gray

            vblagoje Vladimir Blagojevic (Inactive)
            vblagoje Vladimir Blagojevic (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Created:
              Updated:
              Resolved: