Uploaded image for project: 'WildFly Core'
  1. WildFly Core
  2. WFCORE-756

Core management server shutting down a thread pool early, resulting in RejectedExecutionException

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Critical
    • 2.0.0.CR5
    • None
    • Management
    • None

    Description

      In one of our tests, we've seen this exception during server shutdown:

      2015-06-05 10:34:44,387 ERROR [org.xnio.listener] (XNIO-1 I/O-2) XNIO001007: A channel event listener threw an exception: java.util.concurrent.RejectedExecutionException: Task org.jboss.remoting3.remote.RemoteReadListener$1$1@5e7925db rejected from org.xnio.XnioWorker$TaskPool@3d00aa37[Shutting down, pool size = 7, active threads = 0, queued tasks = 0, completed tasks = 52]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
	at org.xnio.XnioWorker.execute(XnioWorker.java:741)
	at org.jboss.remoting3.remote.RemoteReadListener$1.handleEvent(RemoteReadListener.java:54)
	...

      This is a very basic test that only starts the server and then shuts it down using :shutdown.

      After looking into this for a while, I believe that this is caused by the core management server (UndertowHttpManagementService) shutting down an XNIO worker (= thread pool) while the :shutdown management operation is still running (or, in fact, finishing, trying to close the network connection).

      I have a Byteman-based reproducer that inserts artifical pauses to certain well-defined places. I'm not sure if this has some connection to the graceful shutdown system, but I believe that even if it does, something like this shouldn't happen.

      Steps to reproduce:

      1. ./bin/standalone.sh -c standalone-full-ha.xml and wait until it starts completely
      2. jps -v | grep "\-D\[Standalone\]" to figure out the PID of the newly started server
      3. bminstall.sh -b -Dorg.jboss.byteman.transform.all $PID
      4. bmsubmit.sh reproducer.btm, where reproducer.btm is a Byteman script reproduced below
      5. ./bin/jboss-cli.sh -c
      6. :read-resource repeat few times
      7. :shutdown(timeout=1) (or plain :shutdown)

      The Byteman script:

      RULE XnioWorker.TaskPool/ThreadPoolExecutor shutdown
CLASS java.util.concurrent.ThreadPoolExecutor
METHOD shutdown()
AFTER INVOKE advanceRunState
IF TRUE
DO Thread.sleep(10000)
ENDRULE

RULE Remoting onClose handler
CLASS org.jboss.remoting3.remote.RemoteReadListener$1
METHOD handleEvent(java.nio.channels.Channel)
AT ENTRY
IF TRUE
DO Thread.sleep(5000)
ENDRULE

      Attachments

        1. server.log
          314 kB
        2. server.log
          304 kB

        Issue Links

          clones

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-307 Core management server shutting down a thread pool early, resulting in RejectedExecutionException

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          is duplicated by

          Bug - A problem that impairs or prevents the functions of the product. WFLY-4852 (XNIO-2 I/O-1) XNIO001007 java.util.concurrent.RejectedExecutionException error shown intermittently during server shutdown

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          relates to

          Bug - A problem that impairs or prevents the functions of the product. JBEAP-380 (XNIO-2 I/O-1) XNIO001007 java.util.concurrent.RejectedExecutionException error shown intermittently during server shutdown

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              ehugonne1@redhat.com Emmanuel Hugonnet
              lthon@redhat.com Ladislav Thon
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: