Uploaded image for project: 'JBoss ESB'
  1. JBoss ESB
  2. JBESB-3473

Message Loss or premature execution termination in an action during shutdown

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: 4.9
    • Fix Version/s: 4.9 CP1
    • Component/s: Rosetta
    • Labels:
      None
    • Environment:

      RHEL 64bit

    • Steps to Reproduce:
      Hide

      Enqueued several thousand mesages into an ESB application. Make sure the action takes time. I simulated 400 mSec via sleep. While the messages were being processed I toggle(stop/start) lifecycle state for the jms listeners via JMX in order to simulate shutdown.

      Inspect logs, and view final message tally. If you see an error/warning which looks like the following, you have reproduced the issue.[org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle] (Thread-200) Unexpected error from doRun()
      java.util.concurrent.RejectedExecutionException
      at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecutionThreadPoolExecutor.java:1768)
      at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
      at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
      at org.jboss.soa.esb.listeners.message.MessageAwareListener.waitForEventAndProcess(MessageAwareListener.java:359)
      at org.jboss.soa.esb.listeners.message.MessageAwareListener.doRun(MessageAwareListener.java:253)
      at org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle.run(AbstractThreadedManagedLifecycle.java:115)
      at java.lang.Thread.run(Thread.java:619)

      Show
      Enqueued several thousand mesages into an ESB application. Make sure the action takes time. I simulated 400 mSec via sleep. While the messages were being processed I toggle(stop/start) lifecycle state for the jms listeners via JMX in order to simulate shutdown. Inspect logs, and view final message tally. If you see an error/warning which looks like the following, you have reproduced the issue. [org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle] (Thread-200) Unexpected error from doRun() java.util.concurrent.RejectedExecutionException at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecutionThreadPoolExecutor.java:1768) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658) at org.jboss.soa.esb.listeners.message.MessageAwareListener.waitForEventAndProcess(MessageAwareListener.java:359) at org.jboss.soa.esb.listeners.message.MessageAwareListener.doRun(MessageAwareListener.java:253) at org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle.run(AbstractThreadedManagedLifecycle.java:115) at java.lang.Thread.run(Thread.java:619)
    • Estimated Difficulty:
      Low

      Description

      I'm able to reproduce JMS message loss or premature action termination during shutdown. I enqueued several thousand mesages into an ESB application. While the messages were being processed I toggle (stop/start) lifecycle state for the acting jms listeners via JMX in order to simulate shutdown. I noticed periodic warnings in the serverlog (listing 1). The final tally indicated message loss. Via code inspection the bug appears to be self-evident.

      Basically when the doStop method (JMXConsole) is called it sets state to STOPPING and immediately terminates the executor thread (MessageAwareListener.java). If the thread was currently processing bad things happen.

      I'm using transcated queues and it appears the JMS message transaction evaporates and the message is not returned to the queue. I added some code to AbstractThreadedManagedLifecycle.java to remedy the issue. It uses the default terminationPeriod (60 seconds), unless it is specified in the jboss-esb.xml file.

      AbstractThreadedManagedLifecycle.java
      ---------------------------------------
      protected void doStop()
      throws ManagedLifecycleException
      {
      runningLock.lock() ;
      try
      {
      if (isRunning())

      { setRunning(ManagedLifecycleThreadStatae.STOPPING) ; }

      // Dave Siracusa -start
      if (!waitUntilStopped())

      { throw new ManagedLifecycleException("Thread still active!") ; }

      // Dave Siracusa - end
      }
      finally

      { runningLock.unlock() ; }

      }
      MessageAwareListener.java
      ---------------------------
      protected void doStop()
      throws ManagedLifecycleException

      { super.doStop(); _execService.shutdown() ; }

      Listing 1
      ----------
      2010-09-01 15:04:40,924 WARN [org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle] (Thread-200) Unexpected error from doRun()
      java.util.concurrent.RejectedExecutionException
      at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecutionThreadPoolExecutor.java:1768)
      at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
      at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
      at org.jboss.soa.esb.listeners.message.MessageAwareListener.waitForEventAndProcess(MessageAwareListener.java:359)
      at org.jboss.soa.esb.listeners.message.MessageAwareListener.doRun(MessageAwareListener.java:253)
      at org.jboss.soa.esb.listeners.lifecycle.AbstractThreadedManagedLifecycle.run(AbstractThreadedManagedLifecycle.java:115)
      at java.lang.Thread.run(Thread.java:619)

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  kconner Kevin Conner
                  Reporter:
                  davesiracusa Dave Siracusa
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  0 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: