Details

      Description

      When the application server is restarted out from under a live running listener (via the Standalonebootstrapper) it does appear to work once the app server is brought back to life. However, it produces a long stacktrace. Even after it redeploys the services (even though it shouldn't hav redeployed the servcies because their config file was untouched) it continues to complain and throw up errors/warnings until freshly restarted.

      [java] 19:15:14,465 WARN [Connection Monitor Thread][Connection] Connectio
      n failure, use javax.jms.Connection.setExceptionListener() to handle this error
      and reconnect
      [java] org.jboss.mq.SpyJMSException: No pong received; - nested throwable:
      (java.io.IOException: ping timeout.)
      [java] at org.jboss.mq.Connection$PingTask.run(Connection.java:1277)
      [java] at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(Cloc
      kDaemon.java:364)
      [java] at java.lang.Thread.run(Thread.java:595)
      [java] Caused by: java.io.IOException: ping timeout.
      [java] ... 3 more
      [java] 19:15:18,027 WARN [Connection Monitor Thread][Connection] Connectio
      n failure, use javax.jms.Connection.setExceptionListener() to handle this error
      and reconnect
      [java] org.jboss.mq.SpyJMSException: No pong received; - nested throwable:
      (java.io.IOException: ping timeout.)
      [java] at org.jboss.mq.Connection$PingTask.run(Connection.java:1277)
      [java] at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(Cloc
      kDaemon.java:364)
      [java] at java.lang.Thread.run(Thread.java:595)
      [java] Caused by: java.io.IOException: ping timeout.
      [java] ... 3 more
      [java] 19:15:18,027 WARN [Connection Monitor Thread][Connection] Connectio
      n failure, use javax.jms.Connection.setExceptionListener() to handle this error
      and reconnect
      [java] org.jboss.mq.SpyJMSException: No pong received; - nested throwable:
      (java.io.IOException: ping timeout.)
      [java] at org.jboss.mq.Connection$PingTask.run(Connection.java:1277)
      [java] at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(Cloc
      kDaemon.java:364)
      [java] at java.lang.Thread.run(Thread.java:595)
      [java] Caused by: java.io.IOException: ping timeout.
      [java] ... 3 more
      [java] 19:15:18,183 WARN [Connection Monitor Thread][Connection] Connectio
      n failure, use javax.jms.Connection.setExceptionListener() to handle this error
      and reconnect
      [java] org.jboss.mq.SpyJMSException: No pong received; - nested throwable:
      (java.io.IOException: ping timeout.)
      [java] at org.jboss.mq.Connection$PingTask.run(Connection.java:1277)
      [java] at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(Cloc
      kDaemon.java:364)
      [java] at java.lang.Thread.run(Thread.java:595)
      [java] Caused by: java.io.IOException: ping timeout.
      [java] ... 3 more
      [java] 19:15:18,308 WARN [Connection Monitor Thread][Connection] Connectio
      n failure, use javax.jms.Connection.setExceptionListener() to handle this error
      and reconnect
      [java] org.jboss.mq.SpyJMSException: No pong received; - nested throwable:
      (java.io.IOException: ping timeout.)
      [java] at org.jboss.mq.Connection$PingTask.run(Connection.java:1277)
      [java] at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(Cloc
      kDaemon.java:364)
      [java] at java.lang.Thread.run(Thread.java:595)
      [java] Caused by: java.io.IOException: ping timeout.
      [java] ... 3 more
      [java] 19:15:20,043 WARN [Connection Monitor Thread][Connection] Connectio
      n failure, use javax.jms.Connection.setExceptionListener() to handle this error
      and reconnect
      [java] org.jboss.mq.SpyJMSException: No pong received; - nested throwable:
      (java.io.IOException: ping timeout.)
      [java] at org.jboss.mq.Connection$PingTask.run(Connection.java:1277)
      [java] at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(Cloc

        Gliffy Diagrams

          Activity

          Hide
          estebanschifman Esteban Schifman added a comment -

          Fix was introduced to JmsCourier so it will attempt to reconnect whenever a JMSException is caught at 'deliver' or 'pickup' time
          Local tests pass. Burr to try and reproduce the problem he reported to verify if code change fixed the bug.

          Show
          estebanschifman Esteban Schifman added a comment - Fix was introduced to JmsCourier so it will attempt to reconnect whenever a JMSException is caught at 'deliver' or 'pickup' time Local tests pass. Burr to try and reproduce the problem he reported to verify if code change fixed the bug.
          Hide
          burrsutter Burr Sutter added a comment -

          The reconnect does appear to work. However, it continues to cough up error/warn messages to the console. Tested with the helloworld quickstart. The following are some of the messages:

          [java] 15:12:07,750 WARN [Connection Monitor Thread][Connection] Connectio
          n failure, use javax.jms.Connection.setExceptionListener() to handle this error
          and reconnect
          [java] org.jboss.mq.SpyJMSException: No pong received; - nested throwable:
          (java.io.IOException: ping timeout.)
          [java] at org.jboss.mq.Connection$PingTask.run(Connection.java:1277)
          [java] at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(Cloc
          kDaemon.java:364)
          [java] at java.lang.Thread.run(Thread.java:595)
          [java] Caused by: java.io.IOException: ping timeout.
          [java] ... 3 more
          [java] 15:12:29,750 WARN [Connection Monitor Thread][Connection] Connectio
          n failure, use javax.jms.Connection.setExceptionListener() to handle this error
          and reconnect
          [java] org.jboss.mq.SpyJMSException: No pong received; - nested throwable:
          (java.io.IOException: ping timeout.)
          [java] at org.jboss.mq.Connection$PingTask.run(Connection.java:1277)
          [java] at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(Cloc
          kDaemon.java:364)
          [java] at java.lang.Thread.run(Thread.java:595)
          [java] Caused by: java.io.IOException: ping timeout.
          [java] ... 3 more
          [java] 15:12:37,609 ERROR [ExceptionListener Connection@13402014[token=Conn
          ectionToken:ID:2/18dabc2b6fedea2925c4eeb4d19afbff rcvstate=STOPPED]][JmsCourier]
          JMS error. Attempting JMS reconnect.
          [java] org.jboss.mq.SpyJMSException: No pong received; - nested throwable:
          (java.io.IOException: ping timeout.)
          [java] at org.jboss.mq.Connection$PingTask.run(Connection.java:1277)
          [java] at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(Cloc
          kDaemon.java:364)
          [java] at java.lang.Thread.run(Thread.java:595)
          [java] Caused by: java.io.IOException: ping timeout.
          [java] ... 3 more

          Show
          burrsutter Burr Sutter added a comment - The reconnect does appear to work. However, it continues to cough up error/warn messages to the console. Tested with the helloworld quickstart. The following are some of the messages: [java] 15:12:07,750 WARN [Connection Monitor Thread] [Connection] Connectio n failure, use javax.jms.Connection.setExceptionListener() to handle this error and reconnect [java] org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.) [java] at org.jboss.mq.Connection$PingTask.run(Connection.java:1277) [java] at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(Cloc kDaemon.java:364) [java] at java.lang.Thread.run(Thread.java:595) [java] Caused by: java.io.IOException: ping timeout. [java] ... 3 more [java] 15:12:29,750 WARN [Connection Monitor Thread] [Connection] Connectio n failure, use javax.jms.Connection.setExceptionListener() to handle this error and reconnect [java] org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.) [java] at org.jboss.mq.Connection$PingTask.run(Connection.java:1277) [java] at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(Cloc kDaemon.java:364) [java] at java.lang.Thread.run(Thread.java:595) [java] Caused by: java.io.IOException: ping timeout. [java] ... 3 more [java] 15:12:37,609 ERROR [ExceptionListener Connection@13402014[token=Conn ectionToken:ID:2/18dabc2b6fedea2925c4eeb4d19afbff rcvstate=STOPPED]] [JmsCourier] JMS error. Attempting JMS reconnect. [java] org.jboss.mq.SpyJMSException: No pong received; - nested throwable: (java.io.IOException: ping timeout.) [java] at org.jboss.mq.Connection$PingTask.run(Connection.java:1277) [java] at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(Cloc kDaemon.java:364) [java] at java.lang.Thread.run(Thread.java:595) [java] Caused by: java.io.IOException: ping timeout. [java] ... 3 more
          Hide
          marklittle Mark Little added a comment -

          I have added this to the release notes, so we can move this to the MP1 release for final fix.

          Show
          marklittle Mark Little added a comment - I have added this to the release notes, so we can move this to the MP1 release for final fix.
          Hide
          kconner Kevin Conner added a comment -

          The code has changes somewhat since this was reported.

          The two main issues in the current codebase were the following

          • no exception listener in the connection pool, hence reconnect errors
          • exception logging via high levels, including when propagating or ignoring the exception

          The exception logging has been changed to log at DEBUG with appropriate additional messages (at higher levels) where appropriate. The log4j.xml has also been changed to restrict the CONSOLE output to INFO.

          Executing this in standalone mode no longer displays all the stacktraces.

          Show
          kconner Kevin Conner added a comment - The code has changes somewhat since this was reported. The two main issues in the current codebase were the following no exception listener in the connection pool, hence reconnect errors exception logging via high levels, including when propagating or ignoring the exception The exception logging has been changed to log at DEBUG with appropriate additional messages (at higher levels) where appropriate. The log4j.xml has also been changed to restrict the CONSOLE output to INFO. Executing this in standalone mode no longer displays all the stacktraces.
          Hide
          kconner Kevin Conner added a comment -

          Forgot to mention that it was fixed in revision 11152

          Show
          kconner Kevin Conner added a comment - Forgot to mention that it was fixed in revision 11152

            People

            • Assignee:
              kconner Kevin Conner
              Reporter:
              burrsutter Burr Sutter
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved:

                Development