Uploaded image for project: 'AMQ Broker'
  1. AMQ Broker
  2. ENTMQBR-3112

[AMQ7, purge message, OutOfMemoryException] with a large queue size, removeAllMessages() takes a long time and eventually results in an OOM exception (if enough messages on the queue)

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • AMQ 7.4.2.GA
    • AMQ 7.4.0.CR2
    • broker-core
    • None
    • Release Notes
    • +
    • Hide
      Previously, for a queue with a large number (that is, in the order of millions) of messages, the JMX operation `removeAllMessages` could take up to several minutes to complete. In addition, the operation sometimes failed to complete, returning an out-of-memory (OOM) exception instead. This issue is now resolved.
      Show
      Previously, for a queue with a large number (that is, in the order of millions) of messages, the JMX operation `removeAllMessages` could take up to several minutes to complete. In addition, the operation sometimes failed to complete, returning an out-of-memory (OOM) exception instead. This issue is now resolved.
    • Documented as Resolved Issue
    • Verified in a release
    • Hide

      1) generate a broker instance using the default options.

      2) use defaults for "./artemis producer" to generate 4 million messages on the "TEST" queue.

      3) invoke JMX operation "removeAllMessages()" on the "TEST" queue.

      RESULT:

      • takes excessively long time to complete (on my system more than 4 minutes).
      • eventually returns with an OOM exception.
      Show
      1) generate a broker instance using the default options. 2) use defaults for "./artemis producer" to generate 4 million messages on the "TEST" queue. 3) invoke JMX operation "removeAllMessages()" on the "TEST" queue. RESULT: takes excessively long time to complete (on my system more than 4 minutes). eventually returns with an OOM exception.

    Description

      On a queue that contains 4,000, 000 messages the JMX operation "removeAllMessages()" eventually results in an OOM exception on the broker (below).

      The messages were generated using the defaults for sample producer

      ./artemis producer

      For my test I was using the default generated broker configuration for AMQ 7.4.0 . Before "removeAllMessages()" was invoked the destination was PAGING and had more than 50 pages.

      From the users perspective there are 2 closely related usability issues:

      1) Out of Memory exception should not occur
      2) The operation took more than 4 minutes to complete. 

      2019-08-22 17:37:03,315 WARN  [org.apache.activemq.artemis.journal] AMQ142024: Error completing callback: java.lang.OutOfMemoryError: Java heap space
	at java.util.LinkedList.listIterator(LinkedList.java:868) [rt.jar:1.8.0_212]
	at java.util.AbstractList.listIterator(AbstractList.java:299) [rt.jar:1.8.0_212]
	at java.util.AbstractSequentialList.iterator(AbstractSequentialList.java:239) [rt.jar:1.8.0_212]
	at org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.checkTasks(OperationContextImpl.java:216) [artemis-server-2.9.0.redhat-00002.jar:2.9.0.redhat-00002]
	at org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.done(OperationContextImpl.java:197) [artemis-server-2.9.0.redhat-00002.jar:2.9.0.redhat-00002]
	at org.apache.activemq.artemis.core.journal.impl.TransactionCallback.done(TransactionCallback.java:51) [artemis-journal-2.9.0.redhat-00002.jar:2.9.0.redhat-00002]
	at org.apache.activemq.artemis.core.io.IOCallback.lambda$done$0(IOCallback.java:43) [artemis-journal-2.9.0.redhat-00002.jar:2.9.0.redhat-00002]
	at org.apache.activemq.artemis.core.io.IOCallback$$Lambda$62/631008546.accept(Unknown Source)
	at java.util.ArrayList.forEach(ArrayList.java:1257) [rt.jar:1.8.0_212]
	at org.apache.activemq.artemis.core.io.IOCallback.done(IOCallback.java:41) [artemis-journal-2.9.0.redhat-00002.jar:2.9.0.redhat-00002]
	at org.apache.activemq.artemis.core.io.DelegateCallback.done(DelegateCallback.java:41) [artemis-journal-2.9.0.redhat-00002.jar:2.9.0.redhat-00002]
	at org.apache.activemq.artemis.core.io.aio.AIOSequentialFileFactory$AIOSequentialCallback.sequentialDone(AIOSequentialFileFactory.java:400) [artemis-journal-2.9.0.redhat-00002.jar:2.9.0.redhat-00002]
	at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.done(AIOSequentialFile.java:302) [artemis-journal-2.9.0.redhat-00002.jar:2.9.0.redhat-00002]
	at org.apache.activemq.artemis.core.io.aio.AIOSequentialFileFactory$AIOSequentialCallback.done(AIOSequentialFileFactory.java:384) [artemis-journal-2.9.0.redhat-00002.jar:2.9.0.redhat-00002]
	at org.apache.activemq.artemis.nativo.jlibaio.LibaioContext.done(LibaioContext.java:362) [activemq-artemis-native-1.0.0.redhat-00001.jar:1.0.0.redhat-00001]
	at org.apache.activemq.artemis.nativo.jlibaio.LibaioContext.blockedPoll(Native Method) [activemq-artemis-native-1.0.0.redhat-00001.jar:1.0.0.redhat-00001]
	at org.apache.activemq.artemis.nativo.jlibaio.LibaioContext.poll(LibaioContext.java:354) [activemq-artemis-native-1.0.0.redhat-00001.jar:1.0.0.redhat-00001]
	at org.apache.activemq.artemis.core.io.aio.AIOSequentialFileFactory$PollerThread.run(AIOSequentialFileFactory.java:422) [artemis-journal-2.9.0.redhat-00002.jar:2.9.0.redhat-00002]

      Attachments

        Issue Links

          clones

          Bug - A problem that impairs or prevents the functions of the product. ENTMQBR-2767 [AMQ7, purge message, OutOfMemoryException] with a large queue size, removeAllMessages() takes a long time and eventually results in an OOM exception (if enough messages on the queue)

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed

          Activity

            People

              dbruscin Domenico Francesco Bruscino
              dbruscin Domenico Francesco Bruscino
              Oleg Sushchenko Oleg Sushchenko
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: