Encountered OOME with majority of the heap loaded in the amqp connection pending persistentSize

2024-01-08 22:04:05,517 ERROR [org.apache.activemq.artemis.core.journal.impl.JournalImpl] Exception during appendAddRecord:
java.lang.OutOfMemoryError: Java heap space
» at org.apache.activemq.artemis.utils.collections.ConcurrentLongHashMap$Section.rehash(ConcurrentLongHashMap.java:452) ~[artemis-commons-2.28.0.redhat-00012.jar:?]
» at org.apache.activemq.artemis.utils.collections.ConcurrentLongHashMap$Section.put(ConcurrentLongHashMap.java:334) ~[artemis-commons-2.28.0.redhat-00012.jar:?]
» at org.apache.activemq.artemis.utils.collections.ConcurrentLongHashMap.put(ConcurrentLongHashMap.java:128) ~[artemis-commons-2.28.0.redhat-00012.jar:?]
» at org.apache.activemq.artemis.core.journal.impl.JournalImpl$1.run(JournalImpl.java:937) ~[artemis-journal-2.28.0.redhat-00012.jar:2.28.0.redhat-00012]
» at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57) ~[artemis-commons-2.28.0.redhat-00012.jar:?]
» at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32) ~[artemis-commons-2.28.0.redhat-00012.jar:?]
» at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68) ~[artemis-commons-2.28.0.redhat-00012.jar:?]
» at org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$198/0x00000008003af220.run(Unknown Source) ~[?:?]
» at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
» at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
» at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118) ~[artemis-commons-2.28.0.redhat-00012.jar:?]

One instance of "org.apache.activemq.artemis.core.server.impl.QueueImpl" loaded by "java.net.URLClassLoader @ 0x700000000" occupies 4,039,332,072 (94.66%) bytes.

name     | consumers.consumers.size | pendingMetrics.messageCount | a.pendingMetrics.persistentSize | a.deliveringMetrics.messageCount | a.deliveringMetrics.persistentSize
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
$ACTIVEMQ_ARTEMIS_MIRROR_-01-to-02 |       1 |   282,612 |  3,149,248,978 |  1,379 | 144,795
Q0002                              |   1,114 |       679 |      7,626,740 |      0 |       0    
DLQ                                |       0 |         1 |         11,327 |      0 |       0    
DR_BROKER:61616?sslEnabled=true    |       0 |         6 |          7,182 |      0 |       0    
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------