Messages Fail to Page In from JDBC Datasource On Broker Restart

Upon internal broker restart (triggered by JDBC IOException), we see that not all messages are successfully paged into the queue. The queue count accurately reflects the count for the destination in the JDBC source; however, none of the messages are browseable and the messages do not appear to be loaded into the cursor.

In this instance, we had 1 "stuck" message that was queryable in the database, showed in the queue count, but was not consumed by the consumer and was not browseable via Hawtio:

2018-02-16 15:08:49,158 | DEBUG | ce[amq02] Task-4 | Queue                            | che.activemq.broker.region.Queue 1926 | 162 - org.apache.activemq.activemq-osgi - 5.11.0.redhat-630310 | queue://TEST.INBOUND_QUEUE, subscriptions=1, memory=0%, size=1, pending=0 toPageIn: 1, force:false, Inflight: 0, pagedInMessages.size 0, pagedInPendingDispatch.size 0, enqueueCount: 19, dequeueCount: 23, memUsage:0, maxPageSize:200

We could send and receive other messages, but the message in question remained unbrowseable, but still counting against the queue depth. This issue occurred after an IOException restarted the broker, with the same broker obtaining the lock following the restart. We were able to reproduce the issue several times and upon subsequent internal restarts, some of the "stuck" messages got paged in and consumed, but in the case above one orphaned message remained.

Restarting consumers had no effect, but a JVM restart on the broker resulted in the messages all being paged in and consumed.

In the reproducer environment, there is a single network connection to another broker, but the queue depth for the same queue reported there was "0."