Thinking on this a bit, the delivery flag needs to be treated like a message ack.
Currently the broker becomes aware of redelivery when a consumer closes and provides the broker sequence id of the last message delivered to the consumer. Any inflight messages with id > that that sequence are deemed to be redelivered.
In the even of a consumer failure, the close info from the consumer is lost, all unconsumed are deemed redelivered and there can be false positives.
In the event of a broker failure, all redelivery information is lost.
The change would have the consumer send an delivered_ack before a message is consumed, this would be a sync call that would require the broker to sync to disk. This will be as costly as a transaction commit!
There is a chance of a false positive on the redelivered flag, if the consumer fails after the delivered ack is persisted by the broker but before delivery of a message.
It may not be necessary to track updates in the index, in the event of failure we could always rebuild the index if this feature is enabled as it will be a rare event.
Question: is fast recovery vital in this case?