Live node dropping out of cluster can cause orphaned messages

When a live node is kicked out of the cluster (for whatever reason) its JBoss Messaging ServerPeer remains active which means the node is still available to receive messages and, if necessary, persist them to the database. However, the CHANNEL_ID values used for the persisted messages will be invalid since the fail-over procedure would have removed the "dead" node's channels from the JBM_POSTOFFICE and new CHANNEL_ID values will be used when the "dead" node is finally restarted. This essentially orphans any persistent messages sent to the "dead" node after it has been kicked out of the cluster. Recovering the orphaned messages would require manual intervention with the database, but even then it might not be possible to match the old CHANNEL_ID values with the new ones.

The simplest way to solve this problem is to add a constraint to the database that forces the inserted message's CHANNEL_ID to correspond to a CHANNEL_ID from JBM_POSTOFFICE. Of course, such a constraint will hurt performance a bit, but it could be optional - customers could trade robustness for speed.

The node would still need to be restarted in order to rejoin the cluster appropriately, but the constraint would avoid orphaned messages.