SEQUENCER problems and slowness

On high volumes of messages (e.g. MPerf), SEQUENCER runs out of memory.

The culprit is the forward-table, which is unbounded, and if messages are added as a faster rate than the time it takes to forward the message to the coordinator (sequencer) and for it to be received as a multicast and subsequently be removed from the forward-table, memory increases.

This also makes SEQUENCER slow.

SOLUTION:

• Look into bounding the forward-table, e.g. define the max number of bytes to be in the table at any given time
• When a message is to be added, its length is checked (Message.getLength())
  • If the length is 0 or length + accumulated bytes is less than max: add, else block
• A received message (sent by self) causes the entry to be removed from the forward-table and its length to be subtracted from accumulated bytes (unblocking potentially blocked threads)

We should also look at delivery table and see if we can make it simpler, e.g. by running a gathering round when a new sequencer is chosen, determining the highest seqnos of all members