StreamingStateTransfer: queue requests during state transfer

STATE should queue requests to be delivered to the application while a state transfer is in progress.

Draft design:

• MessageQueue: queue for application messages
• STATE-REQ queue: queue for STATE-REQ requests

• On STATE-REQ:
  • If busy --> add request to STATE-REQ queue, return
  • Else --> processRequest
  • Close MessageQueue and deliver all messages in it to the application
    • Regular message deliver needs to be blocked until all messages from the MessageQueue have been delivered
  • Kick off a STABLE round, wait until it completes
  • If STATE-REQ queue is not empty --> processRequests()

• processRequest():
  • Close BARRIER, suspend STABLE
  • Get digest
  • Open MessageQueue: all new messages (MSG) will be queued now, rather than delivered to the application
    • (Only MSG, not VIEW !)
  • Open BARRIER (leave STABLE suspended !)
  • Transfer state to state requester
  • Resume STABLE

Advantages:

• The state provider is not blocked, e.g. to handle JOIN requests
• Only the state requester is blocked, this is OK though
• Between the state requests, the state provider gets a little breathing room to process
  queued-up messages, and to purge messages (through stability)

• Disadvantages:
• If blocking messages (e.g. RPCs) are queued, we won't be able to send responses back until the queue is drained

Misc:

• Should the MessageQueue use the disk to handle large message sets ?
• If modifications are idempotent, we don't need to queue messages (make this configurable !)
  (for example Infinispan ?)