Sticky site masters

If members in one site send messages to site masters of other sites, if we have multiple site masters, then reordering can occur.

Example:

• Member C in site LON sends messages C1 and C2 to site NYC
• C1 picks SM1 and C2 picks SM2
• Although site masters process messages in delivery order, because C1 and C2 travel through different site masters, C2 could pass C1, leading to ordering issues

Design

• When sending a message to the one of the site masters of the current site, or from a site master to a remote site master, we invoke a callback to pick the site master based on the original caller
• Note that this only applies when we have more than one site master.
• The callback passes the address of the original caller and the list of available site masters, and needs to return a site master

Transactions

• Transactions by different originators with overlapping key sets cannot be reliably ordered
• Transactional support therefore requires single site masters
• This should not be an issue as transactions bundle multiple updates, so the number of messages sent is less than non-transactional updates