If members in one site send messages to site masters of other sites, if we have multiple site masters, then reordering can occur.
- Member C in site LON sends messages C1 and C2 to site NYC
- C1 picks SM1 and C2 picks SM2
- Although site masters process messages in delivery order, because C1 and C2 travel through different site masters, C2 could pass C1, leading to ordering issues
- When sending a message to the one of the site masters of the current site, or from a site master to a remote site master, we invoke a callback to pick the site master based on the original caller
- Note that this only applies when we have more than one site master.
- The callback passes the address of the original caller and the list of available site masters, and needs to return a site master
- Transactions by different originators with overlapping key sets cannot be reliably ordered
- Transactional support therefore requires single site masters
- This should not be an issue as transactions bundle multiple updates, so the number of messages sent is less than non-transactional updates