Handling X-Site split brains

With ASYNC x-site configurations, sites can get out of sync when the replication link is down. We use RELAY2, which basically forwards traffic to other sites but what happen is one of them is flaky?

The biggest hurdle here is the way state transfer works. Because it's manual, it requires someone (or some script) detecting the split and when it heals pushing the state via JMX op. Automatic rebalancing could take time given the links' extra latency, so it's not clear what the solution should be.

We do definitely need to implement some soft of conflict resolution and apply the same semantics we use for inner cluster communication regardless.