State transfer in x-site scenarios

Currently if state stransfer is required in the x-site scenarios, it needs to be triggered manually through CLI or JMX. Will be nice if there is a possibility to somehow automatically setup state transfer OOTB, which will be triggered when new site joins the cloud OR when site joins the cloud after previously temporarily disconnected (split-brain).

• It could be good to setup this unidirectionally or bi-directionally. In case of bi-directional, every site may receive the entities, which were created in the meantime on the other site. For already existing entities, there could be some strategies like IGNORE_EXISTING, OVERRIDE_EXISTING etc. So in case that some entity exist on both sites, it may be possible to choose one site to have bigger priority and "win" . So for example in case that site1 has bigger priority than site2 and strategy is OVERRIDE_EXISTING, then entity "123" will be always overriden on site2 with the value from site1.

• Things might be even more tricky to properly support removals or updates during split-brain. That could require to track the removed entities for some time. For example, the scenario like:
  • The entity "123" exists on both sites
  • In time 10:00 there is split-brain
  • In time 10:05 the entity "123" was removed on site2
  • In time 10:06 the entity "456" was created on site1
  • In time 10:10 the split-brain is "fixed" and sites are re-connected. At this moment, both entities with keys "123" and "456" are available just on site1, but not on site2. ATM I want the new entity "456" to be created on site2, but I don't want the entity "123" to be re-created on site2 based on the value from site1. Instead I want "123" to be removed on site1.

I don't know how hard would be to properly support the example above or if it's even doable considering various corner cases. Just added it for completeness