Rolling Upgrade: use of Remote Store in mode read-only causes data inconsistencies

We recommend to do rolling upgrades with the target cluster with a remote-store pointing to the source cluster, in mode ready-only. Clients are switched to point to the target cluster.

This cause situations like this in the client:

client.put("K","value")

// will get "value" back
client.get("K")                 

// Deletes will not propagate to the source cluster, 
// the RemoteStore is in 'read-only' mode.
client.remove("K")     

// Returns "value". Although deleted, value will be retrieved 
// from the remote store      
cache.get("K")     
       

This can break existing applications that expect a consistent access to data during a Rolling Upgrade process. Clearly the remote store should not be in read only mode, but at the same time, as the Rolling Upgrade does a put in the target cache reading data from the source cache, it should not trigger a write back to the source cluster.