Data Inconsistency in case of Topology change in Infinispan Cluster

Data Inconsistency in case of Topology change in Infinispan Cluster

Infinispan Version : 8.2.5
Hibernate Version : 5.2.8
JGROUPS Version : 3.6.7

Clustering Mode : Replication
We have tested the same with invalidation mode too.

Refer below config cache-config for hibernate L2 entity types:

<replicated-cache-configuration name="entity" mode="SYNC" remote-timeout="20000" statistics="false" statistics-available="false">
<state-transfer enabled="false" timeout="20000000"/>
<locking isolation="READ_COMMITTED" concurrency-level="1000" acquire-timeout="15000" striping="false"/>
<transaction mode="NONE" auto-commit="false" locking="OPTIMISTIC"/>
<eviction size="-1" strategy="NONE"/>
<expiration max-idle="-1" interval="5000" lifespan="-1" />
</replicated-cache-configuration>

When a disconnected node rejoins the cluster, data remains inconsistent on the reconnected node.
This is happening for both Hibernate L2 Cache and some custom cache (AdvancedCache).
The below 4 scenarios should explain the issue.

For Example:
Scenario (Issue) 1:
-Initially the cluster comprises of 4 nodes, namely

{A,B,C,D}.
-Somehow, node D gets removed from the cluster view.
-Then some updates/inserts in hibernate L2 cache is done on Node B.
-These updates/inserts gets propagated to all the nodes in the current cluster view, i.e. {A,B,C}.
-And these updates/inserts doesn't get propagated to the Node D.
-Now D has stale state of the L2 cache.
-We expect, Node D should get the updated state of L2 Cache from {A,B,C}.

Scenario (Issue) 2:
-Initially the cluster comprises of 4 nodes, namely {A,B,C,D}

.
-Somehow, node D gets removed from the cluster view.
-Then some updates/inserts in hibernate L2 cache is done on Node B.
-These updates/inserts gets propagated to all the nodes in the current cluster view, i.e.

{A,B,C}.
-And these updates/inserts doesn't get propagated to the Node D.
-Now D has stale state of the L2 cache.
-Now D rejoins the cluster {A,B,C}

.
-Now the updated cluster view is

{A,B,C,D}.
-Still D has stale state of the L2 cache.
-We expect, Node D should get the updated state of L2 Cache from {A,B,C}.

Scenario (Issue) 3:
-Initially the cluster comprises of 4 nodes, namely {A,B,C,D}

.
-Somehow, node D gets removed from the cluster view.
-Then some updates/inserts in hibernate L2 cache is done on Node B.
-These updates/inserts gets propagated to all the nodes in the current cluster view, i.e.

{A,B,C}.
-And these updates/inserts doesn't get propagated to the Node D.
-Subsequently, some updates/inserts in hibernate L2 cache is done on Node D too. These updates are done on some other keys, and not on the keys on which the updates were done by Node B.
-Now {A,B,C}

and Node D have different but updated state of the L2 cache, And are not in sync.
-Now D rejoins the cluster

{A,B,C}.
-Now the updated cluster view is {A,B,C,D}.
-Still {A,B,C}

and Node D have different but updated state of the L2 cache, And are not in sync.
-We expect, updates from

{A,B,C} and Node D should get merged to all the nodes in cluster.

Scenario (Issue) 4:
-Initially the cluster comprises of 4 nodes, namely {A,B,C,D}.
-Somehow, node D gets removed from the cluster view.
-Then some updates/inserts in hibernate L2 cache is done on Node B.
-These updates/inserts gets propagated to all the nodes in the current cluster view, i.e. {A,B,C}

.
-And these updates/inserts doesn't get propagated to the Node D.
-Subsequently, some updates/inserts in hibernate L2 cache is done on Node D too. These updates might be on the same keys on which the updates were done by Node B.
-Now D has more updated state of the L2 cache.
-And

{A,B,C} are having stale state of the L2 Cache.
-Now D rejoins the cluster {A,B,C}

.
-Now the updated cluster view is

{A,B,C,D}

.
-Still

{A,B,C} have stale state of the L2 cache.
-We expect, {A,B,C}

should get the updated state of L2 Cache from Node D.