Merging on distributed caches can lead rehash failure

In the case of a MergeView, a node might be required to rejoin, meaning that it needs to rehash its keys. This can happen if the node is not in the winning partition.

In this cases, rehash might fail because after the initial DistributionManagerImpl.joinComplete, because joinComplete is still set to true.

Example:

2010-09-08 04:39:28,234 INFO [JGroupsTransport] (Incoming-1,Infinispan-Cluster,lnxdevvm102-4758) Received new cluster view: MergeView::[lnxdevvm102-45158|3] [lnxdevvm102-45158, lnxdevvm102-4758], subgroups=[[lnxdevvm102-45158|1] [lnxde
vvm102-4758], [lnxdevvm102-45158|2] [lnxdevvm102-45158]]
2010-09-08 04:39:28,354 DEBUG [JoinTask] (Rehasher-lnxdevvm102-4758) Commencing rehash on node: lnxdevvm102-4758. Before start, dmi.joinComplete = true
2010-09-08 04:39:28,372 DEBUG [FLUSH] (Incoming-1,Infinispan-Cluster,lnxdevvm102-4758) lnxdevvm102-4758: installing view MergeView::[lnxdevvm102-45158|3] [lnxdevvm102-45158, lnxdevvm102-4758], subgroups=[[lnxdevvm102-45158|1] [lnxdevvm1
02-4758], [lnxdevvm102-45158|2] [lnxdevvm102-45158]]
2010-09-08 04:39:28,376 DEBUG [JoinTask] (Rehasher-lnxdevvm102-4758) Commencing rehash on node: lnxdevvm102-4758. Before start, dmi.joinComplete = true
Commencing rehash on node: lnxdevvm102-4758. Before start, dmi.joinComplete = true
2010-09-08 04:39:28,476 ERROR [JoinTask] (Rehasher-lnxdevvm102-4758) Caught exception!
java.lang.IllegalStateException: Join cannot be complete without rehash to finish (node lnxdevvm102-4758 )
at org.infinispan.distribution.JoinTask.performRehash(JoinTask.java:82)
at org.infinispan.distribution.RehashTask.call(RehashTask.java:52)
at org.infinispan.distribution.RehashTask.call(RehashTask.java:32)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
2010-09-08 04:39:28,526 ERROR [JoinTask] (Rehasher-lnxdevvm102-4758) Caught exception!
java.lang.IllegalStateException: Join cannot be complete without rehash to finish (node lnxdevvm102-4758 )
at org.infinispan.distribution.JoinTask.performRehash(JoinTask.java:82)
at org.infinispan.distribution.RehashTask.call(RehashTask.java:52)
at org.infinispan.distribution.RehashTask.call(RehashTask.java:32)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)