HAModClusterService needs to handle cluster splits

The case where a split of the JGroups group occurs but nodes are still able to contact the httpd servers needs to be handled. There is a brief discussion of this on https://www.jboss.org/community/docs/DOC-11431 under "Split-Brain Syndrome". Problem is split-brain will result in nodes removing each other from httpd, resulting in no nodes active.

The wiki page describes a simple approach. A more complex approach would be to take a "primary partition" approach, whereby say an initial cluster of size n==6

{A, B, C, D, E, F}

splits into two cluster

{A, B, C, D}

and

{ E, F}

. To continue to handle requests a partition would need to have at least Math.floor((float) n / 2 + 1) members.

What kind of approach is appropriate would probably depend on the deployed webapps and how they interact with the cluster. If there is no clustered state that can become inconsistent across the cluster split, the simple approach described on the wiki can work fine (an HAModClusterService master doesn't disable a node if httpd reports it is still available). If there is shared state that needs to remain consistent (e.g. a clustered Hibernate Second Level Cache) then primary partition works better.

Most likely this overall problem will be resolved in stages, e.g. the simple approach from the wiki first.