Clustering performance regression

Our clustering stress tests show huge performance drop of 7.0.0.DR8 bits compared to 6.4.0 bits.

Compare these results:

stress-heavyread-session-repl-sync
7.0.0.DR8: throughput (max. 6250 TPS) + response times (max. 825 ms)
6.4.0.GA: throughput (max. 55000 TPS) + response times (max. 8 ms)

Jenkins runs (server configs + logs can be found here):
7.0.0.DR8: http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-7x-stress-heavyread-session-repl-sync_noperf21/2/
6.4.0.GA: http://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/eap-6x-stress-heavyread-session-repl-sync_noperf21/1/

One can observe that when using 7.0.0.DR8 bits, throughput reaches a threshold of approximately 6000 TPS. On the other hand, using 6.4.0 bits, the performance was scaling up to 55000 TPS. This happens in each test using REPL-SYNC cache. Another example:

stress-session-repl-sync
7.0.0.DR8: throughput (max. 6250 TPS) + response times (max. 675 ms)
6.4.0.GA: throughput (max. 12500 TPS) + response times (max. 850 ms)

All tests use a 4-node EAP cluster and 5 nodes that generate load. Adding another node to the EAP cluster, the threshold is lowered to 5200 TPS and the max. response time raises from 825 to 1000 ms, see:

stress-heavyread-session-repl-sync with 5-node cluster
7.0.0.DR8: throughput (max. 5500 TPS) + response times (max. 1000 ms)

Most of the other scenarios indicate performance regression as well.

stress-session-dist-sync
7.0.0.DR8: throughput (max. 8500 TPS) + response times (max. 550 ms)
6.4.0.GA: throughput (max. 23000 TPS) + response times (max. 170 ms)

stress-session-repl-async
7.0.0.DR8: throughput (max. 3500 TPS) + response times (max. 1700 ms)
6.4.0.GA: throughput (max. 10500 TPS) + response times (max. 1050 ms)

This issue is related to https://issues.jboss.org/browse/JBEAP-768