Details
-
Bug
-
Resolution: Done
-
Blocker
-
No Release, 7.1.0.Beta1
Description
Update:
This was a suspicion first, now I have compared results for AS5. Clustering performs 10x slower with the same use case in 2 node cluster*. Also this has been tested on revision with updated Infinispan to CR1. CPU usage seems relatively high, network is not used to its maximum, most likely looks like blocking in threading.
AS7
Nodes: 2, Sessions: 100, active: 100, samples: 8363, throughput 418.1 samples/s, 0.0 MB/s, mean response: 138 ms, sampling errors: 0, invalid samples: 0, valid samples: 8363 (100%) Nodes: 2, Sessions: 200, active: 200, samples: 9607, throughput 480.3 samples/s, 0.0 MB/s, mean response: 298 ms, sampling errors: 0, invalid samples: 0, valid samples: 9607 (100%) Nodes: 2, Sessions: 300, active: 300, samples: 12779, throughput 555.6 samples/s, 0.0 MB/s, mean response: 550 ms, sampling errors: 0, invalid samples: 151, valid samples: 12628 (98%) Nodes: 2, Sessions: 400, active: 400, samples: 13336, throughput 666.7 samples/s, 0.0 MB/s, mean response: 498 ms, sampling errors: 0, invalid samples: 1, valid samples: 13335 (99%) Nodes: 2, Sessions: 500, active: 500, samples: 11852, throughput 592.5 samples/s, 0.0 MB/s, mean response: 441 ms, sampling errors: 0, invalid samples: 1, valid samples: 11851 (99%) Nodes: 2, Sessions: 600, active: 600, samples: 16012, throughput 800.5 samples/s, 0.0 MB/s, mean response: 634 ms, sampling errors: 0, invalid samples: 0, valid samples: 16012 (100%) Nodes: 2, Sessions: 700, active: 700, samples: 14436, throughput 721.7 samples/s, 0.0 MB/s, mean response: 888 ms, sampling errors: 0, invalid samples: 350, valid samples: 14086 (97%) Nodes: 2, Sessions: 800, active: 800, samples: 14479, throughput 629.5 samples/s, 0.0 MB/s, mean response: 1080 ms, sampling errors: 0, invalid samples: 0, valid samples: 14479 (100%) Nodes: 2, Sessions: 900, active: 900, samples: 14473, throughput 723.6 samples/s, 0.0 MB/s, mean response: 1345 ms, sampling errors: 0, invalid samples: 450, valid samples: 14023 (96%) Nodes: 2, Sessions: 1000, active: 1000, samples: 14570, throughput 728.4 samples/s, 0.0 MB/s, mean response: 1274 ms, sampling errors: 0, invalid samples: 1, valid samples: 14569 (99%) Nodes: 2, Sessions: 1100, active: 1100, samples: 13808, throughput 690.3 samples/s, 0.0 MB/s, mean response: 1933 ms, sampling errors: 0, invalid samples: 1100, valid samples: 12708 (92%) Nodes: 2, Sessions: 1200, active: 1200, samples: 13252, throughput 662.5 samples/s, 0.0 MB/s, mean response: 1833 ms, sampling errors: 0, invalid samples: 600, valid samples: 12652 (95%) Nodes: 2, Sessions: 1300, active: 1300, samples: 17815, throughput 754.8 samples/s, 0.0 MB/s, mean response: 1581 ms, sampling errors: 0, invalid samples: 1300, valid samples: 16515 (92%) Nodes: 2, Sessions: 1400, active: 1400, samples: 17558, throughput 877.8 samples/s, 0.0 MB/s, mean response: 1509 ms, sampling errors: 0, invalid samples: 1, valid samples: 17557 (99%)
AS 5
Nodes: 2, Sessions: 100, active: 100, samples: 19240, throughput 961.2 samples/s, 0.0 MB/s, mean response: 4 ms, sampling errors: 0, invalid samples: 0, valid samples: 19240 (100%) Nodes: 2, Sessions: 200, active: 200, samples: 37887, throughput 1,894.1 samples/s, 0.0 MB/s, mean response: 5 ms, sampling errors: 0, invalid samples: 0, valid samples: 37887 (100%) Nodes: 2, Sessions: 300, active: 300, samples: 69732, throughput 2,843.5 samples/s, 0.0 MB/s, mean response: 4 ms, sampling errors: 0, invalid samples: 0, valid samples: 69732 (100%) Nodes: 2, Sessions: 400, active: 400, samples: 75148, throughput 3,755.1 samples/s, 0.0 MB/s, mean response: 6 ms, sampling errors: 0, invalid samples: 0, valid samples: 75148 (100%) Nodes: 2, Sessions: 500, active: 500, samples: 92979, throughput 4,648.4 samples/s, 0.0 MB/s, mean response: 6 ms, sampling errors: 0, invalid samples: 3, valid samples: 92976 (99%) Nodes: 2, Sessions: 600, active: 600, samples: 108989, throughput 5,449.0 samples/s, 0.0 MB/s, mean response: 10 ms, sampling errors: 0, invalid samples: 1, valid samples: 108988 (99%) Nodes: 2, Sessions: 700, active: 700, samples: 112413, throughput 5,508.6 samples/s, 0.0 MB/s, mean response: 26 ms, sampling errors: 0, invalid samples: 0, valid samples: 112413 (100%) Nodes: 2, Sessions: 800, active: 800, samples: 88810, throughput 4,435.9 samples/s, 0.0 MB/s, mean response: 78 ms, sampling errors: 0, invalid samples: 0, valid samples: 88810 (100%) Nodes: 2, Sessions: 900, active: 900, samples: 63417, throughput 3,170.6 samples/s, 0.0 MB/s, mean response: 188 ms, sampling errors: 0, invalid samples: 487, valid samples: 62930 (99%) Nodes: 2, Sessions: 1000, active: 1000, samples: 85718, throughput 4,285.4 samples/s, 0.0 MB/s, mean response: 129 ms, sampling errors: 0, invalid samples: 0, valid samples: 85718 (100%) Nodes: 2, Sessions: 1100, active: 1100, samples: 67710, throughput 3,385.1 samples/s, 0.0 MB/s, mean response: 232 ms, sampling errors: 0, invalid samples: 0, valid samples: 67710 (100%) Nodes: 2, Sessions: 1200, active: 1200, samples: 88369, throughput 4,417.9 samples/s, 0.0 MB/s, mean response: 165 ms, sampling errors: 0, invalid samples: 2, valid samples: 88367 (99%) Nodes: 2, Sessions: 1300, active: 1300, samples: 113719, throughput 5,685.4 samples/s, 0.0 MB/s, mean response: 124 ms, sampling errors: 0, invalid samples: 0, valid samples: 113719 (100%) Nodes: 2, Sessions: 1400, active: 1400, samples: 119857, throughput 5,210.8 samples/s, 0.0 MB/s, mean response: 163 ms, sampling errors: 0, invalid samples: 0, valid samples: 119857 (100%) Nodes: 2, Sessions: 1500, active: 1500, samples: 102037, throughput 5,099.0 samples/s, 0.0 MB/s, mean response: 197 ms, sampling errors: 0, invalid samples: 0, valid samples: 102037 (100%) Nodes: 2, Sessions: 1600, active: 1600, samples: 175482, throughput 4,535.1 samples/s, 0.0 MB/s, mean response: 244 ms, sampling errors: 0, invalid samples: 0, valid samples: 175482 (100%) Nodes: 2, Sessions: 1700, active: 1700, samples: 110511, throughput 5,524.7 samples/s, 0.0 MB/s, mean response: 236 ms, sampling errors: 0, invalid samples: 1992, valid samples: 108519 (98%) Nodes: 2, Sessions: 1800, active: 1800, samples: 120824, throughput 6,009.3 samples/s, 0.0 MB/s, mean response: 201 ms, sampling errors: 0, invalid samples: 0, valid samples: 120824 (100%) Nodes: 2, Sessions: 1900, active: 1900, samples: 159295, throughput 7,875.4 samples/s, 0.0 MB/s, mean response: 140 ms, sampling errors: 0, invalid samples: 0, valid samples: 159295 (100%) Nodes: 2, Sessions: 2000, active: 2000, samples: 75813, throughput 3,769.3 samples/s, 0.0 MB/s, mean response: 430 ms, sampling errors: 0, invalid samples: 2000, valid samples: 73813 (97%) Nodes: 2, Sessions: 2100, active: 2100, samples: 134139, throughput 5,831.5 samples/s, 0.0 MB/s, mean response: 252 ms, sampling errors: 0, invalid samples: 0, valid samples: 134139 (100%)
----------------------------------------------------------------------------
This was suspected.
Summary: Cluster suspected to perform 100x slower than unclustered
Cluster performs 100x slower than unclustered when using HTTP session replication benchmark in our perf lab. Throughput with 200 clients drops from 49,851.0 r/s to 425.3 samples/s when using clustering in standalone-ha.xml server configuration. I cannot guarantee the runs, this might be a result of a hidden networking issue in the lab.
Here are some sample runs:
Unclustered
Nodes: 2, Sessions: 100, active: 100, samples: 680317, throughput 45,346.9 samples/s, 0.2 MB/s, mean response: 2 ms, sampling errors: 0, invalid samples: 0, valid samples: 680317 (100%) Nodes: 2, Sessions: 200, active: 200, samples: 747890, throughput 49,851.0 samples/s, 0.2 MB/s, mean response: 3 ms, sampling errors: 0, invalid samples: 0, valid samples: 747890 (100%) Nodes: 2, Sessions: 300, active: 300, samples: 722338, throughput 48,149.4 samples/s, 0.2 MB/s, mean response: 6 ms, sampling errors: 0, invalid samples: 0, valid samples: 722338 (100%) Nodes: 2, Sessions: 400, active: 400, samples: 702323, throughput 46,813.7 samples/s, 0.2 MB/s, mean response: 8 ms, sampling errors: 0, invalid samples: 0, valid samples: 702323 (100%) Nodes: 2, Sessions: 500, active: 500, samples: 690571, throughput 46,028.9 samples/s, 0.2 MB/s, mean response: 10 ms, sampling errors: 0, invalid samples: 0, valid samples: 690571 (100%) Nodes: 2, Sessions: 600, active: 600, samples: 689405, throughput 45,949.6 samples/s, 0.2 MB/s, mean response: 13 ms, sampling errors: 0, invalid samples: 0, valid samples: 689405 (100%) Nodes: 2, Sessions: 700, active: 700, samples: 681118, throughput 45,400.3 samples/s, 0.2 MB/s, mean response: 15 ms, sampling errors: 0, invalid samples: 0, valid samples: 681118 (100%) Nodes: 2, Sessions: 800, active: 800, samples: 685431, throughput 45,684.7 samples/s, 0.2 MB/s, mean response: 16 ms, sampling errors: 0, invalid samples: 0, valid samples: 685431 (100%) Nodes: 2, Sessions: 900, active: 900, samples: 668869, throughput 44,580.9 samples/s, 0.2 MB/s, mean response: 20 ms, sampling errors: 0, invalid samples: 0, valid samples: 668869 (100%) Nodes: 2, Sessions: 1000, active: 1000, samples: 675849, throughput 45,046.1 samples/s, 0.2 MB/s, mean response: 21 ms, sampling errors: 0, invalid samples: 0, valid samples: 675849 (100%)
Clustered
Nodes: 2, Sessions: 100, active: 100, samples: 7350, throughput 489.9 samples/s, 0.0 MB/s, mean response: 196 ms, sampling errors: 0, invalid samples: 49, valid samples: 7301 (99%) Nodes: 2, Sessions: 200, active: 200, samples: 6380, throughput 425.3 samples/s, 0.0 MB/s, mean response: 458 ms, sampling errors: 0, invalid samples: 362, valid samples: 6018 (94%) Nodes: 2, Sessions: 300, active: 300, samples: 5821, throughput 382.4 samples/s, 0.0 MB/s, mean response: 748 ms, sampling errors: 0, invalid samples: 484, valid samples: 5337 (91%) Nodes: 2, Sessions: 400, active: 262, samples: 3846, throughput 256.3 samples/s, 0.0 MB/s, mean response: 1056 ms, sampling errors: 0, invalid samples: 333, valid samples: 3513 (91%) Nodes: 2, Sessions: 500, active: 452, samples: 6389, throughput 425.6 samples/s, 0.0 MB/s, mean response: 1104 ms, sampling errors: 0, invalid samples: 565, valid samples: 5824 (91%) Nodes: 2, Sessions: 600, active: 55, samples: 985, throughput 65.7 samples/s, 0.0 MB/s, mean response: 1590 ms, sampling errors: 457, invalid samples: 55, valid samples: 473 (48%)