Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Won't Do
Priority: Major
Fix Version/s: None
Affects Version/s: 9.1.0.Final
Component/s: Cloud, Loaders and Stores
Labels:
None

Description

When testing Infinispan Persistent Docker container (with RocksDB) we encountered very interesting problem. The container went up to 100% of memory limit and then was killed by the OOM killer. After removing RocksDB Cache Store the problem went away.

After looking at dmsg it turned out that the container was killed because of the OOM. The interesting thing is that the OOM allows to process to keep using 100% of the memory but only for a short amount of time.

The second thing we investigated are GC logs:

2017-08-01T10:06:33.421+0000: [GC (Allocation Failure) 
Desired survivor size 35651584 bytes, new threshold 2 (max 15)
[PSYoungGen: 38624K->3328K(70144K)] 95950K->63878K(280576K), 0.0089506 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] 
2017-08-01T10:06:33.431+0000: Total time for which application threads were stopped: 0.0117935 seconds, Stopping threads took: 0.0002661 seconds
2017-08-01T10:06:36.797+0000: Total time for which application threads were stopped: 0.1213646 seconds, Stopping threads took: 0.0153583 seconds
2017-08-01T10:06:37.425+0000: Total time for which application threads were stopped: 0.3444540 seconds, Stopping threads took: 0.0092094 seconds
2017-08-01T10:06:40.719+0000: Total time for which application threads were stopped: 0.2126636 seconds, Stopping threads took: 0.0643458 seconds
2017-08-01T10:06:42.663+0000: Total time for which application threads were stopped: 0.9234535 seconds, Stopping threads took: 0.0554174 seconds
TLAB: gc thread: 0x00007fc6b402c000 [id: 1038] desired_size: 4KB slow allocs: 6  refill waste: 64B alloc: 0.00609      139KB refills: 12 waste  3.9% gc: 1464B slow: 528B fast: 72B
TLAB: gc thread: 0x00007fc6b4021800 [id: 1037] desired_size: 4KB slow allocs: 6  refill waste: 64B alloc: 0.00609      139KB refills: 13 waste  7.5% gc: 3656B slow: 560B fast: 72B
TLAB: gc thread: 0x00007fc6f4088000 [id: 975] desired_size: 2KB slow allocs: 0  refill waste: 32B alloc: 0.00118       27KB refills: 14 waste  4.5% gc: 928B slow: 312B fast: 56B
TLAB: gc thread: 0x00007fc6ec019800 [id: 788] desired_size: 3KB slow allocs: 5  refill waste: 48B alloc: 0.00431       98KB refills: 32 waste  2.0% gc: 960B slow: 1016B fast: 0B
TLAB: gc thread: 0x00007fc6d0016800 [id: 787] desired_size: 3KB slow allocs: 9  refill waste: 48B alloc: 0.00432       98KB refills: 30 waste  2.4% gc: 1160B slow: 1112B fast: 0B
TLAB: gc thread: 0x00007fc70c039800 [id: 786] desired_size: 2KB slow allocs: 5  refill waste: 40B alloc: 0.00419       95KB refills: 35 waste  2.1% gc: 1024B slow: 1192B fast: 0B
TLAB: gc thread: 0x00007fc6b4016800 [id: 785] desired_size: 3KB slow allocs: 7  refill waste: 48B alloc: 0.00427       97KB refills: 31 waste  1.5% gc: 264B slow: 1144B fast: 0B
TLAB: gc thread: 0x00007fc69c072000 [id: 784] desired_size: 3KB slow allocs: 10  refill waste: 48B alloc: 0.00432       98KB refills: 33 waste  1.3% gc: 256B slow: 1048B fast: 0B
TLAB: gc thread: 0x00007fc6c0022000 [id: 783] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00449      102KB refills: 23 waste  2.4% gc: 1000B slow: 808B fast: 0B
TLAB: gc thread: 0x00007fc6f4079800 [id: 782] desired_size: 3KB slow allocs: 4  refill waste: 48B alloc: 0.00427       97KB refills: 28 waste  2.9% gc: 1496B slow: 968B fast: 0B
TLAB: gc thread: 0x00007fc6bc047800 [id: 781] desired_size: 3KB slow allocs: 5  refill waste: 48B alloc: 0.00426       97KB refills: 30 waste  3.5% gc: 2192B slow: 1072B fast: 0B
TLAB: gc thread: 0x00007fc6c800d000 [id: 780] desired_size: 3KB slow allocs: 5  refill waste: 48B alloc: 0.00442      101KB refills: 26 waste  2.2% gc: 1064B slow: 792B fast: 0B
TLAB: gc thread: 0x00007fc6b0018000 [id: 779] desired_size: 3KB slow allocs: 4  refill waste: 48B alloc: 0.00426       97KB refills: 29 waste  3.2% gc: 1832B slow: 1024B fast: 0B
TLAB: gc thread: 0x00007fc6a402e800 [id: 778] desired_size: 3KB slow allocs: 7  refill waste: 48B alloc: 0.00440      100KB refills: 34 waste  3.1% gc: 2328B slow: 976B fast: 0B
TLAB: gc thread: 0x00007fc6b8062000 [id: 777] desired_size: 3KB slow allocs: 10  refill waste: 48B alloc: 0.00435       99KB refills: 31 waste  2.9% gc: 1744B slow: 1056B fast: 0B
TLAB: gc thread: 0x00007fc694016800 [id: 776] desired_size: 3KB slow allocs: 5  refill waste: 48B alloc: 0.00427       97KB refills: 32 waste  4.2% gc: 2992B slow: 1160B fast: 0B
TLAB: gc thread: 0x00007fc6cc016000 [id: 775] desired_size: 2KB slow allocs: 7  refill waste: 40B alloc: 0.00421       96KB refills: 30 waste  2.5% gc: 1312B slow: 984B fast: 0B
TLAB: gc thread: 0x00007fc6c4069800 [id: 774] desired_size: 3KB slow allocs: 9  refill waste: 48B alloc: 0.00431       98KB refills: 28 waste  1.6% gc: 480B slow: 952B fast: 0B
TLAB: gc thread: 0x0000000001823800 [id: 773] desired_size: 3KB slow allocs: 7  refill waste: 48B alloc: 0.00447      102KB refills: 31 waste  3.1% gc: 2152B slow: 1000B fast: 0B
TLAB: gc thread: 0x00007fc6b4014800 [id: 772] desired_size: 3KB slow allocs: 5  refill waste: 48B alloc: 0.00444      101KB refills: 33 waste  2.5% gc: 1472B slow: 1152B fast: 0B
TLAB: gc thread: 0x00007fc70c0ab800 [id: 771] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00451      103KB refills: 31 waste  1.1% gc: 40B slow: 1096B fast: 0B
TLAB: gc thread: 0x00007fc694014800 [id: 770] desired_size: 3KB slow allocs: 8  refill waste: 48B alloc: 0.00436       99KB refills: 30 waste  3.0% gc: 1808B slow: 1064B fast: 0B
TLAB: gc thread: 0x00007fc6d0014800 [id: 769] desired_size: 3KB slow allocs: 8  refill waste: 48B alloc: 0.00466      106KB refills: 29 waste  3.4% gc: 2456B slow: 904B fast: 0B
TLAB: gc thread: 0x00007fc69c070000 [id: 768] desired_size: 3KB slow allocs: 9  refill waste: 48B alloc: 0.00430       98KB refills: 32 waste  2.2% gc: 920B slow: 1272B fast: 0B
TLAB: gc thread: 0x00007fc6c4068000 [id: 767] desired_size: 3KB slow allocs: 8  refill waste: 48B alloc: 0.00426       97KB refills: 31 waste  1.3% gc: 136B slow: 1128B fast: 0B
TLAB: gc thread: 0x00007fc6ec017000 [id: 766] desired_size: 2KB slow allocs: 8  refill waste: 40B alloc: 0.00414       94KB refills: 32 waste  3.5% gc: 2336B slow: 1048B fast: 0B
TLAB: gc thread: 0x00007fc694012800 [id: 765] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00429       98KB refills: 28 waste  1.3% gc: 160B slow: 936B fast: 0B
TLAB: gc thread: 0x00007fc6bc046000 [id: 764] desired_size: 3KB slow allocs: 4  refill waste: 48B alloc: 0.00466      106KB refills: 31 waste  3.4% gc: 2504B slow: 1088B fast: 0B
TLAB: gc thread: 0x00007fc6a402d000 [id: 763] desired_size: 3KB slow allocs: 4  refill waste: 48B alloc: 0.00442      101KB refills: 30 waste  3.7% gc: 2592B slow: 920B fast: 0B
TLAB: gc thread: 0x00007fc6b8060000 [id: 762] desired_size: 2KB slow allocs: 7  refill waste: 40B alloc: 0.00410       93KB refills: 36 waste  1.2% gc: 32B slow: 1272B fast: 0B
TLAB: gc thread: 0x00007fc6f4077000 [id: 761] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00446      101KB refills: 28 waste  1.2% gc: 192B slow: 872B fast: 0B
TLAB: gc thread: 0x00007fc6d0012800 [id: 760] desired_size: 3KB slow allocs: 3  refill waste: 48B alloc: 0.00435       99KB refills: 34 waste  2.7% gc: 1856B slow: 1040B fast: 0B
TLAB: gc thread: 0x00007fc694011000 [id: 759] desired_size: 3KB slow allocs: 4  refill waste: 48B alloc: 0.00441      100KB refills: 30 waste  3.2% gc: 2120B slow: 944B fast: 0B
TLAB: gc thread: 0x00007fc70c0aa000 [id: 758] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00426       97KB refills: 30 waste  1.2% gc: 136B slow: 960B fast: 0B
TLAB: gc thread: 0x00007fc6c4066000 [id: 757] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00439      100KB refills: 27 waste  2.0% gc: 928B slow: 816B fast: 0B
TLAB: gc thread: 0x00007fc69c06e000 [id: 756] desired_size: 3KB slow allocs: 13  refill waste: 48B alloc: 0.00434       99KB refills: 31 waste  1.2% gc: 120B slow: 1040B fast: 0B
TLAB: gc thread: 0x00007fc6c001f800 [id: 755] desired_size: 2KB slow allocs: 4  refill waste: 40B alloc: 0.00410       93KB refills: 32 waste  1.3% gc: 280B slow: 992B fast: 0B
TLAB: gc thread: 0x00007fc6c800b000 [id: 754] desired_size: 3KB slow allocs: 5  refill waste: 48B alloc: 0.00442      101KB refills: 32 waste  2.1% gc: 992B slow: 1192B fast: 0B
TLAB: gc thread: 0x00007fc6b4012800 [id: 753] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00433       98KB refills: 29 waste  1.6% gc: 504B slow: 984B fast: 0B
TLAB: gc thread: 0x00007fc6ec03c000 [id: 752] desired_size: 3KB slow allocs: 5  refill waste: 48B alloc: 0.00445      101KB refills: 30 waste  1.5% gc: 520B slow: 920B fast: 0B
TLAB: gc thread: 0x00007fc6cc013800 [id: 751] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00425       97KB refills: 33 waste  1.4% gc: 376B slow: 1024B fast: 0B
TLAB: gc thread: 0x00007fc6b0016000 [id: 750] desired_size: 3KB slow allocs: 12  refill waste: 48B alloc: 0.00431       98KB refills: 35 waste  1.1% gc: 24B slow: 1152B fast: 0B
TLAB: gc thread: 0x00007fc70c0a8000 [id: 749] desired_size: 3KB slow allocs: 7  refill waste: 48B alloc: 0.00442      101KB refills: 30 waste  1.5% gc: 448B slow: 1016B fast: 0B
TLAB: gc thread: 0x00007fc6c001d800 [id: 748] desired_size: 3KB slow allocs: 8  refill waste: 48B alloc: 0.00449      102KB refills: 28 waste  2.4% gc: 1120B slow: 1064B fast: 0B
TLAB: gc thread: 0x00007fc6c8009000 [id: 747] desired_size: 3KB slow allocs: 7  refill waste: 48B alloc: 0.00455      104KB refills: 30 waste  1.5% gc: 648B slow: 824B fast: 0B
TLAB: gc thread: 0x00007fc69c06c000 [id: 746] desired_size: 3KB slow allocs: 5  refill waste: 48B alloc: 0.00427       97KB refills: 37 waste  1.9% gc: 928B slow: 1288B fast: 0B
TLAB: gc thread: 0x00007fc69400e800 [id: 745] desired_size: 3KB slow allocs: 9  refill waste: 48B alloc: 0.00429       98KB refills: 36 waste  3.0% gc: 2088B slow: 1288B fast: 0B
TLAB: gc thread: 0x00007fc6c4077800 [id: 744] desired_size: 3KB slow allocs: 5  refill waste: 48B alloc: 0.00432       98KB refills: 31 waste  2.3% gc: 1336B slow: 904B fast: 0B
TLAB: gc thread: 0x0000000001821800 [id: 743] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00435       99KB refills: 26 waste  1.5% gc: 96B slow: 1104B fast: 0B
TLAB: gc thread: 0x00007fc6d0010800 [id: 742] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00446      101KB refills: 34 waste  3.4% gc: 2544B slow: 1128B fast: 0B
TLAB: gc thread: 0x00007fc69400d000 [id: 741] desired_size: 3KB slow allocs: 10  refill waste: 48B alloc: 0.00428       97KB refills: 32 waste  1.5% gc: 344B slow: 1104B fast: 0B
TLAB: gc thread: 0x00007fc6c001b800 [id: 740] desired_size: 3KB slow allocs: 10  refill waste: 48B alloc: 0.00459      104KB refills: 32 waste  1.4% gc: 152B slow: 1304B fast: 0B
TLAB: gc thread: 0x00007fc70c0a6000 [id: 739] desired_size: 3KB slow allocs: 7  refill waste: 48B alloc: 0.00454      103KB refills: 31 waste  1.2% gc: 248B slow: 960B fast: 0B
TLAB: gc thread: 0x00007fc6b0013800 [id: 738] desired_size: 2KB slow allocs: 9  refill waste: 40B alloc: 0.00422       96KB refills: 34 waste  4.0% gc: 3016B slow: 1136B fast: 0B
TLAB: gc thread: 0x00007fc69c06a000 [id: 737] desired_size: 2KB slow allocs: 12  refill waste: 40B alloc: 0.00423       96KB refills: 33 waste  1.2% gc: 208B slow: 1008B fast: 0B
TLAB: gc thread: 0x00007fc6cc011800 [id: 736] desired_size: 2KB slow allocs: 2  refill waste: 40B alloc: 0.00407       93KB refills: 29 waste  3.3% gc: 1920B slow: 880B fast: 0B
TLAB: gc thread: 0x00007fc6a402a800 [id: 735] desired_size: 2KB slow allocs: 8  refill waste: 40B alloc: 0.00415       94KB refills: 31 waste  3.7% gc: 2400B slow: 1008B fast: 0B
TLAB: gc thread: 0x00007fc6c8007000 [id: 734] desired_size: 3KB slow allocs: 3  refill waste: 48B alloc: 0.00451      103KB refills: 31 waste  2.7% gc: 1640B slow: 1056B fast: 0B
TLAB: gc thread: 0x000000000181f800 [id: 733] desired_size: 3KB slow allocs: 2  refill waste: 48B alloc: 0.00426       97KB refills: 24 waste  2.1% gc: 696B slow: 848B fast: 0B
TLAB: gc thread: 0x00007fc6d000e800 [id: 732] desired_size: 3KB slow allocs: 7  refill waste: 48B alloc: 0.00442      101KB refills: 33 waste  2.6% gc: 1544B slow: 1184B fast: 0B
TLAB: gc thread: 0x00007fc6bc043800 [id: 731] desired_size: 3KB slow allocs: 9  refill waste: 48B alloc: 0.00434       99KB refills: 34 waste  1.2% gc: 200B slow: 1048B fast: 0B
TLAB: gc thread: 0x00007fc6c4075800 [id: 730] desired_size: 3KB slow allocs: 3  refill waste: 48B alloc: 0.00449      102KB refills: 32 waste  2.5% gc: 1592B slow: 976B fast: 0B
TLAB: gc thread: 0x00007fc6f4075000 [id: 729] desired_size: 3KB slow allocs: 10  refill waste: 48B alloc: 0.00437       99KB refills: 31 waste  4.3% gc: 3064B slow: 1184B fast: 0B
TLAB: gc thread: 0x00007fc69400b000 [id: 728] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00443      101KB refills: 27 waste  2.9% gc: 1552B slow: 928B fast: 0B
TLAB: gc thread: 0x00007fc6b805d800 [id: 727] desired_size: 3KB slow allocs: 5  refill waste: 48B alloc: 0.00461      105KB refills: 28 waste  1.7% gc: 632B slow: 928B fast: 0B
TLAB: gc thread: 0x00007fc6b0012000 [id: 726] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00435       99KB refills: 24 waste  3.9% gc: 2192B slow: 760B fast: 0B
TLAB: gc thread: 0x00007fc6c8070800 [id: 725] desired_size: 3KB slow allocs: 9  refill waste: 48B alloc: 0.00436       99KB refills: 34 waste  4.1% gc: 3152B slow: 1272B fast: 0B
TLAB: gc thread: 0x00007fc6c4073800 [id: 724] desired_size: 3KB slow allocs: 7  refill waste: 48B alloc: 0.00441      100KB refills: 27 waste  2.4% gc: 1216B slow: 888B fast: 0B
TLAB: gc thread: 0x00007fc6cc00f800 [id: 723] desired_size: 2KB slow allocs: 3  refill waste: 40B alloc: 0.00424       96KB refills: 30 waste  2.4% gc: 1240B slow: 952B fast: 0B
TLAB: gc thread: 0x00007fc6bc041800 [id: 722] desired_size: 2KB slow allocs: 5  refill waste: 40B alloc: 0.00402       91KB refills: 37 waste  3.8% gc: 2808B slow: 1256B fast: 0B
TLAB: gc thread: 0x00007fc6ec039800 [id: 721] desired_size: 3KB slow allocs: 3  refill waste: 48B alloc: 0.00449      102KB refills: 31 waste  2.0% gc: 1184B slow: 880B fast: 0B
TLAB: gc thread: 0x00007fc6c001a000 [id: 720] desired_size: 3KB slow allocs: 4  refill waste: 48B alloc: 0.00429       98KB refills: 28 waste  3.6% gc: 2208B slow: 912B fast: 0B
TLAB: gc thread: 0x00007fc6cc00d800 [id: 719] desired_size: 3KB slow allocs: 9  refill waste: 48B alloc: 0.00438      100KB refills: 28 waste  1.7% gc: 656B slow: 824B fast: 0B
TLAB: gc thread: 0x00007fc6f4084800 [id: 718] desired_size: 3KB slow allocs: 5  refill waste: 48B alloc: 0.00442      101KB refills: 29 waste  3.5% gc: 2320B slow: 928B fast: 0B
TLAB: gc thread: 0x00007fc6c806e800 [id: 717] desired_size: 3KB slow allocs: 7  refill waste: 48B alloc: 0.00450      102KB refills: 30 waste  2.9% gc: 1936B slow: 928B fast: 0B
TLAB: gc thread: 0x00007fc6b0010000 [id: 716] desired_size: 3KB slow allocs: 7  refill waste: 48B alloc: 0.00425       97KB refills: 37 waste  2.4% gc: 1368B slow: 1408B fast: 0B
TLAB: gc thread: 0x00007fc6b4010800 [id: 715] desired_size: 3KB slow allocs: 7  refill waste: 48B alloc: 0.00463      105KB refills: 28 waste  2.3% gc: 1104B slow: 1016B fast: 0B
TLAB: gc thread: 0x00007fc6d000d000 [id: 714] desired_size: 3KB slow allocs: 8  refill waste: 48B alloc: 0.00460      105KB refills: 34 waste  1.8% gc: 864B slow: 1136B fast: 0B
TLAB: gc thread: 0x00007fc6cc00b800 [id: 713] desired_size: 3KB slow allocs: 5  refill waste: 48B alloc: 0.00435       99KB refills: 26 waste  4.3% gc: 2488B slow: 1000B fast: 0B
TLAB: gc thread: 0x00007fc6b805b800 [id: 712] desired_size: 3KB slow allocs: 7  refill waste: 48B alloc: 0.00448      102KB refills: 34 waste  1.6% gc: 424B slow: 1320B fast: 0B
TLAB: gc thread: 0x00007fc6c806c800 [id: 711] desired_size: 3KB slow allocs: 8  refill waste: 48B alloc: 0.00458      104KB refills: 25 waste  3.0% gc: 1600B slow: 920B fast: 0B
TLAB: gc thread: 0x00007fc6a4028800 [id: 710] desired_size: 3KB slow allocs: 13  refill waste: 48B alloc: 0.00453      103KB refills: 37 waste  2.9% gc: 2256B slow: 1256B fast: 0B
TLAB: gc thread: 0x00007fc69c068000 [id: 709] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00451      103KB refills: 29 waste  2.1% gc: 944B slow: 1016B fast: 0B
TLAB: gc thread: 0x00007fc6ec038000 [id: 708] desired_size: 3KB slow allocs: 5  refill waste: 48B alloc: 0.00447      102KB refills: 27 waste  3.5% gc: 2208B slow: 880B fast: 0B
TLAB: gc thread: 0x00007fc6c806b000 [id: 707] desired_size: 2KB slow allocs: 6  refill waste: 40B alloc: 0.00419       95KB refills: 27 waste  1.5% gc: 384B slow: 824B fast: 0B
TLAB: gc thread: 0x00007fc6bc03f800 [id: 706] desired_size: 2KB slow allocs: 7  refill waste: 40B alloc: 0.00410       93KB refills: 31 waste  2.3% gc: 1136B slow: 976B fast: 0B
TLAB: gc thread: 0x000000000181d800 [id: 705] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00457      104KB refills: 31 waste  2.0% gc: 984B slow: 1024B fast: 0B
TLAB: gc thread: 0x00007fc6b000d800 [id: 704] desired_size: 3KB slow allocs: 9  refill waste: 48B alloc: 0.00444      101KB refills: 27 waste  3.7% gc: 2160B slow: 1064B fast: 0B
TLAB: gc thread: 0x00007fc70c065000 [id: 703] desired_size: 3KB slow allocs: 7  refill waste: 48B alloc: 0.00430       98KB refills: 30 waste  2.0% gc: 816B slow: 1048B fast: 0B
TLAB: gc thread: 0x00007fc6f4082800 [id: 702] desired_size: 3KB slow allocs: 7  refill waste: 48B alloc: 0.00435       99KB refills: 29 waste  3.0% gc: 1608B slow: 1136B fast: 0B
TLAB: gc thread: 0x00007fc6d000b000 [id: 701] desired_size: 3KB slow allocs: 5  refill waste: 48B alloc: 0.00441      100KB refills: 31 waste  4.3% gc: 3160B slow: 1104B fast: 0B
TLAB: gc thread: 0x00007fc694009000 [id: 700] desired_size: 3KB slow allocs: 9  refill waste: 48B alloc: 0.00436       99KB refills: 32 waste  4.0% gc: 2944B slow: 1048B fast: 0B
TLAB: gc thread: 0x000000000181c000 [id: 699] desired_size: 3KB slow allocs: 7  refill waste: 48B alloc: 0.00440      100KB refills: 30 waste  4.2% gc: 3048B slow: 1000B fast: 0B
TLAB: gc thread: 0x00007fc6ec035000 [id: 698] desired_size: 3KB slow allocs: 7  refill waste: 48B alloc: 0.00435       99KB refills: 31 waste  3.3% gc: 2208B slow: 1000B fast: 0B
TLAB: gc thread: 0x00007fc6b000c000 [id: 697] desired_size: 3KB slow allocs: 9  refill waste: 48B alloc: 0.00460      105KB refills: 30 waste  4.1% gc: 2952B slow: 1128B fast: 0B
TLAB: gc thread: 0x00007fc6b400e800 [id: 696] desired_size: 2KB slow allocs: 4  refill waste: 40B alloc: 0.00419       95KB refills: 27 waste  4.3% gc: 2712B slow: 768B fast: 0B
TLAB: gc thread: 0x00007fc6bc03d000 [id: 695] desired_size: 3KB slow allocs: 9  refill waste: 48B alloc: 0.00445      101KB refills: 32 waste  3.2% gc: 2200B slow: 1080B fast: 0B
TLAB: gc thread: 0x00007fc6c0018000 [id: 694] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00439      100KB refills: 34 waste  1.8% gc: 824B slow: 1064B fast: 0B
TLAB: gc thread: 0x00007fc6b8059800 [id: 693] desired_size: 3KB slow allocs: 7  refill waste: 48B alloc: 0.00441      100KB refills: 30 waste  1.9% gc: 808B slow: 1056B fast: 0B
TLAB: gc thread: 0x00007fc6c4071000 [id: 692] desired_size: 3KB slow allocs: 8  refill waste: 48B alloc: 0.00431       98KB refills: 32 waste  2.4% gc: 1288B slow: 1112B fast: 0B
TLAB: gc thread: 0x000000000181a000 [id: 691] desired_size: 3KB slow allocs: 8  refill waste: 48B alloc: 0.00429       98KB refills: 31 waste  1.9% gc: 712B slow: 1096B fast: 0B
TLAB: gc thread: 0x00007fc6b400c000 [id: 690] desired_size: 3KB slow allocs: 10  refill waste: 48B alloc: 0.00445      101KB refills: 27 waste  1.3% gc: 216B slow: 936B fast: 0B
TLAB: gc thread: 0x00007fc694007000 [id: 689] desired_size: 2KB slow allocs: 4  refill waste: 40B alloc: 0.00413       94KB refills: 30 waste  2.5% gc: 1368B slow: 864B fast: 0B
TLAB: gc thread: 0x00007fc6a4027000 [id: 688] desired_size: 2KB slow allocs: 4  refill waste: 40B alloc: 0.00421       96KB refills: 33 waste  2.6% gc: 1616B slow: 1008B fast: 0B
TLAB: gc thread: 0x00007fc69c066000 [id: 687] desired_size: 3KB slow allocs: 5  refill waste: 48B alloc: 0.00445      101KB refills: 32 waste  2.5% gc: 1568B slow: 1032B fast: 0B
TLAB: gc thread: 0x00007fc70c063000 [id: 686] desired_size: 2KB slow allocs: 6  refill waste: 40B alloc: 0.00422       96KB refills: 32 waste  2.9% gc: 1736B slow: 1064B fast: 0B
TLAB: gc thread: 0x00007fc6d0008800 [id: 685] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00433       98KB refills: 29 waste  2.5% gc: 1296B slow: 960B fast: 0B
TLAB: gc thread: 0x00007fc6bc03b000 [id: 684] desired_size: 3KB slow allocs: 9  refill waste: 48B alloc: 0.00429       98KB refills: 33 waste  2.6% gc: 1456B slow: 1168B fast: 0B
TLAB: gc thread: 0x00007fc6c8060800 [id: 683] desired_size: 2KB slow allocs: 5  refill waste: 40B alloc: 0.00418       95KB refills: 35 waste  2.1% gc: 1144B slow: 1104B fast: 0B
TLAB: gc thread: 0x00007fc6b0009800 [id: 682] desired_size: 3KB slow allocs: 8  refill waste: 48B alloc: 0.00427       97KB refills: 30 waste  4.3% gc: 3056B slow: 928B fast: 0B
TLAB: gc thread: 0x00007fc6cc009000 [id: 681] desired_size: 3KB slow allocs: 4  refill waste: 48B alloc: 0.00441      100KB refills: 32 waste  1.5% gc: 640B slow: 912B fast: 0B
TLAB: gc thread: 0x00007fc6b8057000 [id: 680] desired_size: 3KB slow allocs: 7  refill waste: 48B alloc: 0.00440      100KB refills: 31 waste  1.7% gc: 656B slow: 1040B fast: 0B
TLAB: gc thread: 0x00007fc6bc039000 [id: 679] desired_size: 2KB slow allocs: 7  refill waste: 40B alloc: 0.00419       95KB refills: 33 waste  2.0% gc: 1024B slow: 1016B fast: 0B
TLAB: gc thread: 0x00007fc6c0015800 [id: 678] desired_size: 3KB slow allocs: 8  refill waste: 48B alloc: 0.00427       97KB refills: 31 waste  3.6% gc: 2360B slow: 1056B fast: 0B
TLAB: gc thread: 0x00007fc6ec02e000 [id: 677] desired_size: 3KB slow allocs: 8  refill waste: 48B alloc: 0.00454      103KB refills: 28 waste  3.0% gc: 1856B slow: 936B fast: 0B
TLAB: gc thread: 0x00007fc6c406f800 [id: 676] desired_size: 3KB slow allocs: 5  refill waste: 48B alloc: 0.00435       99KB refills: 36 waste  3.7% gc: 3080B slow: 1160B fast: 0B
TLAB: gc thread: 0x00007fc6d0006800 [id: 675] desired_size: 3KB slow allocs: 11  refill waste: 48B alloc: 0.00454      103KB refills: 33 waste  3.9% gc: 3088B slow: 1144B fast: 0B
TLAB: gc thread: 0x00007fc6c0014000 [id: 674] desired_size: 3KB slow allocs: 9  refill waste: 48B alloc: 0.00442      101KB refills: 28 waste  1.1% gc: 136B slow: 872B fast: 0B
TLAB: gc thread: 0x00007fc6bc037800 [id: 673] desired_size: 3KB slow allocs: 5  refill waste: 48B alloc: 0.00455      103KB refills: 34 waste  2.4% gc: 1520B slow: 1128B fast: 0B
TLAB: gc thread: 0x00007fc6b8055800 [id: 672] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00427       97KB refills: 31 waste  1.5% gc: 296B slow: 1096B fast: 0B
TLAB: gc thread: 0x0000000001818000 [id: 671] desired_size: 3KB slow allocs: 8  refill waste: 48B alloc: 0.00435       99KB refills: 34 waste  1.8% gc: 672B slow: 1296B fast: 0B
TLAB: gc thread: 0x00007fc6b0007800 [id: 670] desired_size: 3KB slow allocs: 4  refill waste: 48B alloc: 0.00455      103KB refills: 27 waste  2.6% gc: 1488B slow: 856B fast: 0B
TLAB: gc thread: 0x00007fc6a4024800 [id: 669] desired_size: 3KB slow allocs: 10  refill waste: 48B alloc: 0.00449      102KB refills: 31 waste  1.1% gc: 64B slow: 1040B fast: 0B
TLAB: gc thread: 0x00007fc6cc007000 [id: 668] desired_size: 2KB slow allocs: 3  refill waste: 40B alloc: 0.00404       92KB refills: 31 waste  0.9% gc: 16B slow: 816B fast: 0B
TLAB: gc thread: 0x00007fc6d0004800 [id: 667] desired_size: 3KB slow allocs: 5  refill waste: 48B alloc: 0.00450      102KB refills: 29 waste  2.6% gc: 1456B slow: 1040B fast: 0B
TLAB: gc thread: 0x00007fc694004800 [id: 666] desired_size: 2KB slow allocs: 5  refill waste: 40B alloc: 0.00413       94KB refills: 32 waste  3.4% gc: 2264B slow: 1016B fast: 0B
TLAB: gc thread: 0x00007fc6c0011800 [id: 665] desired_size: 3KB slow allocs: 4  refill waste: 48B alloc: 0.00429       97KB refills: 31 waste  1.7% gc: 568B slow: 1088B fast: 0B
TLAB: gc thread: 0x00007fc70c061000 [id: 664] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00435       99KB refills: 38 waste  2.2% gc: 1480B slow: 1128B fast: 0B
TLAB: gc thread: 0x00007fc6bc035800 [id: 663] desired_size: 3KB slow allocs: 13  refill waste: 48B alloc: 0.00434       99KB refills: 32 waste  4.1% gc: 2960B slow: 1168B fast: 0B
TLAB: gc thread: 0x00007fc69c063800 [id: 662] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00437       99KB refills: 30 waste  3.2% gc: 2144B slow: 888B fast: 0B
TLAB: gc thread: 0x00007fc6b400a000 [id: 661] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00454      103KB refills: 32 waste  1.5% gc: 552B slow: 1016B fast: 0B
TLAB: gc thread: 0x00007fc6ec02c800 [id: 660] desired_size: 3KB slow allocs: 8  refill waste: 48B alloc: 0.00437       99KB refills: 34 waste  2.0% gc: 1040B slow: 1128B fast: 0B
TLAB: gc thread: 0x00007fc6bc034800 [id: 659] desired_size: 3KB slow allocs: 5  refill waste: 48B alloc: 0.00466      106KB refills: 32 waste  1.7% gc: 672B slow: 1136B fast: 0B
TLAB: gc thread: 0x00007fc6b4008000 [id: 658] desired_size: 3KB slow allocs: 8  refill waste: 48B alloc: 0.00441      100KB refills: 30 waste  1.7% gc: 656B slow: 944B fast: 0B
TLAB: gc thread: 0x00007fc6f4080800 [id: 657] desired_size: 3KB slow allocs: 3  refill waste: 48B alloc: 0.00449      102KB refills: 25 waste  4.2% gc: 2576B slow: 848B fast: 0B
TLAB: gc thread: 0x00007fc6b0005800 [id: 656] desired_size: 3KB slow allocs: 9  refill waste: 48B alloc: 0.00426       97KB refills: 32 waste  1.1% gc: 128B slow: 1000B fast: 0B
TLAB: gc thread: 0x00007fc6b8053800 [id: 655] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00430       98KB refills: 29 waste  1.5% gc: 432B slow: 888B fast: 0B
TLAB: gc thread: 0x00007fc6ec02a800 [id: 654] desired_size: 2KB slow allocs: 6  refill waste: 40B alloc: 0.00421       96KB refills: 29 waste  1.7% gc: 696B slow: 832B fast: 0B
TLAB: gc thread: 0x00007fc6c8060000 [id: 653] desired_size: 2KB slow allocs: 6  refill waste: 40B alloc: 0.00418       95KB refills: 36 waste  2.3% gc: 1384B slow: 1128B fast: 0B
TLAB: gc thread: 0x00007fc6a4023000 [id: 652] desired_size: 2KB slow allocs: 7  refill waste: 40B alloc: 0.00420       96KB refills: 37 waste  3.6% gc: 2720B slow: 1328B fast: 0B
TLAB: gc thread: 0x0000000001815800 [id: 651] desired_size: 3KB slow allocs: 7  refill waste: 48B alloc: 0.00431       98KB refills: 35 waste  3.4% gc: 2392B slow: 1344B fast: 0B
TLAB: gc thread: 0x00007fc6c0010800 [id: 650] desired_size: 2KB slow allocs: 9  refill waste: 40B alloc: 0.00424       97KB refills: 34 waste  3.8% gc: 2896B slow: 1072B fast: 0B
TLAB: gc thread: 0x00007fc70c05f000 [id: 649] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00446      102KB refills: 27 waste  1.1% gc: 112B slow: 880B fast: 0B
TLAB: gc thread: 0x00007fc6cc005000 [id: 648] desired_size: 2KB slow allocs: 5  refill waste: 40B alloc: 0.00420       96KB refills: 30 waste  0.9% gc: 64B slow: 800B fast: 0B
TLAB: gc thread: 0x00007fc694003000 [id: 647] desired_size: 3KB slow allocs: 4  refill waste: 48B alloc: 0.00439      100KB refills: 33 waste  1.7% gc: 736B slow: 1064B fast: 0B
TLAB: gc thread: 0x00007fc69c062000 [id: 646] desired_size: 3KB slow allocs: 5  refill waste: 48B alloc: 0.00426       97KB refills: 31 waste  4.1% gc: 2944B slow: 1016B fast: 0B
TLAB: gc thread: 0x00007fc6f407e000 [id: 645] desired_size: 3KB slow allocs: 5  refill waste: 48B alloc: 0.00446      102KB refills: 30 waste  2.2% gc: 1040B slow: 1072B fast: 0B
TLAB: gc thread: 0x00007fc6ec028000 [id: 644] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00434       99KB refills: 30 waste  2.4% gc: 1256B slow: 1032B fast: 0B
TLAB: gc thread: 0x00007fc6cc003000 [id: 643] desired_size: 3KB slow allocs: 9  refill waste: 48B alloc: 0.00431       98KB refills: 30 waste  1.9% gc: 728B slow: 1008B fast: 0B
TLAB: gc thread: 0x00007fc6d0001800 [id: 642] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00442      101KB refills: 29 waste  1.5% gc: 552B slow: 880B fast: 0B
TLAB: gc thread: 0x00007fc6c4091000 [id: 641] desired_size: 3KB slow allocs: 7  refill waste: 48B alloc: 0.00427       97KB refills: 35 waste  2.2% gc: 1200B slow: 1120B fast: 0B
TLAB: gc thread: 0x00007fc6c80b8000 [id: 639] desired_size: 3KB slow allocs: 9  refill waste: 48B alloc: 0.00442      100KB refills: 35 waste  3.6% gc: 2816B slow: 1224B fast: 0B
TLAB: gc thread: 0x00007fc6b0004000 [id: 637] desired_size: 2KB slow allocs: 8  refill waste: 40B alloc: 0.00424       97KB refills: 34 waste  1.5% gc: 392B slow: 1136B fast: 0B
TLAB: gc thread: 0x00007fc6b8052000 [id: 636] desired_size: 3KB slow allocs: 8  refill waste: 48B alloc: 0.00454      103KB refills: 31 waste  1.6% gc: 584B slow: 1064B fast: 0B
TLAB: gc thread: 0x00007fc6bc028800 [id: 635] desired_size: 3KB slow allocs: 6  refill waste: 48B alloc: 0.00439      100KB refills: 27 waste  2.1% gc: 904B slow: 872B fast: 0B
TLAB: gc thread: 0x00007fc6b4006800 [id: 634] desired_size: 2KB slow allocs: 6  refill waste: 40B alloc: 0.00417       95KB refills: 31 waste  4.3% gc: 2976B slow: 1040B fast: 0B
TLAB: gc thread: 0x00007fc6a4021800 [id: 633] desired_size: 2KB slow allocs: 5  refill waste: 40B alloc: 0.00424       96KB refills: 30 waste  2.5% gc: 1472B slow: 816B fast: 0B
TLAB: gc thread: 0x00007fc6c000d800 [id: 632] desired_size: 3KB slow allocs: 7  refill waste: 48B alloc: 0.00457      104KB refills: 25 waste  3.3% gc: 1880B slow: 824B fast: 0B
TLAB: gc thread: 0x00007fc69c061000 [id: 631] desired_size: 3KB slow allocs: 9  refill waste: 48B alloc: 0.00428       97KB refills: 34 waste  2.2% gc: 1160B slow: 1120B fast: 0B
TLAB: gc thread: 0x00007fc764018800 [id: 624] desired_size: 12KB slow allocs: 4  refill waste: 200B alloc: 0.01775      404KB refills: 25 waste  3.5% gc: 10360B slow: 816B fast: 48B
TLAB: gc thread: 0x00007fc764017000 [id: 623] desired_size: 47KB slow allocs: 0  refill waste: 752B alloc: 0.06673     1518KB refills: 33 waste  1.6% gc: 25240B slow: 1032B fast: 0B
TLAB: gc thread: 0x00007fc694001000 [id: 622] desired_size: 3KB slow allocs: 7  refill waste: 48B alloc: 0.00425       97KB refills: 32 waste  1.1% gc: 88B slow: 992B fast: 0B
TLAB: gc thread: 0x00007fc764015800 [id: 621] desired_size: 47KB slow allocs: 0  refill waste: 752B alloc: 0.06686     1521KB refills: 34 waste  2.8% gc: 44816B slow: 1104B fast: 0B
TLAB: gc thread: 0x00007fc764015000 [id: 620] desired_size: 47KB slow allocs: 0  refill waste: 752B alloc: 0.06696     1523KB refills: 34 waste  3.0% gc: 47240B slow: 1464B fast: 0B
TLAB: gc thread: 0x0000000001814800 [id: 619] desired_size: 2KB slow allocs: 9  refill waste: 40B alloc: 0.00415       94KB refills: 33 waste  4.1% gc: 2936B slow: 1112B fast: 0B
TLAB: gc thread: 0x00007fc76400e800 [id: 618] desired_size: 47KB slow allocs: 0  refill waste: 752B alloc: 0.06704     1525KB refills: 33 waste  0.4% gc: 5272B slow: 1072B fast: 0B
TLAB: gc thread: 0x00007fc70c054000 [id: 617] desired_size: 3KB slow allocs: 8  refill waste: 48B alloc: 0.00438      100KB refills: 27 waste  2.3% gc: 976B slow: 952B fast: 0B
TLAB: gc thread: 0x00007fc70c050800 [id: 582] desired_size: 3KB slow allocs: 5  refill waste: 48B alloc: 0.00436       99KB refills: 28 waste  1.4% gc: 176B slow: 1080B fast: 0B
TLAB: gc thread: 0x00007fc6f805f000 [id: 443] desired_size: 2KB slow allocs: 0  refill waste: 32B alloc: 0.00014        3KB refills: 5 waste  4.3% gc: 352B slow: 96B fast: 0B
TLAB: gc thread: 0x00007fc728241800 [id: 328] desired_size: 2KB slow allocs: 0  refill waste: 32B alloc: 0.00263       60KB refills: 1 waste 44.2% gc: 912B slow: 0B fast: 0B
TLAB: gc thread: 0x00007fc73c56a800 [id: 282] desired_size: 9KB slow allocs: 0  refill waste: 152B alloc: 0.01413      321KB refills: 1 waste 88.2% gc: 9008B slow: 0B fast: 0B
TLAB: gc thread: 0x00007fc73c567800 [id: 281] desired_size: 4KB slow allocs: 0  refill waste: 64B alloc: 0.00597      136KB refills: 1 waste 98.9% gc: 4264B slow: 0B fast: 0B
TLAB: gc thread: 0x00007fc73c4a4000 [id: 279] desired_size: 13KB slow allocs: 0  refill waste: 208B alloc: 0.01852      421KB refills: 1 waste 99.3% gc: 13304B slow: 0B fast: 0B
TLAB: gc thread: 0x00007fc73803f000 [id: 271] desired_size: 2KB slow allocs: 0  refill waste: 40B alloc: 0.00372       85KB refills: 1 waste 90.5% gc: 2432B slow: 0B fast: 0B
TLAB: gc thread: 0x00007fc73803a800 [id: 270] desired_size: 2KB slow allocs: 1  refill waste: 32B alloc: 0.00031        7KB refills: 9 waste  2.1% gc: 136B slow: 232B fast: 16B
TLAB: gc thread: 0x00007fc72c11a000 [id: 267] desired_size: 2KB slow allocs: 0  refill waste: 32B alloc: 0.00013        3KB refills: 7 waste  0.6% gc: 48B slow: 16B fast: 16B
TLAB: gc thread: 0x00007fc73007d800 [id: 266] desired_size: 2KB slow allocs: 4  refill waste: 32B alloc: 0.00163       37KB refills: 31 waste  4.2% gc: 1840B slow: 728B fast: 88B
TLAB: gc thread: 0x00007fc7300c1000 [id: 225] desired_size: 2KB slow allocs: 0  refill waste: 32B alloc: 0.00269       61KB refills: 1 waste 100.0% gc: 2064B slow: 0B fast: 0B
TLAB: gc thread: 0x00007fc70c14b800 [id: 224] desired_size: 2KB slow allocs: 6  refill waste: 32B alloc: 0.00112       25KB refills: 31 waste  4.2% gc: 1904B slow: 688B fast: 120B
TLAB: gc thread: 0x00007fc728007000 [id: 209] desired_size: 26KB slow allocs: 0  refill waste: 424B alloc: 0.03771      858KB refills: 1 waste 99.6% gc: 27160B slow: 0B fast: 0B
TLAB: gc thread: 0x00007fc7b8227000 [id: 196] desired_size: 40KB slow allocs: 0  refill waste: 648B alloc: 0.05801     1320KB refills: 1 waste 99.7% gc: 41856B slow: 0B fast: 0B
TLAB totals: thrds: 181  refills: 5230 max: 38 slow allocs: 1093 max 13 waste:  2.8% gc: 461904B max: 47240B slow: 173544B max: 1464B fast: 488B max: 120B
2017-08-01T10:06:50.679+0000: [GC (System.gc()) 
Desired survivor size 33554432 bytes, new threshold 3 (max 15)
[PSYoungGen: 26076K->5312K(72192K)] 86627K->65870K(282624K), 0.0141440 secs] [Times: user=0.02 sys=0.01, real=0.02 secs] 
2017-08-01T10:06:50.694+0000: [Full GC (System.gc()) [PSYoungGen: 5312K->0K(72192K)] [ParOldGen: 60558K->50324K(210432K)] 65870K->50324K(282624K), [Metaspace: 65689K->65689K(1110016K)], 0.1561113 secs] [Times: user=0.52 sys=0.05, real=0.16 secs] 
2017-08-01T10:06:50.863+0000: Total time for which application threads were stopped: 0.1985033 seconds, Stopping threads took: 0.0037365 seconds
2017-08-01T10:07:04.502+0000: Total time for which application threads were stopped: 2.5434673 seconds, Stopping threads took: 0.0128751 seconds
TLAB: gc thread: 0x00007fc6f4088000 [id: 975] desired_size: 2KB slow allocs: 0  refill waste: 32B alloc: 0.00120        0KB refills: 10 waste  4.9% gc: 784B slow: 168B fast: 56B
TLAB: gc thread: 0x00007fc6b8059800 [id: 693] desired_size: 3KB slow allocs: 3  refill waste: 48B alloc: 0.00432        1KB refills: 4 waste 10.4% gc: 1240B slow: 40B fast: 88B
TLAB: gc thread: 0x00007fc6f805f000 [id: 443] desired_size: 2KB slow allocs: 0  refill waste: 32B alloc: 0.00022        0KB refills: 10 waste  3.5% gc: 496B slow: 168B fast: 56B
TLAB: gc thread: 0x00007fc6e002e800 [id: 396] desired_size: 2KB slow allocs: 0  refill waste: 32B alloc: 0.00107        0KB refills: 1 waste 98.4% gc: 2032B slow: 0B fast: 0B
TLAB: gc thread: 0x00007fc720002000 [id: 332] desired_size: 2KB slow allocs: 0  refill waste: 32B alloc: 0.00252        1KB refills: 1 waste 100.0% gc: 2064B slow: 0B fast: 0B
TLAB: gc thread: 0x00007fc73803a800 [id: 270] desired_size: 2KB slow allocs: 1  refill waste: 32B alloc: 0.00045        0KB refills: 8 waste 10.0% gc: 1376B slow: 280B fast: 0B
TLAB: gc thread: 0x00007fc72c11a000 [id: 267] desired_size: 2KB slow allocs: 0  refill waste: 32B alloc: 0.00029        0KB refills: 7 waste  6.6% gc: 944B slow: 16B fast: 0B
TLAB: gc thread: 0x00007fc73007d800 [id: 266] desired_size: 2KB slow allocs: 4  refill waste: 32B alloc: 0.00200        1KB refills: 31 waste  4.1% gc: 1808B slow: 728B fast: 88B
TLAB: gc thread: 0x00007fc7300c1000 [id: 225] desired_size: 2KB slow allocs: 0  refill waste: 32B alloc: 0.00175        0KB refills: 1 waste 98.4% gc: 2032B slow: 0B fast: 0B
TLAB: gc thread: 0x00007fc70c14b800 [id: 224] desired_size: 2KB slow allocs: 8  refill waste: 32B alloc: 0.00167        0KB refills: 61 waste  2.8% gc: 1904B slow: 1416B fast: 152B
TLAB totals: thrds: 10  refills: 134 max: 61 slow allocs: 16 max 8 waste:  6.4% gc: 14680B max: 2064B slow: 2816B max: 1416B fast: 440B max: 152B
2017-08-01T10:07:07.702+0000: [GC (System.gc()) 
Desired survivor size 33030144 bytes, new threshold 4 (max 15)
[PSYoungGen: 282K->224K(70144K)] 50606K->50548K(280576K), 2.6367725 secs] [Times: user=0.08 sys=9.35, real=2.65 secs] 
2017-08-01T10:07:10.509+0000: [Full GC (System.gc()) *** JBossAS process (179) received KILL signal ***

It became clear that the JVM was killed while it was reclaiming space. Unfortunately it took a little while and OOM killer kicked in.

The only missing bit is why it happened with Rocks DB? The answer is in System.gc() called while reinitializing the database. The test harness used cache.clear() method which cased that.

The bottom line: if you are using persistent Docker image, please make sure you're not calling cache.clear() too often or (this is suggested solution) increase memory limits.

Attachments

Issue Links

is related to

ISPN-8198 RocksDB should not call System.gc() on non-windows systems

Closed

Activity

People

Assignee:: Sebastian Łaskawiec (Inactive)

Reporter:: Sebastian Łaskawiec (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 2017/08/01 6:27 AM

Updated:: 2017/08/10 11:09 AM

Resolved:: 2017/08/01 6:27 AM