Issue with Elasticsearch

scottalanmiller

Maybe there is a memory issue? Do you have sar running (via sysstat) to see what the system was doing at the time?

Ambarishrh

I need to install that and check, on elastic forum i saw a comment:
I have seen these types of issues because the heap size was not big enough. It WILL just die and you will not know what happened.

And as per the doc,https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html its not advisable to set the default heap size. Thinking of testing this now.

scottalanmiller

Had you set the heap size already? Or are you looking at increasing it now?

Ambarishrh

Just increased, to 2 gb just to check and so far its been running well. Will keep monitoring this.

Ambarishrh

It failed again!

scottalanmiller

What does SAR show?

Ambarishrh

Is it sar -v?

scottalanmiller

Output of sar and sar -r would be best.

Ambarishrh

@scottalanmiller said:

sar

sar

Linux 3.10.0-329.7.2.lve1.3.58.el7.x86_64 01/21/2016 x86_64 (4 CPU)

12:00:01 AM 12:10:01 AM all 12:20:01 AM all 12:30:02 AM all 12:40:01 AM all 12:50:01 AM all 01:00:01 AM all 01:10:01 AM all 01:20:01 AM all 01:30:01 AM all 01:40:01 AM all 01:50:01 AM all 02:00:01 AM all 02:10:01 AM all 02:20:01 AM all 02:30:01 AM all 02:40:01 AM all 02:50:02 AM all 03:00:01 AM all 03:10:01 AM all 03:20:01 AM all 03:30:01 AM all 03:40:01 AM all 03:50:02 AM all 04:00:02 AM all 04:10:01 AM all 04:20:01 AM all 04:30:01 AM all 04:40:01 AM all 04:50:01 AM all 05:00:01 AM all 05:10:01 AM all 05:20:01 AM all 05:30:01 AM all 05:40:01 AM all 05:50:01 AM all 06:00:01 AM all 06:10:02 AM all 06:20:02 AM all 06:30:01 AM all 06:40:01 AM all 06:50:01 AM all CPU %user %nice %system %iowait %steal %idle
0.44 0.01 0.16 0.07 0.00 99.31
0.63 0.01 0.21 0.04 0.00 99.10
1.03 0.01 0.29 0.06 0.00 98.61
2.61 0.01 0.61 0.28 0.00 96.49
0.49 0.02 0.17 0.07 0.00 99.25
0.42 0.01 0.15 0.03 0.00 99.38
0.34 0.02 0.14 0.07 0.00 99.44
0.32 0.01 0.13 0.03 0.00 99.51
0.31 0.01 0.12 0.05 0.00 99.51
0.33 0.02 0.13 0.05 0.00 99.48
0.31 0.01 0.13 0.04 0.00 99.51
0.34 0.01 0.13 0.05 0.00 99.47
0.34 0.01 0.14 0.06 0.00 99.46
0.32 0.01 0.13 0.06 0.00 99.48
0.35 0.02 0.14 0.04 0.00 99.45
0.33 0.01 0.13 0.04 0.00 99.49
0.32 0.01 0.13 0.04 0.00 99.51
0.31 0.01 0.12 0.03 0.00 99.53
0.33 0.01 0.13 0.05 0.00 99.47
0.32 0.01 0.13 0.04 0.00 99.50
0.33 0.02 0.13 0.03 0.00 99.49
0.35 0.25 0.18 0.04 0.00 99.18
0.32 0.41 0.18 0.06 0.00 99.04
0.40 0.01 0.14 0.03 0.00 99.42
0.33 0.01 0.13 0.03 0.00 99.50
0.32 0.02 0.12 0.03 0.00 99.51
0.35 0.01 0.12 0.05 0.00 99.47
0.28 0.01 0.12 0.03 0.00 99.55
0.27 0.01 0.12 0.03 0.00 99.57
0.31 0.01 0.13 0.03 0.00 99.52
0.34 0.03 0.14 0.06 0.00 99.43
0.31 0.01 0.12 0.03 0.00 99.52
0.32 0.01 0.14 0.03 0.00 99.50
0.33 0.01 0.13 0.03 0.00 99.49
0.32 0.01 0.12 0.03 0.00 99.52
0.31 0.02 0.12 0.04 0.00 99.51
0.33 0.01 0.13 0.04 0.00 99.48
0.33 0.01 0.13 0.03 0.00 99.50
0.32 0.01 0.13 0.04 0.00 99.50
0.33 0.01 0.12 0.04 0.00 99.49
0.32 0.02 0.13 0.03 0.00 99.50

06:50:01 AM CPU %user %nice %system %iowait %steal %idle
07:00:02 AM all 0.31 0.01 0.13 0.03 0.00 99.52
07:10:01 AM all 0.36 0.01 0.14 0.05 0.00 99.44
07:20:01 AM all 0.32 0.02 0.13 0.81 0.00 98.72
07:30:01 AM all 0.32 0.01 0.13 0.03 0.00 99.51
07:40:01 AM all 0.33 0.02 0.13 0.03 0.00 99.49
07:50:01 AM all 0.32 0.01 0.13 0.03 0.00 99.51
08:00:01 AM all 0.35 0.01 0.14 0.04 0.00 99.47
08:10:01 AM all 1.11 0.01 0.29 0.09 0.00 98.49
08:20:01 AM all 0.84 0.01 0.27 0.05 0.00 98.82
08:30:01 AM all 0.45 0.02 0.16 0.07 0.00 99.29
08:40:01 AM all 0.62 0.01 0.19 0.05 0.00 99.13
08:50:01 AM all 3.06 0.01 0.94 0.08 0.00 95.92
09:00:01 AM all 1.22 0.01 0.32 0.06 0.00 98.39
09:10:01 AM all 1.88 0.01 0.56 0.08 0.00 97.47
09:20:01 AM all 1.71 0.03 0.49 0.19 0.00 97.59
09:30:01 AM all 2.90 0.01 0.78 0.56 0.00 95.75
09:40:01 AM all 2.58 0.01 0.68 0.13 0.00 96.60
09:50:01 AM all 2.37 0.01 0.59 0.13 0.00 96.90
10:00:01 AM all 2.27 0.01 0.65 0.12 0.00 96.95
10:10:01 AM all 2.19 0.02 0.59 0.15 0.00 97.05
10:20:01 AM all 2.17 0.01 0.54 0.21 0.00 97.06
10:30:01 AM all 3.82 0.01 1.01 2.10 0.00 93.06
10:40:01 AM all 5.28 0.01 1.43 7.57 0.00 85.72
10:50:01 AM all 3.47 0.01 0.90 9.17 0.00 86.46
11:00:01 AM all 5.07 0.02 1.09 15.09 0.00 78.72
11:10:01 AM all 5.19 0.02 1.22 17.64 0.00 75.93
11:20:01 AM all 3.35 0.01 0.82 14.19 0.00 81.62
11:30:01 AM all 3.14 0.01 0.77 16.37 0.00 79.70
11:40:02 AM all 2.90 0.01 0.66 2.57 0.00 93.86
11:50:01 AM all 2.18 0.02 0.51 0.35 0.00 96.93
12:00:01 PM all 2.03 0.01 0.48 0.23 0.00 97.25
12:10:01 PM all 2.27 0.01 0.55 0.43 0.00 96.74
12:20:01 PM all 1.96 0.01 0.47 0.74 0.00 96.81
12:30:01 PM all 3.52 0.01 0.80 0.82 0.00 94.86
12:40:01 PM all 4.09 0.02 0.99 1.77 0.00 93.12
12:50:01 PM all 4.67 0.01 0.99 2.44 0.00 91.89
01:00:01 PM all 3.67 0.01 0.87 2.23 0.00 93.22
01:10:01 PM all 3.89 0.02 0.90 7.38 0.00 87.82
01:20:01 PM all 3.98 0.01 0.89 11.28 0.00 83.84
01:30:02 PM all 5.36 0.02 1.30 17.06 0.00 76.26
01:40:01 PM all 4.11 0.01 0.90 27.56 0.00 67.42

01:40:01 PM CPU %user %nice %system %iowait %steal %idle
01:50:01 PM all 4.35 0.01 0.99 32.19 0.00 62.46
02:00:01 PM all 4.01 0.01 0.94 32.89 0.00 62.15
02:10:01 PM all 4.50 0.01 1.03 40.93 0.00 53.54
02:20:01 PM all 4.47 0.02 1.01 37.02 0.00 57.48
02:30:01 PM all 3.53 0.01 0.91 35.38 0.00 60.17
02:40:01 PM all 3.53 0.01 0.84 36.42 0.00 59.20
02:50:01 PM all 3.74 0.01 0.89 33.36 0.00 62.00
03:00:01 PM all 3.74 0.01 0.87 36.47 0.00 58.91
03:10:01 PM all 3.77 0.03 0.90 29.90 0.00 65.41
03:20:01 PM all 3.89 0.01 0.94 33.49 0.00 61.68
03:30:01 PM all 4.21 0.01 0.97 34.12 0.00 60.69
03:40:01 PM all 3.92 0.01 0.92 32.45 0.00 62.69
03:50:01 PM all 5.17 0.01 1.11 36.15 0.00 57.56
04:00:01 PM all 3.41 0.02 0.83 28.26 0.00 67.48
04:10:01 PM all 3.22 0.01 0.77 26.23 0.00 69.76
04:20:01 PM all 3.27 0.14 0.78 26.05 0.00 69.76
04:30:01 PM all 4.45 0.01 1.16 30.20 0.00 64.18
04:40:01 PM all 2.94 0.01 0.71 26.44 0.00 69.89
04:50:01 PM all 2.77 0.02 0.66 23.58 0.00 72.97
05:00:01 PM all 3.38 0.01 0.81 27.17 0.00 68.63
05:10:01 PM all 4.01 0.01 0.96 27.31 0.00 67.71
05:20:01 PM all 4.10 0.02 1.02 26.95 0.00 67.92
05:30:01 PM all 3.75 0.01 0.88 27.31 0.00 68.05
05:40:01 PM all 3.92 0.01 0.93 29.09 0.00 66.04
05:50:01 PM all 3.84 0.02 0.95 25.08 0.00 70.10
06:00:01 PM all 3.54 0.01 0.89 21.13 0.00 74.44
06:10:01 PM all 2.59 0.01 0.65 17.90 0.00 78.84
06:20:01 PM all 3.87 0.01 0.91 21.03 0.00 74.18
06:30:01 PM all 2.33 0.01 0.60 15.90 0.00 81.16
06:40:01 PM all 2.96 0.02 0.75 18.03 0.00 78.23
06:50:01 PM all 3.90 0.01 0.88 25.25 0.00 69.95
07:00:01 PM all 2.42 0.01 0.60 14.81 0.00 82.16
07:10:01 PM all 2.50 0.01 0.63 13.39 0.00 83.47
07:20:01 PM all 1.43 0.02 0.42 10.56 0.00 87.57
07:30:01 PM all 1.84 0.02 0.52 10.98 0.00 86.64
07:40:01 PM all 1.68 0.01 0.49 9.95 0.00 87.86
07:50:01 PM all 1.60 0.01 0.46 8.79 0.00 89.14
08:00:01 PM all 1.43 0.01 0.45 7.08 0.00 91.03
08:10:01 PM all 1.19 0.01 0.41 6.17 0.00 92.22
08:20:01 PM all 1.20 0.02 0.40 6.89 0.00 91.49
08:30:01 PM all 1.10 0.01 0.38 5.96 0.00 92.55

08:30:01 PM CPU %user %nice %system %iowait %steal %idle
08:40:01 PM all 1.12 0.01 0.39 6.17 0.00 92.30
08:50:01 PM all 1.57 0.01 0.47 5.42 0.00 92.52
09:00:01 PM all 0.81 0.01 0.32 3.35 0.00 95.52
09:10:01 PM all 0.50 0.03 0.25 2.72 0.00 96.50
09:20:01 PM all 0.49 0.01 0.24 3.41 0.00 95.84
09:30:01 PM all 0.54 0.01 0.23 2.79 0.00 96.43
09:40:01 PM all 0.45 0.01 0.20 1.09 0.00 98.25
09:50:01 PM all 0.59 0.01 0.22 1.20 0.00 97.97
10:00:01 PM all 0.68 0.02 0.26 1.14 0.00 97.90
10:10:01 PM all 0.51 0.01 0.23 1.18 0.00 98.07
Average: all 1.93 0.02 0.50 8.85 0.00 88.69

scottalanmiller

how many vCPU is that? It's definitely very busy, but not crazy.

Ambarishrh

sar -r
Linux 3.10.0-329.7.2.lve1.3.58.el7.x86_64 01/21/2016 x86_64 (4 CPU)

12:00:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
12:10:01 AM 6025324 10239408 62.95 372368 8778188 970508 5.29 2961360 6530568 96
12:20:01 AM 6001740 10262992 63.10 372748 8778568 1008368 5.49 2982276 6530608 60
12:30:02 AM 5998660 10266072 63.12 373212 8779704 1017304 5.54 2985736 6529980 20
12:40:01 AM 5997960 10266772 63.12 374824 8793372 995480 5.42 2980552 6535876 24
12:50:01 AM 5988956 10275776 63.18 375212 8793472 1012052 5.51 2990420 6535240 236
01:00:01 AM 5991172 10273560 63.16 375508 8793940 997732 5.43 2989224 6534632 28
01:10:01 AM 6000544 10264188 63.11 375764 8794132 977256 5.32 2982180 6534588 4
01:20:01 AM 5997780 10266952 63.12 375884 8794520 979552 5.33 2984448 6534340 60
01:30:01 AM 6000132 10264600 63.11 376008 8794480 976636 5.32 2981644 6534168 68
01:40:01 AM 6001280 10263452 63.10 376096 8794644 973124 5.30 2981508 6534036 40
01:50:01 AM 6000664 10264068 63.11 376180 8794816 974588 5.31 2982244 6533876 112
02:00:01 AM 6002492 10262240 63.10 376304 8795508 973352 5.30 2979476 6533764 144
02:10:01 AM 6001188 10263544 63.10 376424 8795480 973612 5.30 2981840 6533596 100
02:20:01 AM 5987324 10277408 63.19 376552 8795752 999096 5.44 2994672 6533220 128
02:30:01 AM 6001600 10263132 63.10 376712 8795904 970548 5.29 2981628 6533108 240
02:40:01 AM 5996764 10267968 63.13 376808 8796064 972456 5.30 2984820 6532976 104
02:50:02 AM 5984320 10280412 63.21 376960 8796224 998964 5.44 2997100 6532768 128
03:00:01 AM 5990208 10274524 63.17 377128 8796604 984900 5.36 2991880 6532580 100
03:10:01 AM 5997776 10266956 63.12 377328 8796600 976412 5.32 2985760 6532480 124
03:20:01 AM 5989828 10274904 63.17 377512 8796788 983072 5.35 2991884 6532340 192
03:30:01 AM 5999616 10265116 63.11 377680 8796940 968480 5.27 2984288 6532200 208
03:40:01 AM 5894952 10369780 63.76 377820 8866816 1025808 5.59 3086532 6531888 712
03:50:02 AM 5908484 10356248 63.67 377984 8867052 1001000 5.45 3064344 6536904 0
04:00:02 AM 5905564 10359168 63.69 378124 8867432 1001260 5.45 3066764 6536804 196
04:10:01 AM 5905444 10359288 63.69 378212 8867396 994408 5.42 3067328 6536620 156
04:20:01 AM 5916956 10347776 63.62 378312 8867596 979360 5.33 3056396 6536492 116
04:30:01 AM 5900332 10364400 63.72 378388 8867760 1008384 5.49 3073760 6536400 64
04:40:01 AM 5887912 10376820 63.80 378452 8867964 1017844 5.54 3084128 6536276 104
04:50:01 AM 5915864 10348868 63.63 378584 8868072 981380 5.34 3058836 6536112 52
05:00:01 AM 5921280 10343452 63.59 378692 8868528 972056 5.29 3053584 6535844 48
05:10:01 AM 5899020 10365712 63.73 378860 8868416 1003484 5.47 3075672 6535652 56
05:20:01 AM 5918032 10346700 63.61 378940 8868604 977356 5.32 3057700 6535480 28
05:30:01 AM 5899304 10365428 63.73 379028 8868732 999884 5.45 3073884 6535332 28
05:40:01 AM 5915104 10349628 63.63 379120 8869104 976160 5.32 3059072 6535292 40
05:50:01 AM 5913348 10351384 63.64 379180 8869088 974576 5.31 3062080 6535164 4
06:00:01 AM 5922300 10342432 63.59 379248 8869244 968492 5.27 3054596 6534992 32
06:10:02 AM 5904988 10359744 63.69 379352 8869436 1001724 5.46 3069592 6534924 4
06:20:02 AM 5920244 10344488 63.60 379440 8869588 966828 5.27 3054916 6534820 40
06:30:01 AM 5907240 10357492 63.68 379544 8878236 986204 5.37 3058660 6542920 0
06:40:01 AM 5913856 10350876 63.64 379616 8878116 973508 5.30 3054876 6542828 4
06:50:01 AM 5904280 10360452 63.70 379684 8878264 992156 5.40 3063284 6542672 4

06:50:01 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
07:00:02 AM 5912128 10352604 63.65 379796 8878420 977420 5.32 3056608 6542524 48
07:10:01 AM 5889812 10374920 63.79 379912 8879004 1015868 5.53 3075856 6542708 0
07:20:01 AM 5892732 10372000 63.77 380064 8879156 1006528 5.48 3073608 6542544 12
07:30:01 AM 5901116 10363616 63.72 380196 8879312 988768 5.38 3066884 6542464 8
07:40:01 AM 5889696 10375036 63.79 380284 8879512 1017340 5.54 3076832 6542388 12
07:50:01 AM 5890504 10374228 63.78 380396 8879708 1005944 5.48 3075160 6542376 4
08:00:01 AM 5881316 10383416 63.84 380496 8879968 1016628 5.54 3082036 6542328 60
08:10:01 AM 5883504 10381228 63.83 381284 8877060 1029780 5.61 3081176 6541996 768
08:20:01 AM 5897232 10367500 63.74 381556 8877300 995828 5.42 3068992 6542268 12
08:30:01 AM 5808640 10456092 64.29 381772 8895388 1229204 6.69 3138636 6557408 3892
08:40:01 AM 5881972 10382760 63.84 382088 8882340 1008304 5.49 3082672 6542900 4
08:50:01 AM 5900344 10364388 63.72 382584 8884960 985124 5.37 3065436 6542204 112
09:00:01 AM 5894592 10370140 63.76 382928 8885748 998416 5.44 3070588 6542208 172
09:10:01 AM 5872228 10392504 63.90 383292 8887864 1022560 5.57 3091092 6542956 216
09:20:01 AM 5889288 10375444 63.79 383848 8893308 988000 5.38 3075028 6543220 184
09:30:01 AM 5880828 10383904 63.84 385192 8896072 997160 5.43 3082896 6542920 184
09:40:01 AM 5867528 10397204 63.92 385820 8897576 1011536 5.51 3095016 6542864 176
09:50:01 AM 5859224 10405508 63.98 386392 8903012 1014052 5.52 3101908 6542908 56
10:00:01 AM 5863900 10400832 63.95 386848 8903596 1006612 5.48 3097692 6542816 160
10:10:01 AM 5852340 10412392 64.02 387404 8904604 1009764 5.50 3108644 6542708 112
10:20:01 AM 5864948 10399784 63.94 387944 8905536 1002832 5.46 3096564 6542644 308
10:30:01 AM 5780608 10484124 64.46 389124 8939444 1311612 7.14 3145784 6569140 920
10:40:01 AM 5837332 10427400 64.11 389820 8923748 1002232 5.46 3115600 6549644 212
10:50:01 AM 5828308 10436424 64.17 390536 8919912 1030264 5.61 3129568 6542984 76
11:00:01 AM 5473064 10791668 66.35 391524 8943948 1677000 9.13 3461580 6559372 184
11:10:01 AM 5824532 10440200 64.19 392236 8924476 1011276 5.51 3134548 6541744 316
11:20:01 AM 5745984 10518748 64.67 392836 8961620 1369552 7.46 3174812 6576760 180
11:30:01 AM 5724932 10539800 64.80 393612 8986176 1532652 8.35 3172272 6598360 336
11:40:02 AM 5786168 10478564 64.43 394320 8931156 1060592 5.78 3166480 6542496 476
11:50:01 AM 5809964 10454768 64.28 394960 8931356 1007068 5.48 3144212 6542732 220
12:00:01 PM 5791436 10473296 64.39 395624 8931912 1032868 5.63 3159268 6543244 192
12:10:01 PM 5806960 10457772 64.30 396304 8932228 1003964 5.47 3146784 6543820 208
12:20:01 PM 5809164 10455568 64.28 396892 8934016 1002772 5.46 3143140 6545400 184
12:30:01 PM 5531200 10733532 65.99 397620 8932056 1482212 8.07 3413360 6546464 216
12:40:01 PM 5520128 10744604 66.06 398296 8930624 1482560 8.07 3427456 6544084 676
12:50:01 PM 5495372 10769360 66.21 399012 8933496 1493948 8.14 3450176 6545092 336
01:00:01 PM 5496440 10768292 66.21 399676 8935104 1465380 7.98 3449384 6545368 116
01:10:01 PM 5784296 10480436 64.44 400328 8931996 1043380 5.68 3164112 6545176 904
01:20:01 PM 5745068 10519664 64.68 400992 8958344 1283364 6.99 3175812 6570800 236
01:30:02 PM 5741672 10523060 64.70 401644 8953560 1217600 6.63 3184008 6564552 164
01:40:01 PM 5797220 10467512 64.36 402172 8935492 1028188 5.60 3150456 6546956 416

01:40:01 PM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
01:50:01 PM 5638940 10625792 65.33 402712 9004040 1765916 9.62 3230012 6613360 604
02:00:01 PM 5590332 10674400 65.63 403184 9042928 1897828 10.34 3242240 6653456 608
02:10:01 PM 5556148 10708584 65.84 403816 9045944 1962864 10.69 3295028 6629896 1164
02:20:01 PM 5509304 10755428 66.13 404508 9073732 1906292 10.38 3673768 6296372 2828
02:30:01 PM 5673324 10591408 65.12 405068 9012208 1151772 6.27 3580592 6232296 508
02:40:01 PM 5536180 10728552 65.96 405748 9076012 1834620 9.99 3648332 6297956 1460
02:50:01 PM 5550452 10714280 65.87 406240 9056160 1655128 9.01 3651052 6278020 748
03:00:01 PM 5557972 10706760 65.83 406612 9038052 1549576 8.44 3662052 6255912 120
03:10:01 PM 5548792 10715940 65.88 407120 9040716 1538408 8.38 3667072 6261500 756
03:20:01 PM 5577988 10686744 65.71 407584 9022316 1373956 7.48 3660572 6241316 956
03:30:01 PM 5560396 10704336 65.81 408000 9044180 1522536 8.29 3657976 6259868 464
03:40:01 PM 5607288 10657444 65.52 408424 9006760 1204784 6.56 3651936 6222100 604
03:50:01 PM 5243004 11021728 67.76 408936 9061244 2129252 11.60 3953200 6277908 272
04:00:01 PM 5598412 10666320 65.58 409724 9010160 1196612 6.52 3655376 6224416 240
04:10:01 PM 5606268 10658464 65.53 410544 9011096 1176592 6.41 3646956 6225568 224
04:20:01 PM 5586716 10678016 65.65 411252 9015148 1258160 6.85 3654444 6235952 32
04:30:01 PM 5570128 10694604 65.75 412248 9014156 1325484 7.22 3671424 6231988 1180
04:40:01 PM 5615592 10649140 65.47 412844 9009188 1188988 6.48 3638548 6224236 140
04:50:01 PM 5575720 10689012 65.72 413328 9020892 1279204 6.97 3665900 6235100 248
05:00:01 PM 5538056 10726676 65.95 413904 9038964 1475500 8.04 3678772 6256300 880
05:10:01 PM 5286112 10978620 67.50 414476 9045872 1981472 10.79 3925280 6257844 704
05:20:01 PM 5346152 10918580 67.13 415120 9024444 1609168 8.76 3890676 6238280 480
05:30:01 PM 5254220 11010512 67.70 415668 9058792 1974780 10.75 3943560 6271992 296
05:40:01 PM 5309660 10955072 67.35 416272 9029132 1725180 9.40 3917004 6244704 1348
05:50:01 PM 5526476 10738256 66.02 416780 9049804 1449260 7.89 3682268 6264760 1288
06:00:01 PM 5499392 10765340 66.19 417316 9050308 1491584 8.12 3705484 6264116 1420
06:10:01 PM 5535120 10729612 65.97 417780 9047020 1360840 7.41 3681144 6258600 200
06:20:01 PM 5478576 10786156 66.32 418264 9073384 1710968 9.32 3708724 6285316 384
06:30:01 PM 5477120 10787612 66.33 418604 9058152 1575532 8.58 3723160 6271208 232
06:40:01 PM 5536520 10728212 65.96 419004 9041224 1369436 7.46 3685644 6252604 760
06:50:01 PM 5325620 10939112 67.26 419308 9026604 1546120 8.42 3913680 6234552 148
07:00:01 PM 5272024 10992708 67.59 419708 9023680 1645932 8.96 3962396 6233864 84
07:10:01 PM 5233800 11030932 67.82 420108 9042336 1823736 9.93 3982280 6251200 352
07:20:01 PM 5287672 10977060 67.49 420540 9025232 1629064 8.87 3948568 6233784 216
07:30:01 PM 5267804 10996928 67.61 420904 9036208 1643712 8.95 3959144 6241968 88
07:40:01 PM 5239172 11025560 67.79 421268 9036460 1692812 9.22 3987036 6242132 484
07:50:01 PM 5272668 10992064 67.58 421596 9036820 1630136 8.88 3954964 6242080 172
08:00:01 PM 5276660 10988072 67.56 421896 9037200 1618636 8.82 3951700 6242136 792
08:10:01 PM 5264780 10999952 67.63 422116 9037372 1627188 8.86 3963264 6242152 160
08:20:01 PM 5263348 11001384 67.64 422376 9037704 1624484 8.85 3964928 6242196 208
08:30:01 PM 5275808 10988924 67.56 422576 9037968 1623648 8.84 3952360 6242220 60

08:30:01 PM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
08:40:01 PM 5272356 10992376 67.58 422760 9038556 1627940 8.87 3955484 6242220 144
08:50:01 PM 5263264 11001468 67.64 423004 9036280 1638696 8.92 3962820 6244016 208
09:00:01 PM 5273976 10990756 67.57 423200 9036896 1627500 8.86 3952744 6244044 80
09:10:01 PM 5279188 10985544 67.54 423332 9036824 1606860 8.75 3949060 6244132 228
09:20:01 PM 5257448 11007284 67.68 423464 9036884 1637644 8.92 3968956 6244136 140
09:30:01 PM 5546232 10718500 65.90 423528 9037552 1211984 6.60 3680828 6244452 168
09:40:01 PM 5543888 10720844 65.91 423580 9037992 1202388 6.55 3684388 6244412 36
09:50:01 PM 5561096 10703636 65.81 423644 9038320 1181832 6.44 3668428 6244408 80
10:00:01 PM 5536364 10728368 65.96 423760 9041536 1220312 6.65 3687880 6245000 92
10:10:01 PM 5553448 10711284 65.86 423856 9041600 1182624 6.44 3673536 6245012 32
Average: 5710455 10554277 64.89 395677 8934957 1242379 6.77 3345983 6438022 316

Ambarishrh

@scottalanmiller said:

how many vCPU is that? It's definitely very busy, but not crazy.

(4 CPU)

Ambarishrh

It has ActiveCollab used which uses Elasticsearch, used by around 50 users

scottalanmiller

At what time mark is the crash happening?

Ambarishrh

It used to be around 30 minutes after the service is started, not sure if changing the heap size made any difference but the last failure was after an hour later. May be because the usage is less, not sure

scottalanmiller

I mean where in the SAR time listing would I see what it looked like at crash time.

Ambarishrh

That would be around 9
09:00:01 PM 5273976 10990756 67.57 423200 9036896 1627500 8.86 3952744 6244044 80
09:10:01 PM 5279188 10985544 67.54 423332 9036824 1606860 8.75 3949060 6244132 228
09:20:01 PM 5257448 11007284 67.68 423464 9036884 1637644 8.92 3968956 6244136 140
09:30:01 PM 5546232 10718500 65.90 423528 9037552 1211984 6.60 3680828 6244452 168
09:40:01 PM 5543888 10720844 65.91 423580 9037992 1202388 6.55 3684388 6244412 36

scottalanmiller

Odd, I don't see memory freeing up after the crash.

stacksofplates

This doesn't fix the problem, but you could use something like monit or supervisord to start it up again after it quits.

Ambarishrh

@johnhooks monit looks promising. Is there any specific option by which I can monitor the Elasticsearch failing?

Solved Issue with Elasticsearch

sar