FreePBX - High Load Averages
-
Update: Looks like the spike has fixed itself and the episode lasted for about 20 minutes. The
topdata from above was about halfway through the episode.Current
topreadout.top - 11:23:00 up 4 days, 23:47, 2 users, load average: 1.87, 0.80, 1.34 Tasks: 112 total, 1 running, 111 sleeping, 0 stopped, 0 zombie %Cpu(s): 4.9 us, 2.7 sy, 0.0 ni, 92.1 id, 0.2 wa, 0.0 hi, 0.0 si, 0.2 st KiB Mem : 1016380 total, 164472 free, 543580 used, 308328 buff/cache KiB Swap: 2097148 total, 1734024 free, 363124 used. 223956 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 28491 root 20 0 157568 2036 1500 R 6.7 0.2 0:00.01 top 1 root 20 0 43380 2448 1424 S 0.0 0.2 2:45.09 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.09 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:03.61 ksoftirqd/0 7 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh 9 root 20 0 0 0 0 S 0.0 0.0 0:50.28 rcu_sched 10 root rt 0 0 0 0 S 0.0 0.0 0:02.97 watchdog/0 12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs 13 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns 14 root 20 0 0 0 0 S 0.0 0.0 0:02.03 khungtaskd 15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback 16 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd 17 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset 18 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd 19 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 md 25 root 20 0 0 0 0 S 0.0 0.0 0:23.46 kswapd0 26 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd 27 root 39 19 0 0 0 S 0.0 0.0 0:07.74 khugepaged 28 root 20 0 0 0 0 S 0.0 0.0 0:00.00 fsnotify_mark 29 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto 37 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kthrotld 39 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kmpath_rdacd 40 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kpsmoused 41 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ipv6_addrconf 60 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 deferwq 92 root 20 0 0 0 0 S 0.0 0.0 0:03.11 kauditd 264 root 0 -20 0 0 0 S 0.0 0.0 0:00.03 ata_sff 280 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0 281 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_0 282 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_1 283 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_1 285 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ttm_swap 359 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kdmflush 360 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset 371 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kdmflush 372 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset 385 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfsalloc 386 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs_mru_cache 387 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs-buf/dm-0 388 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs-data/dm-0 389 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs-conv/dm-0 390 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs-cil/dm-0 391 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs-reclaim/dm- 392 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs-log/dm-0 393 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs-eofblocks/d 394 root 20 0 0 0 0 S 0.0 0.0 2:23.22 xfsaild/dm-0 469 root 20 0 36120 3020 2880 S 0.0 0.3 1:39.27 systemd-journal 484 root 20 0 127084 0 0 S 0.0 0.0 0:00.02 lvmetad 498 root 20 0 43632 292 288 S 0.0 0.0 0:00.07 systemd-udevd 516 root 20 0 0 0 0 S 0.0 0.0 0:00.00 vballoon 517 root 20 0 0 0 0 S 0.0 0.0 0:03.66 hwrng 581 root 20 0 0 0 0 S 0.0 0.0 0:00.00 jbd2/vda1-8 582 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-rsv-conver 597 root 16 -4 55412 428 308 S 0.0 0.0 0:15.62 auditd 619 avahi 20 0 27976 340 224 S 0.0 0.0 0:11.23 avahi-daemon 620 dbus 20 0 24536 796 456 S 0.0 0.1 2:12.72 dbus-daemon 623 avahi 20 0 27976 24 0 S 0.0 0.0 0:00.00 avahi-daemon 627 root 20 0 105476 332 236 S 0.0 0.0 0:00.03 sshd 629 root 20 0 553148 552 268 S 0.0 0.1 1:09.80 tuned 630 root 20 0 24308 988 772 S 0.0 0.1 1:11.72 systemd-logind 636 root 20 0 6964 180 140 S 0.0 0.0 0:00.15 mdadm 637 root 20 0 262428 3476 3152 S 0.0 0.3 0:47.90 rsyslogd 639 root 20 0 541036 18292 12560 S 0.0 1.8 0:30.71 httpd 645 polkitd 20 0 527508 936 608 S 0.0 0.1 0:48.51 polkitd 653 root 20 0 14988 420 288 S 0.0 0.0 0:00.03 incrond 659 root 20 0 27100 24 24 S 0.0 0.0 0:00.00 xinetd 661 chrony 20 0 115844 868 652 S 0.0 0.1 0:01.42 chronyd 665 nobody 20 0 15540 96 92 S 0.0 0.0 0:00.00 dnsmasq 697 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 cfg80211 708 root 20 0 25840 0 0 S 0.0 0.0 0:00.02 atd 721 root 20 0 126248 808 400 S 0.0 0.1 0:28.04 crond 738 root 20 0 110032 64 60 S 0.0 0.0 0:00.00 agetty 752 mysql 20 0 113252 52 52 S 0.0 0.0 0:00.01 mysqld_safe 828 mongodb 20 0 483944 9632 608 S 0.0 0.9 23:02.65 mongod 850 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 dio/dm-0 1034 mysql 20 0 1175812 165620 2668 S 0.0 16.3 47:24.37 mysqld 1148 root 20 0 89032 276 192 S 0.0 0.0 0:03.28 master 1150 postfix 20 0 89312 496 372 S 0.0 0.0 0:00.65 qmgr 1207 root 20 0 112876 536 304 S 0.0 0.1 0:00.05 dhclient 2037 asterisk 20 0 390124 16112 9440 S 0.0 1.6 0:02.46 php 2108 asterisk 20 0 649804 21732 9704 S 0.0 2.1 0:09.89 httpd 2943 asterisk 20 0 747464 52088 31572 S 0.0 5.1 0:46.08 httpd 2949 asterisk 20 0 651364 22804 9760 S 0.0 2.2 0:46.66 httpd 2951 asterisk 20 0 657644 40948 21644 S 0.0 4.0 0:29.10 httpd 3843 root 0 -20 0 0 0 S 0.0 0.0 0:00.74 kworker/0:2H 4574 postfix 20 0 89136 3588 2592 S 0.0 0.4 0:00.03 pickup 5837 root 20 0 115240 44 40 S 0.0 0.0 0:00.00 safe_asterisk 5844 asterisk 20 0 1645160 51940 5540 S 0.0 5.1 60:21.48 asterisk 14153 root 0 -20 0 0 0 S 0.0 0.0 0:02.17 kworker/0:1H 14446 asterisk 20 0 725692 23224 11100 S 0.0 2.3 0:43.66 httpd 14447 asterisk 20 0 745696 43244 22556 S 0.0 4.3 0:56.37 httpd 14448 asterisk 20 0 659616 46120 26956 S 0.0 4.5 0:33.52 httpd 14449 asterisk 20 0 745100 45560 27228 S 0.0 4.5 0:31.79 httpd 16487 asterisk 20 0 928928 16728 2160 S 0.0 1.6 5:58.75 PM2 v2.7.1: God 16503 asterisk 20 0 1214940 48604 1844 S 0.0 4.8 5:57.58 letschat 16833 asterisk 20 0 195328 640 232 S 0.0 0.1 0:00.20 pnp_server 20229 root 20 0 0 0 0 S 0.0 0.0 0:00.52 kworker/0:1 21407 asterisk 20 0 652420 26280 13676 S 0.0 2.6 0:33.10 httpd 23805 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kworker/u2:0 24324 root 20 0 0 0 0 S 0.0 0.0 0:00.57 kworker/0:0 26812 root 20 0 148384 5936 4588 S 0.0 0.6 0:00.10 sshd 26840 asterisk 20 0 651172 22260 9416 S 0.0 2.2 0:07.31 httpd 26856 root 20 0 115380 2048 1656 S 0.0 0.2 0:00.01 bash 27117 root 20 0 0 0 0 S 0.0 0.0 0:00.19 kworker/0:2 30030 asterisk 20 0 417364 26320 10636 S 0.0 2.6 0:14.90 php 30462 root 20 0 148248 4524 3180 S 0.0 0.4 0:00.09 sshd 30530 root 20 0 115380 1808 1432 S 0.0 0.2 0:00.02 bash 30857 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u2:2 30959 asterisk 20 0 1085388 35716 5012 S 0.0 3.5 0:20.63 node /var/www/h 31123 root 20 0 398024 11020 4956 S 0.0 1.1 10:54.80 php 31176 root 20 0 185668 1504 1144 S 0.0 0.1 0:07.21 fail2ban-server -
By default you will have about 8-12 httpd processes. How many did you have?
-
My guess is that you are waiting on disks. What does your SAR report say?
-
Using
sar -dshould tell you what @scottalanmiller is looking for. (I think) -
@eddiejennings said in FreePBX - High Load Averages:
This is related to another thread concerning what
topis telling us.At the moment the FreePBX GUI is running super slow, so I ssh in, and run to top to find load averages of 8.74, 6.74, and 3.57. The last time this happened, I panicked and restarted. This time, I'm much more calm and am ready to figure out the problem.
It looks like there are a number of
httpdprocesses running; however, I don't know what I should consider "normal." Any verteran FreePBX folks see a smoking gun in the top output that could explain the load averages?top - 11:01:30 up 4 days, 23:26, 2 users, load average: 8.74, 6.74, 3.57 Tasks: 122 total, 9 running, 112 sleeping, 0 stopped, 1 zombie %Cpu(s): 4.9 us, 2.7 sy, 0.0 ni, 92.1 id, 0.2 wa, 0.0 hi, 0.0 si, 0.2 st KiB Mem : 1016380 total, 171816 free, 586868 used, 257696 buff/cache KiB Swap: 2097148 total, 1733740 free, 363408 used. 182016 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9 root 20 0 0 0 0 R 6.2 0.0 0:49.83 rcu_sched 4442 root 20 0 0 0 0 S 6.2 0.0 0:00.69 kworker/0:0 14446 asterisk 20 0 731588 28916 11100 S 6.2 2.8 0:35.91 httpd 14447 asterisk 20 0 744928 42548 22628 S 6.2 4.2 0:44.56 httpd 21407 asterisk 20 0 657796 31416 13676 R 6.2 3.1 0:19.66 httpd 1 root 20 0 43380 2448 1424 S 0.0 0.2 2:44.59 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.09 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:03.57 ksoftirqd/0 7 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh 10 root rt 0 0 0 0 S 0.0 0.0 0:02.96 watchdog/0 12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs 13 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns 14 root 20 0 0 0 0 S 0.0 0.0 0:02.03 khungtaskd 15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback 16 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd 17 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset 18 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd 19 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 md 25 root 20 0 0 0 0 S 0.0 0.0 0:23.46 kswapd0 26 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd 27 root 39 19 0 0 0 S 0.0 0.0 0:07.73 khugepaged 28 root 20 0 0 0 0 S 0.0 0.0 0:00.00 fsnotify_mark 29 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto 37 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kthrotld 39 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kmpath_rdacd 40 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kpsmoused 41 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ipv6_addrconf 60 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 deferwq 92 root 20 0 0 0 0 S 0.0 0.0 0:03.09 kauditd 264 root 0 -20 0 0 0 S 0.0 0.0 0:00.03 ata_sff 280 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0 281 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_0 282 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_1 283 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_1 285 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ttm_swap 359 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kdmflush 360 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset 371 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kdmflush 372 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset 385 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfsalloc 386 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs_mru_cache 387 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs-buf/dm-0 388 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs-data/dm-0 389 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs-conv/dm-0 390 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs-cil/dm-0 391 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs-reclaim/dm- 392 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs-log/dm-0 393 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs-eofblocks/d 394 root 20 0 0 0 0 S 0.0 0.0 2:22.78 xfsaild/dm-0 469 root 20 0 36120 2476 2336 S 0.0 0.2 1:38.96 systemd-journal 484 root 20 0 127084 0 0 S 0.0 0.0 0:00.02 lvmetad 498 root 20 0 43632 292 288 S 0.0 0.0 0:00.07 systemd-udevd 516 root 20 0 0 0 0 S 0.0 0.0 0:00.00 vballoon 517 root 20 0 0 0 0 R 0.0 0.0 0:03.62 hwrng 572 root 20 0 0 0 0 S 0.0 0.0 0:00.20 kworker/u2:0 581 root 20 0 0 0 0 S 0.0 0.0 0:00.00 jbd2/vda1-8 582 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-rsv-conver 597 root 16 -4 55412 428 308 S 0.0 0.0 0:15.56 auditd 619 avahi 20 0 27976 340 224 S 0.0 0.0 0:11.20 avahi-daemon 620 dbus 20 0 24536 796 456 S 0.0 0.1 2:12.32 dbus-daemon 623 avahi 20 0 27976 24 0 S 0.0 0.0 0:00.00 avahi-daemon 627 root 20 0 105476 204 132 S 0.0 0.0 0:00.03 sshd 629 root 20 0 553148 552 268 S 0.0 0.1 1:09.54 tuned 630 root 20 0 24308 984 768 S 0.0 0.1 1:11.51 systemd-logind 636 root 20 0 6964 144 104 S 0.0 0.0 0:00.15 mdadm 637 root 20 0 262428 2980 2656 S 0.0 0.3 0:47.73 rsyslogd 639 root 20 0 541036 18292 12560 S 0.0 1.8 0:30.62 httpd 645 polkitd 20 0 527508 936 608 S 0.0 0.1 0:48.36 polkitd 653 root 20 0 14988 420 288 S 0.0 0.0 0:00.03 incrond 659 root 20 0 27100 24 24 S 0.0 0.0 0:00.00 xinetd 661 chrony 20 0 115844 884 660 S 0.0 0.1 0:01.41 chronyd 665 nobody 20 0 15540 96 92 S 0.0 0.0 0:00.00 dnsmasq 697 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 cfg80211 708 root 20 0 25840 0 0 S 0.0 0.0 0:00.02 atd 721 root 20 0 126248 808 400 S 0.0 0.1 0:27.96 crond 738 root 20 0 110032 64 60 S 0.0 0.0 0:00.00 agetty 752 mysql 20 0 113252 52 52 S 0.0 0.0 0:00.01 mysqld_safe 828 mongodb 20 0 483944 9632 608 R 0.0 0.9 22:58.14 mongod 850 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 dio/dm-0 1034 mysql 20 0 1175812 165616 2668 S 0.0 16.3 47:01.24 mysqld 1148 root 20 0 89032 276 192 S 0.0 0.0 0:03.27 master 1150 postfix 20 0 89312 496 372 S 0.0 0.0 0:00.65 qmgr 1207 root 20 0 112876 536 304 S 0.0 0.1 0:00.05 dhclient 2037 asterisk 20 0 390124 16108 9436 S 0.0 1.6 0:00.07 php 2043 root 20 0 0 0 0 S 0.0 0.0 0:00.08 kworker/0:2 2105 asterisk 20 0 541036 7228 1428 S 0.0 0.7 0:00.00 httpd 2106 asterisk 20 0 541036 7228 1428 S 0.0 0.7 0:00.00 httpd 2107 asterisk 20 0 541036 7228 1428 S 0.0 0.7 0:00.00 httpd 2108 asterisk 20 0 541036 7228 1428 S 0.0 0.7 0:00.00 httpd 2943 asterisk 20 0 747208 51848 31612 S 0.0 5.1 0:31.82 httpd 2948 asterisk 20 0 746180 43004 22044 S 0.0 4.2 0:12.43 httpd 2949 asterisk 20 0 655456 27236 9760 S 0.0 2.7 0:38.37 httpd 2951 asterisk 20 0 657644 40812 21508 S 0.0 4.0 0:22.15 httpd 3843 root 0 -20 0 0 0 S 0.0 0.0 0:00.74 kworker/0:2H 4574 postfix 20 0 89136 3588 2592 S 0.0 0.4 0:00.02 pickup 5837 root 20 0 115240 44 40 S 0.0 0.0 0:00.00 safe_asterisk 5844 asterisk 20 0 1645160 51364 5172 S 0.0 5.1 60:03.66 asterisk 9113 root 20 0 157568 2036 1500 R 0.0 0.2 0:00.00 top 9131 asterisk 20 0 0 0 0 Z 0.0 0.0 0:00.00 grep 9132 asterisk 20 0 744928 19920 0 R 0.0 2.0 0:00.00 httpd 9133 asterisk 20 0 731588 17816 0 R 0.0 1.8 0:00.00 httpd 9134 asterisk 20 0 390124 6672 0 R 0.0 0.7 0:00.00 php 9135 asterisk 20 0 655456 17476 0 R 0.0 1.7 0:00.00 httpd 14153 root 0 -20 0 0 0 S 0.0 0.0 0:02.08 kworker/0:1H 14448 asterisk 20 0 659616 46120 26956 S 0.0 4.5 0:22.77 httpd 14449 asterisk 20 0 745100 45528 27200 S 0.0 4.5 0:22.54 httpd 16487 asterisk 20 0 928928 16416 2160 S 0.0 1.6 5:57.61 PM2 v2.7.1: God 16503 asterisk 20 0 1214940 48888 1844 S 0.0 4.8 5:56.35 letschat 16833 asterisk 20 0 195328 640 232 S 0.0 0.1 0:00.20 pnp_server 20229 root 20 0 0 0 0 S 0.0 0.0 0:00.40 kworker/0:1 26840 asterisk 20 0 650904 22204 9396 S 0.0 2.2 0:00.52 httpd 26928 asterisk 20 0 657360 39272 20248 S 0.0 3.9 0:07.00 httpd 30030 asterisk 20 0 417364 26320 10636 S 0.0 2.6 0:14.67 php 30462 root 20 0 148248 4524 3180 S 0.0 0.4 0:00.09 sshd 30530 root 20 0 115380 1808 1432 S 0.0 0.2 0:00.02 bash 30857 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u2:2 30959 asterisk 20 0 1084876 35248 4968 S 0.0 3.5 0:19.55 node /var/www/h 31123 root 20 0 398024 11020 4956 S 0.0 1.1 10:52.11 php 31176 root 20 0 185668 1504 1144 S 0.0 0.1 0:07.17 fail2ban-server 31775 root 20 0 148384 5932 4584 S 0.0 0.6 0:00.05 sshd 32603 root 20 0 115380 2024 1648 S 0.0 0.2 0:00.01 bashWhat size instance did you go for on Vultr? Not relevant, but curious.
-
I failed to run
sarin a timely fashion. When this spike occurs again, I'll know to run it. -
@eddiejennings said in FreePBX - High Load Averages:
I failed to run
sarin a timely fashion. When this spike occurs again, I'll know to run it.If you don't have sar installed already, go ahead and install it so you know what "good" looks like too.
-
@dafyre said in FreePBX - High Load Averages:
@eddiejennings said in FreePBX - High Load Averages:
I failed to run
sarin a timely fashion. When this spike occurs again, I'll know to run it.If you don't have sar installed already, go ahead and install it so you know what "good" looks like too.
Yeah, I ran it and got a copy of the results. That how I knew I missed my opportunity.
-
@bigbear said in FreePBX - High Load Averages:
What size instance did you go for on Vultr? Not relevant, but curious.
The 1 CPU, 1 GB RAM, 25 GB SSD instance ($5 / month).
-
@scottalanmiller said in FreePBX - High Load Averages:
By default you will have about 8-12 httpd processes. How many did you have?
19 httpd processes during the problem time. 11 when the problem went away.
-
@eddiejennings said in FreePBX - High Load Averages:
@dafyre said in FreePBX - High Load Averages:
@eddiejennings said in FreePBX - High Load Averages:
I failed to run
sarin a timely fashion. When this spike occurs again, I'll know to run it.If you don't have sar installed already, go ahead and install it so you know what "good" looks like too.
Yeah, I ran it and got a copy of the results. That how I knew I missed my opportunity.
Did you check the logs (usually stored in /var/log/sysstat/)?
Try
sar -f /var/log/sysstat/sa02to see the stats from yesterday. -
In the middle of an episode now. I'm collecting data and will post in a moment.
-
Here are the
sar -dresults.For ease of reading, I put the window where the load average spike was at the top. Next is
topfrom about 2/3 of the way through. Since this seems to solve itself, I'm more curious as to figure out what's causing tasks to be queued.Linux 3.10.0-514.26.2.el7.x86_64 (blah.domain.com) 10/10/2017 _x86_64_ (1 CPU) DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util 01:30:02 PM dev252-0 9.94 0.74 385.64 38.87 0.01 0.86 0.18 0.18 01:30:02 PM dev253-0 9.81 0.66 385.64 39.40 0.01 0.90 0.19 0.18 01:30:02 PM dev253-1 0.01 0.08 0.00 8.00 0.00 1.00 0.17 0.00 01:40:02 PM dev252-0 9.34 6.86 354.74 38.70 0.01 0.81 0.17 0.16 01:40:02 PM dev253-0 9.18 5.87 354.74 39.28 0.01 0.84 0.18 0.16 01:40:02 PM dev253-1 0.12 0.99 0.00 8.00 0.00 0.56 0.10 0.00 01:50:01 PM dev252-0 12.80 251.00 333.08 45.64 0.01 1.16 0.30 0.38 01:50:01 PM dev253-0 12.25 238.55 313.44 45.07 0.01 1.16 0.28 0.34 01:50:01 PM dev253-1 4.01 12.45 19.65 8.00 0.07 18.65 0.12 0.05 Linux 3.10.0-514.26.2.el7.x86_64 (blah.domain.com) 10/10/2017 _x86_64_ (1 CPU) 12:00:02 AM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util 12:10:01 AM dev252-0 12.24 233.79 399.69 51.75 0.02 1.27 0.31 0.38 12:10:01 AM dev253-0 12.07 233.47 379.27 50.76 0.01 1.14 0.27 0.33 12:10:01 AM dev253-1 2.59 0.33 20.42 8.00 0.13 49.31 0.26 0.07 12:20:01 AM dev252-0 11.28 129.02 334.39 41.08 0.01 1.19 0.30 0.33 12:20:01 AM dev253-0 9.72 80.71 328.20 42.05 0.01 1.27 0.26 0.26 12:20:01 AM dev253-1 6.81 48.31 6.19 8.00 0.03 4.17 0.12 0.08 12:30:01 AM dev252-0 8.89 0.37 339.30 38.20 0.01 1.01 0.20 0.18 12:30:01 AM dev253-0 8.75 0.00 339.30 38.77 0.01 1.04 0.20 0.18 12:30:01 AM dev253-1 0.05 0.37 0.00 8.00 0.00 0.67 0.11 0.00 12:40:01 AM dev252-0 7.81 0.45 297.80 38.19 0.01 0.83 0.19 0.15 12:40:01 AM dev253-0 7.66 0.03 294.60 38.46 0.01 0.87 0.19 0.15 12:40:01 AM dev253-1 0.45 0.42 3.20 8.00 0.00 1.93 0.04 0.00 12:50:02 AM dev252-0 9.27 0.34 362.79 39.18 0.01 0.69 0.16 0.15 12:50:02 AM dev253-0 9.13 0.09 361.99 39.66 0.01 0.72 0.16 0.15 12:50:02 AM dev253-1 0.13 0.24 0.80 8.00 0.00 1.35 0.12 0.00 01:00:02 AM dev252-0 8.93 21.75 326.97 39.07 0.01 0.77 0.16 0.15 01:00:02 AM dev253-0 8.78 21.45 326.97 39.69 0.01 0.80 0.17 0.15 01:00:02 AM dev253-1 0.04 0.30 0.00 8.00 0.00 0.41 0.09 0.00 01:10:02 AM dev252-0 12.50 256.42 408.88 53.23 0.01 1.09 0.27 0.33 01:10:02 AM dev253-0 12.31 256.17 378.13 51.53 0.01 0.95 0.20 0.25 01:10:02 AM dev253-1 3.88 0.26 30.74 8.00 0.20 51.04 0.24 0.09 01:20:02 AM dev252-0 9.64 6.37 345.35 36.49 0.01 0.79 0.18 0.18 01:20:02 AM dev253-0 9.47 6.15 336.18 36.14 0.01 0.82 0.19 0.18 01:20:02 AM dev253-1 1.17 0.22 9.16 8.00 0.00 2.71 0.02 0.00 01:30:01 AM dev252-0 9.63 10.99 366.75 39.22 0.01 0.89 0.17 0.16 01:30:01 AM dev253-0 9.49 10.78 366.75 39.79 0.01 0.92 0.17 0.16 01:30:01 AM dev253-1 0.03 0.20 0.00 8.00 0.00 0.87 0.20 0.00 01:40:01 AM dev252-0 9.94 40.40 369.56 41.23 0.01 0.88 0.18 0.18 01:40:01 AM dev253-0 9.78 40.21 363.37 41.28 0.01 0.90 0.18 0.18 01:40:01 AM dev253-1 0.80 0.19 6.18 8.00 0.01 7.92 0.08 0.01 01:50:01 AM dev252-0 9.05 39.06 333.04 41.10 0.01 0.85 0.17 0.15 01:50:01 AM dev253-0 8.88 38.42 327.97 41.26 0.01 0.88 0.17 0.15 01:50:01 AM dev253-1 0.72 0.64 5.09 8.00 0.00 6.84 0.06 0.00 02:00:01 AM dev252-0 8.61 0.20 330.00 38.33 0.01 0.84 0.17 0.15 02:00:01 AM dev253-0 8.42 0.01 329.98 39.17 0.01 0.87 0.18 0.15 02:00:01 AM dev253-1 0.02 0.19 0.00 8.00 0.00 0.57 0.14 0.00 02:10:01 AM dev252-0 10.85 178.08 373.68 50.83 0.01 1.00 0.24 0.26 02:10:01 AM dev253-0 10.66 177.83 364.17 50.85 0.01 0.95 0.19 0.20 02:10:01 AM dev253-1 1.22 0.24 9.51 8.00 0.05 42.90 0.51 0.06 02:20:01 AM dev252-0 8.84 5.29 330.72 38.02 0.01 0.80 0.15 0.14 02:20:01 AM dev253-0 8.68 4.91 330.72 38.68 0.01 0.83 0.16 0.14 02:20:01 AM dev253-1 0.05 0.38 0.00 8.00 0.00 0.71 0.14 0.00 02:30:01 AM dev252-0 8.63 0.92 329.72 38.31 0.01 0.78 0.16 0.14 02:30:01 AM dev253-0 8.48 0.00 329.72 38.89 0.01 0.81 0.17 0.14 02:30:01 AM dev253-1 0.12 0.92 0.00 8.00 0.00 0.50 0.09 0.00 02:40:01 AM dev252-0 10.39 18.80 401.01 40.39 0.01 0.86 0.17 0.18 02:40:01 AM dev253-0 10.24 18.65 401.01 40.98 0.01 0.88 0.18 0.18 02:40:01 AM dev253-1 0.02 0.15 0.00 8.00 0.00 1.00 0.18 0.00 02:50:01 AM dev252-0 9.64 79.81 346.50 44.24 0.01 0.90 0.19 0.19 02:50:01 AM dev253-0 9.32 75.35 328.13 43.29 0.01 0.89 0.18 0.17 02:50:01 AM dev253-1 2.85 4.46 18.38 8.00 0.06 19.69 0.07 0.02 03:00:02 AM dev252-0 8.91 16.85 323.36 38.17 0.01 0.77 0.17 0.15 03:00:02 AM dev253-0 8.74 16.46 323.36 38.89 0.01 0.81 0.18 0.15 03:00:02 AM dev253-1 0.05 0.39 0.00 8.00 0.00 0.28 0.03 0.00 03:10:01 AM dev252-0 11.32 259.35 366.04 55.27 0.02 1.95 0.30 0.34 03:10:01 AM dev253-0 11.09 258.06 337.96 53.72 0.02 1.96 0.30 0.33 03:10:01 AM dev253-1 3.67 1.29 28.08 8.00 0.04 10.70 0.05 0.02 03:20:01 AM dev252-0 8.81 8.00 322.37 37.49 0.01 0.98 0.19 0.17 03:20:01 AM dev253-0 8.64 5.87 322.37 37.98 0.01 1.02 0.20 0.17 03:20:01 AM dev253-1 0.27 2.13 0.00 8.00 0.00 0.55 0.08 0.00 03:30:01 AM dev252-0 8.83 25.89 326.49 39.90 0.01 0.80 0.18 0.16 03:30:01 AM dev253-0 8.69 22.91 326.49 40.23 0.01 0.83 0.19 0.16 03:30:01 AM dev253-1 0.37 2.98 0.00 8.00 0.00 0.52 0.08 0.00 03:40:01 AM dev252-0 8.53 0.14 322.66 37.84 0.01 0.80 0.17 0.15 03:40:01 AM dev253-0 8.40 0.00 322.66 38.43 0.01 0.83 0.18 0.15 03:40:01 AM dev253-1 0.02 0.14 0.00 8.00 0.00 0.80 0.10 0.00 03:50:01 AM dev252-0 9.30 0.00 359.01 38.61 0.01 0.77 0.17 0.15 03:50:01 AM dev253-0 9.17 0.00 359.01 39.15 0.01 0.80 0.17 0.16 03:50:01 AM dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 04:00:01 AM dev252-0 9.12 0.88 354.46 38.98 0.01 0.80 0.17 0.16 04:00:01 AM dev253-0 8.97 0.01 354.46 39.52 0.01 0.83 0.18 0.16 04:00:01 AM dev253-1 0.11 0.87 0.00 8.00 0.00 0.39 0.06 0.00 04:10:01 AM dev252-0 18.91 237.02 391.35 33.23 0.02 1.23 0.51 0.96 04:10:01 AM dev253-0 18.68 236.53 364.37 32.17 0.02 1.26 0.50 0.93 04:10:01 AM dev253-1 3.43 0.49 26.98 8.00 0.09 26.85 0.11 0.04 04:20:01 AM dev252-0 8.17 16.84 309.93 39.98 0.01 1.18 0.23 0.19 04:20:01 AM dev253-0 8.06 16.75 309.93 40.52 0.01 1.23 0.24 0.19 04:20:01 AM dev253-1 0.01 0.10 0.00 8.00 0.00 1.00 0.14 0.00 04:30:02 AM dev252-0 8.84 0.00 341.94 38.69 0.01 0.78 0.18 0.16 04:30:02 AM dev253-0 8.72 0.00 341.94 39.22 0.01 0.81 0.18 0.16 04:30:02 AM dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 04:40:02 AM dev252-0 8.16 0.11 306.14 37.52 0.01 0.86 0.17 0.14 04:40:02 AM dev253-0 8.02 0.00 306.14 38.19 0.01 0.90 0.17 0.14 04:40:02 AM dev253-1 0.01 0.11 0.00 8.00 0.00 0.12 0.12 0.00 04:50:01 AM dev252-0 7.09 9.28 251.23 36.76 0.01 0.82 0.16 0.11 04:50:01 AM dev253-0 7.00 9.28 251.23 37.24 0.01 0.86 0.17 0.12 04:50:01 AM dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 05:00:02 AM dev252-0 8.72 0.07 333.19 38.23 0.01 0.80 0.18 0.16 05:00:02 AM dev253-0 8.58 0.00 333.19 38.86 0.01 0.83 0.18 0.16 05:00:02 AM dev253-1 0.01 0.07 0.00 8.00 0.00 0.00 0.00 0.00 05:10:02 AM dev252-0 10.97 163.28 358.36 47.57 0.01 1.13 0.23 0.26 05:10:02 AM dev253-0 10.78 163.06 337.65 46.46 0.01 0.96 0.19 0.20 05:10:02 AM dev253-1 2.62 0.22 20.71 8.00 0.21 79.52 0.23 0.06 05:20:01 AM dev252-0 8.23 7.02 303.95 37.79 0.01 0.90 0.18 0.15 05:20:01 AM dev253-0 8.06 6.67 303.95 38.53 0.01 0.93 0.18 0.14 05:20:01 AM dev253-1 0.04 0.35 0.00 8.00 0.00 1.00 0.85 0.00 05:30:01 AM dev252-0 9.80 0.52 388.72 39.72 0.01 0.82 0.17 0.16 05:30:01 AM dev253-0 9.63 0.41 388.72 40.42 0.01 0.85 0.17 0.17 05:30:01 AM dev253-1 0.01 0.11 0.00 8.00 0.00 1.00 0.13 0.00 05:40:01 AM dev252-0 9.62 0.01 380.32 39.53 0.01 0.84 0.17 0.16 05:40:01 AM dev253-0 9.47 0.01 380.32 40.16 0.01 0.87 0.17 0.16 05:40:01 AM dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 05:50:01 AM dev252-0 7.96 0.11 309.45 38.89 0.01 0.87 0.17 0.14 05:50:01 AM dev253-0 7.84 0.00 309.45 39.49 0.01 0.90 0.18 0.14 05:50:01 AM dev253-1 0.01 0.11 0.00 8.00 0.00 1.00 0.12 0.00 06:00:01 AM dev252-0 8.85 15.84 336.92 39.88 0.01 0.86 0.16 0.15 06:00:01 AM dev253-0 8.71 15.84 336.92 40.48 0.01 0.90 0.17 0.15 06:00:01 AM dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 06:10:02 AM dev252-0 11.35 142.47 406.56 48.35 0.01 1.13 0.22 0.25 06:10:02 AM dev253-0 11.17 142.33 368.50 45.73 0.01 0.91 0.19 0.21 06:10:02 AM dev253-1 4.77 0.14 38.06 8.00 0.30 63.66 0.09 0.04 06:20:01 AM dev252-0 9.11 0.01 348.27 38.25 0.01 0.87 0.16 0.15 06:20:01 AM dev253-0 8.97 0.01 348.27 38.82 0.01 0.90 0.17 0.15 06:20:01 AM dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 06:30:01 AM dev252-0 9.57 0.00 370.63 38.74 0.01 0.79 0.17 0.16 06:30:01 AM dev253-0 9.46 0.00 370.63 39.19 0.01 0.82 0.17 0.16 06:30:01 AM dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 06:40:01 AM dev252-0 8.84 0.11 339.69 38.46 0.01 0.85 0.18 0.16 06:40:01 AM dev253-0 8.70 0.00 339.69 39.05 0.01 0.88 0.19 0.16 06:40:01 AM dev253-1 0.01 0.11 0.00 8.00 0.00 1.00 0.13 0.00 06:50:02 AM dev252-0 8.75 0.12 336.83 38.53 0.01 0.91 0.18 0.16 06:50:02 AM dev253-0 8.65 0.01 336.83 38.94 0.01 0.95 0.19 0.16 06:50:02 AM dev253-1 0.01 0.11 0.00 8.00 0.00 1.00 0.12 0.00 07:00:01 AM dev252-0 6.96 0.11 252.95 36.35 0.01 0.72 0.17 0.12 07:00:01 AM dev253-0 6.83 0.00 252.95 37.01 0.01 0.76 0.17 0.12 07:00:01 AM dev253-1 0.01 0.11 0.00 8.00 0.00 1.00 0.12 0.00 07:10:01 AM dev252-0 9.22 70.69 311.74 41.48 0.01 0.89 0.21 0.19 07:10:01 AM dev253-0 9.12 70.58 306.22 41.30 0.01 0.86 0.18 0.16 07:10:01 AM dev253-1 0.70 0.11 5.53 8.00 0.04 61.35 0.49 0.03 07:20:01 AM dev252-0 7.77 0.07 295.80 38.06 0.01 0.78 0.18 0.14 07:20:01 AM dev253-0 7.66 0.00 295.80 38.64 0.01 0.82 0.18 0.14 07:20:01 AM dev253-1 0.01 0.07 0.00 8.00 0.00 0.00 0.00 0.00 07:30:01 AM dev252-0 9.43 0.00 379.00 40.21 0.01 0.87 0.17 0.16 07:30:01 AM dev253-0 9.31 0.00 379.00 40.71 0.01 0.90 0.18 0.17 07:30:01 AM dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 07:40:02 AM dev252-0 9.80 0.50 384.10 39.24 0.01 0.88 0.17 0.17 07:40:02 AM dev253-0 9.67 0.50 384.10 39.78 0.01 0.91 0.17 0.17 07:40:02 AM dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 07:50:01 AM dev252-0 10.26 0.23 403.08 39.32 0.01 0.84 0.16 0.16 07:50:01 AM dev253-0 10.13 0.00 403.08 39.78 0.01 0.87 0.16 0.16 07:50:01 AM dev253-1 0.03 0.23 0.00 8.00 0.00 1.41 0.24 0.00 08:00:01 AM dev252-0 9.67 0.00 380.94 39.38 0.01 0.95 0.17 0.17 08:00:01 AM dev253-0 9.55 0.00 380.94 39.91 0.01 0.98 0.18 0.17 08:00:01 AM dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 08:10:01 AM dev252-0 8.58 33.98 309.32 40.01 0.01 1.03 0.20 0.17 08:10:01 AM dev253-0 8.46 33.71 295.24 38.88 0.01 0.94 0.17 0.14 08:10:01 AM dev253-1 1.79 0.27 14.08 8.00 0.08 42.57 0.19 0.03 08:20:01 AM dev252-0 9.54 0.10 363.35 38.11 0.01 0.82 0.17 0.16 08:20:01 AM dev253-0 9.37 0.00 363.35 38.77 0.01 0.85 0.17 0.16 08:20:01 AM dev253-1 0.01 0.10 0.00 8.00 0.00 0.00 0.00 0.00 08:30:01 AM dev252-0 7.87 0.26 292.76 37.23 0.01 0.81 0.17 0.13 08:30:01 AM dev253-0 7.74 0.00 292.76 37.85 0.01 0.84 0.17 0.13 08:30:01 AM dev253-1 0.03 0.26 0.00 8.00 0.00 0.84 0.16 0.00 08:40:01 AM dev252-0 12.61 295.61 340.57 50.43 0.01 1.09 0.24 0.31 08:40:01 AM dev253-0 11.40 233.28 334.94 49.83 0.01 1.16 0.22 0.25 08:40:01 AM dev253-1 8.49 62.32 5.63 8.00 0.02 2.13 0.07 0.06 08:50:01 AM dev252-0 10.26 70.09 337.74 39.74 0.01 0.94 0.27 0.28 08:50:01 AM dev253-0 9.42 31.42 325.61 37.89 0.01 0.98 0.25 0.24 08:50:01 AM dev253-1 6.35 38.67 12.13 8.00 0.01 1.39 0.08 0.05 09:00:01 AM dev252-0 9.18 3.81 328.34 36.18 0.01 1.02 0.30 0.28 09:00:01 AM dev253-0 8.99 2.81 328.34 36.85 0.01 1.06 0.31 0.28 09:00:01 AM dev253-1 0.13 1.00 0.00 8.00 0.00 0.68 0.09 0.00 09:10:02 AM dev252-0 11.57 210.57 356.26 49.00 0.01 1.10 0.35 0.41 09:10:02 AM dev253-0 11.32 209.14 335.07 48.07 0.01 1.04 0.31 0.36 09:10:02 AM dev253-1 2.83 1.43 21.19 8.00 0.08 29.33 0.20 0.06 09:20:01 AM dev252-0 9.21 19.13 335.23 38.50 0.01 1.09 0.30 0.28 09:20:01 AM dev253-0 9.02 18.70 329.72 38.62 0.01 1.13 0.31 0.28 09:20:01 AM dev253-1 0.74 0.43 5.51 8.00 0.00 5.74 0.05 0.00 09:30:02 AM dev252-0 11.11 113.42 325.10 39.48 0.01 1.22 0.28 0.31 09:30:02 AM dev253-0 10.52 100.73 325.10 40.46 0.01 1.29 0.29 0.31 09:30:02 AM dev253-1 1.59 12.69 0.00 8.00 0.00 0.62 0.12 0.02 09:40:01 AM dev252-0 7.39 5.41 247.88 34.26 0.01 1.07 0.31 0.23 09:40:01 AM dev253-0 7.24 4.83 246.45 34.71 0.01 1.10 0.30 0.22 09:40:01 AM dev253-1 0.25 0.58 1.43 8.00 0.01 53.57 0.53 0.01 09:50:02 AM dev252-0 9.69 5.50 346.63 36.33 0.01 1.21 0.36 0.34 09:50:02 AM dev253-0 9.53 5.18 343.15 36.56 0.01 1.20 0.32 0.31 09:50:02 AM dev253-1 0.47 0.32 3.47 8.00 0.03 63.39 0.88 0.04 10:00:02 AM dev252-0 9.39 20.13 324.59 36.72 0.01 1.19 0.29 0.27 10:00:02 AM dev253-0 9.12 13.65 324.59 37.09 0.01 1.23 0.30 0.27 10:00:02 AM dev253-1 0.81 6.48 0.00 8.00 0.00 0.44 0.06 0.01 10:10:01 AM dev252-0 13.53 309.25 322.27 46.67 0.02 1.25 0.32 0.43 10:10:01 AM dev253-0 13.34 308.34 317.79 46.92 0.02 1.25 0.29 0.39 10:10:01 AM dev253-1 0.67 0.91 4.49 8.00 0.00 4.88 0.59 0.04 10:20:02 AM dev252-0 9.76 20.04 352.26 38.16 0.01 1.27 0.36 0.35 10:20:02 AM dev253-0 9.61 19.88 325.23 35.93 0.01 1.14 0.31 0.29 10:20:02 AM dev253-1 3.40 0.16 27.03 8.00 0.16 45.86 0.18 0.06 10:30:01 AM dev252-0 9.12 43.14 291.33 36.68 0.01 1.18 0.29 0.27 10:30:01 AM dev253-0 8.97 41.37 291.33 37.08 0.01 1.22 0.30 0.27 10:30:01 AM dev253-1 0.22 1.78 0.00 8.00 0.00 0.55 0.08 0.00 10:40:01 AM dev252-0 11.45 79.54 372.85 39.50 0.01 1.10 0.30 0.35 10:40:01 AM dev253-0 11.07 67.40 359.68 38.57 0.01 1.08 0.28 0.31 10:40:01 AM dev253-1 3.16 12.14 13.16 8.00 0.07 23.53 0.14 0.04 10:50:01 AM dev252-0 7.92 6.46 287.52 37.11 0.01 0.80 0.16 0.13 10:50:01 AM dev253-0 7.78 5.96 287.52 37.72 0.01 0.83 0.17 0.13 10:50:01 AM dev253-1 0.06 0.50 0.00 8.00 0.00 0.46 0.08 0.00 11:00:01 AM dev252-0 10.37 9.63 404.71 39.95 0.01 0.85 0.17 0.17 11:00:01 AM dev253-0 10.20 9.02 404.71 40.58 0.01 0.88 0.17 0.18 11:00:01 AM dev253-1 0.08 0.61 0.00 8.00 0.00 0.73 0.20 0.00 11:10:01 AM dev252-0 8.47 122.05 299.60 49.77 0.01 0.91 0.19 0.16 11:10:01 AM dev253-0 8.37 121.98 299.60 50.36 0.01 0.94 0.19 0.16 11:10:01 AM dev253-1 0.01 0.07 0.00 8.00 0.00 0.60 0.20 0.00 11:20:01 AM dev252-0 8.09 0.70 296.62 36.76 0.01 0.76 0.16 0.13 11:20:01 AM dev253-0 7.95 0.60 296.62 37.39 0.01 0.80 0.17 0.13 11:20:01 AM dev253-1 0.01 0.11 0.00 8.00 0.00 1.00 0.12 0.00 11:30:01 AM dev252-0 7.87 0.20 298.17 37.93 0.01 0.75 0.16 0.13 11:30:01 AM dev253-0 7.71 0.00 298.17 38.67 0.01 0.79 0.17 0.13 11:30:01 AM dev253-1 0.03 0.20 0.00 8.00 0.00 0.53 0.07 0.00 11:40:01 AM dev252-0 7.91 0.20 299.06 37.82 0.01 0.86 0.17 0.14 11:40:01 AM dev253-0 7.78 0.09 299.06 38.47 0.01 0.89 0.18 0.14 11:40:01 AM dev253-1 0.01 0.11 0.00 8.00 0.00 0.38 0.12 0.00 11:50:01 AM dev252-0 9.73 1.34 373.22 38.49 0.01 0.82 0.17 0.16 11:50:01 AM dev253-0 9.61 1.27 373.22 38.97 0.01 0.86 0.17 0.16 11:50:01 AM dev253-1 0.01 0.07 0.00 8.00 0.00 0.00 0.00 0.00 12:00:01 PM dev252-0 9.38 9.96 348.15 38.18 0.01 0.80 0.16 0.15 12:00:01 PM dev253-0 9.18 7.72 348.15 38.76 0.01 0.83 0.17 0.15 12:00:01 PM dev253-1 0.28 2.24 0.00 8.00 0.00 0.61 0.09 0.00 12:10:01 PM dev252-0 11.24 68.67 377.66 39.71 0.01 0.77 0.18 0.20 12:10:01 PM dev253-0 11.16 68.10 377.66 39.96 0.01 0.81 0.18 0.21 12:10:01 PM dev253-1 0.07 0.57 0.00 8.00 0.00 0.38 0.05 0.00 12:20:01 PM dev252-0 10.17 67.09 348.83 40.91 0.01 0.79 0.18 0.18 12:20:01 PM dev253-0 10.01 63.91 348.83 41.23 0.01 0.83 0.18 0.18 12:20:01 PM dev253-1 0.40 3.19 0.00 8.00 0.00 0.63 0.08 0.00 12:30:01 PM dev252-0 11.32 73.12 387.48 40.70 0.01 0.98 0.17 0.19 12:30:01 PM dev253-0 11.14 72.04 383.56 40.89 0.01 1.01 0.17 0.19 12:30:01 PM dev253-1 0.63 1.09 3.92 8.00 0.00 3.98 0.05 0.00 12:40:01 PM dev252-0 9.54 27.83 359.99 40.64 0.01 0.82 0.19 0.18 12:40:01 PM dev253-0 9.24 22.79 349.00 40.24 0.01 0.82 0.18 0.16 12:40:01 PM dev253-1 2.00 5.04 10.98 8.00 0.03 16.11 0.10 0.02 12:50:01 PM dev252-0 13.21 168.62 459.35 47.55 0.01 1.04 0.23 0.30 12:50:01 PM dev253-0 12.65 147.03 436.96 46.15 0.01 1.04 0.19 0.25 12:50:01 PM dev253-1 5.50 21.58 22.40 8.00 0.05 9.44 0.10 0.06 01:00:01 PM dev252-0 9.47 20.41 330.61 37.05 0.01 0.81 0.17 0.16 01:00:01 PM dev253-0 9.30 19.75 329.37 37.54 0.01 0.84 0.17 0.16 01:00:01 PM dev253-1 0.24 0.67 1.24 8.00 0.00 1.63 0.11 0.00 01:10:02 PM dev252-0 11.36 128.90 373.16 44.20 0.01 0.99 0.22 0.25 01:10:02 PM dev253-0 11.15 127.88 334.19 41.45 0.01 0.96 0.19 0.21 01:10:02 PM dev253-1 5.00 1.01 38.96 8.00 0.06 11.88 0.10 0.05 01:20:01 PM dev252-0 11.28 12.59 425.62 38.84 0.01 0.97 0.19 0.21 01:20:01 PM dev253-0 11.07 10.27 425.62 39.36 0.01 1.00 0.19 0.21 01:20:01 PM dev253-1 0.29 2.32 0.00 8.00 0.00 0.61 0.10 0.00topoutputtop - 13:47:33 up 6 days, 2:12, 1 user, load average: 6.47, 5.67, 2.87 Tasks: 111 total, 1 running, 110 sleeping, 0 stopped, 0 zombie %Cpu(s): 5.2 us, 3.7 sy, 0.0 ni, 90.8 id, 0.1 wa, 0.0 hi, 0.0 si, 0.2 st KiB Mem : 1016380 total, 243200 free, 527096 used, 246084 buff/cache KiB Swap: 2097148 total, 1688092 free, 409056 used. 253448 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9871 root 20 0 157568 2072 1500 R 6.7 0.2 0:00.01 top 1 root 20 0 43380 2452 1428 S 0.0 0.2 3:22.61 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.10 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 0:04.54 ksoftirqd/0 7 root rt 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh 9 root 20 0 0 0 0 S 0.0 0.0 1:10.13 rcu_sched 10 root rt 0 0 0 0 S 0.0 0.0 0:03.73 watchdog/0 12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kdevtmpfs 13 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns 14 root 20 0 0 0 0 S 0.0 0.0 0:02.51 khungtaskd 15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback 16 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kintegrityd 17 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset 18 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd 19 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 md 25 root 20 0 0 0 0 S 0.0 0.0 0:26.06 kswapd0 26 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd 27 root 39 19 0 0 0 S 0.0 0.0 0:09.18 khugepaged 28 root 20 0 0 0 0 S 0.0 0.0 0:00.00 fsnotify_mark 29 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto 37 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kthrotld 39 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kmpath_rdacd 40 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kpsmoused 41 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ipv6_addrconf 60 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 deferwq 92 root 20 0 0 0 0 S 0.0 0.0 0:03.85 kauditd 264 root 0 -20 0 0 0 S 0.0 0.0 0:00.05 ata_sff 280 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0 281 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_0 282 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_1 283 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 scsi_tmf_1 285 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ttm_swap 359 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kdmflush 360 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset 371 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kdmflush 372 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset 385 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfsalloc 386 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs_mru_cache 387 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs-buf/dm-0 388 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs-data/dm-0 389 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs-conv/dm-0 390 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs-cil/dm-0 391 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs-reclaim/dm- 392 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs-log/dm-0 393 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 xfs-eofblocks/d 394 root 20 0 0 0 0 S 0.0 0.0 2:56.36 xfsaild/dm-0 469 root 20 0 36120 1340 1200 S 0.0 0.1 2:01.94 systemd-journal 484 root 20 0 127084 0 0 S 0.0 0.0 0:00.02 lvmetad 498 root 20 0 43632 292 288 S 0.0 0.0 0:00.07 systemd-udevd 516 root 20 0 0 0 0 S 0.0 0.0 0:00.00 vballoon 517 root 20 0 0 0 0 S 0.0 0.0 0:06.08 hwrng 581 root 20 0 0 0 0 S 0.0 0.0 0:00.00 jbd2/vda1-8 582 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ext4-rsv-conver 597 root 16 -4 55412 432 312 S 0.0 0.0 0:19.40 auditd 619 avahi 20 0 27976 428 300 S 0.0 0.0 0:13.94 avahi-daemon 620 dbus 20 0 24536 788 456 S 0.0 0.1 2:42.62 dbus-daemon 623 avahi 20 0 27976 24 0 S 0.0 0.0 0:00.00 avahi-daemon 627 root 20 0 105476 160 92 S 0.0 0.0 0:00.04 sshd 629 root 20 0 553148 552 268 S 0.0 0.1 1:26.93 tuned 630 root 20 0 24308 984 768 S 0.0 0.1 1:27.94 systemd-logind 636 root 20 0 6964 180 140 S 0.0 0.0 0:00.20 mdadm 637 root 20 0 262428 1908 1608 S 0.0 0.2 0:59.10 rsyslogd 639 root 20 0 541036 12456 9172 S 0.0 1.2 0:38.44 httpd 645 polkitd 20 0 527508 912 608 S 0.0 0.1 0:59.68 polkitd 653 root 20 0 14988 340 232 S 0.0 0.0 0:00.04 incrond 661 chrony 20 0 115844 728 624 S 0.0 0.1 0:01.74 chronyd 665 nobody 20 0 15540 96 92 S 0.0 0.0 0:00.00 dnsmasq 697 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 cfg80211 708 root 20 0 25840 0 0 S 0.0 0.0 0:00.02 atd 721 root 20 0 126248 808 400 S 0.0 0.1 0:34.60 crond 733 root 20 0 27100 620 616 S 0.0 0.1 0:00.00 xinetd 738 root 20 0 110032 64 60 S 0.0 0.0 0:00.00 agetty 746 asterisk 20 0 749892 49024 27388 S 0.0 4.8 1:35.82 httpd 752 mysql 20 0 113252 52 52 S 0.0 0.0 0:00.01 mysqld_safe 822 asterisk 20 0 748824 47468 27728 S 0.0 4.7 4:24.32 httpd 828 mongodb 20 0 483944 9532 588 S 0.0 0.9 28:42.09 mongod 850 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 dio/dm-0 1034 mysql 20 0 1177564 165648 2656 S 0.0 16.3 59:20.55 mysqld 1148 root 20 0 89032 600 516 S 0.0 0.1 0:04.09 master 1207 root 20 0 112876 468 236 S 0.0 0.0 0:00.06 dhclient 3395 asterisk 20 0 650608 20176 9308 S 0.0 2.0 0:00.04 httpd 3758 asterisk 20 0 657232 38972 20556 S 0.0 3.8 0:45.70 httpd 3843 root 0 -20 0 0 0 S 0.0 0.0 0:00.74 kworker/0:2H 5837 root 20 0 115240 44 40 S 0.0 0.0 0:00.00 safe_asterisk 5844 asterisk 20 0 1645160 41756 5708 S 0.0 4.1 76:50.41 asterisk 6477 root 20 0 0 0 0 S 0.0 0.0 0:00.05 kworker/0:0 7038 asterisk 20 0 746004 43712 23384 S 0.0 4.3 0:35.92 httpd 10255 root 20 0 0 0 0 S 0.0 0.0 0:00.63 kworker/0:1 10422 root 20 0 148384 5936 4588 S 0.0 0.6 0:00.08 sshd 11069 root 0 -20 0 0 0 S 0.0 0.0 0:03.32 kworker/0:1H 14137 asterisk 20 0 747104 47268 28128 S 0.0 4.7 0:10.80 httpd 14138 asterisk 20 0 650384 20712 9536 S 0.0 2.0 0:00.10 httpd 15724 root 20 0 115380 2076 1676 S 0.0 0.2 0:00.02 bash 16487 asterisk 20 0 1005508 19880 4076 S 0.0 2.0 7:23.89 PM2 v2.7.1: God 16503 asterisk 20 0 1214940 18216 1884 S 0.0 1.8 7:22.57 letschat 16833 asterisk 20 0 195328 668 248 S 0.0 0.1 0:00.23 pnp_server 17498 asterisk 20 0 1087448 37676 7188 S 0.0 3.7 0:09.16 node /var/www/h 17615 root 20 0 0 0 0 S 0.0 0.0 0:00.19 kworker/u2:1 19180 asterisk 20 0 741236 28020 13168 S 0.0 2.8 1:13.27 httpd 19600 asterisk 20 0 659456 45256 26600 S 0.0 4.5 0:27.39 httpd 20008 asterisk 20 0 650888 20976 9400 S 0.0 2.1 0:01.55 httpd 20137 asterisk 20 0 390124 16112 9436 S 0.0 1.6 0:00.19 php 20168 asterisk 20 0 650860 20816 9260 S 0.0 2.0 0:01.34 httpd 27522 root 20 0 0 0 0 S 0.0 0.0 0:00.52 kworker/0:2 30030 asterisk 20 0 417620 26224 10636 S 0.0 2.6 0:23.46 php 30330 root 20 0 0 0 0 S 0.0 0.0 0:00.04 kworker/u2:0 30730 postfix 20 0 89136 3572 2572 S 0.0 0.4 0:00.03 pickup 30793 postfix 20 0 89312 2804 2660 S 0.0 0.3 0:00.15 qmgr 31123 root 20 0 398792 11796 4960 S 0.0 1.2 14:21.45 php 31176 root 20 0 185668 1504 1144 S 0.0 0.1 0:09.04 fail2ban-server -
Ah. Reading through the man pages for
sar. Fromsar -qlooks like there were other spikes. Looks like I have some data collection and cross-referencing in my future.
-
One more little tidbit of information. I can replicate the spike by viewing the Connectivity > Firewall page(s).
