FreePBX - High Load Averages



  • This is related to another thread concerning what top is telling us.

    At the moment the FreePBX GUI is running super slow, so I ssh in, and run to top to find load averages of 8.74, 6.74, and 3.57. The last time this happened, I panicked and restarted. This time, I'm much more calm and am ready to figure out the problem.

    It looks like there are a number of httpd processes running; however, I don't know what I should consider "normal." Any verteran FreePBX folks see a smoking gun in the top output that could explain the load averages?

    top - 11:01:30 up 4 days, 23:26,  2 users,  load average: 8.74, 6.74, 3.57
    Tasks: 122 total,   9 running, 112 sleeping,   0 stopped,   1 zombie
    %Cpu(s):  4.9 us,  2.7 sy,  0.0 ni, 92.1 id,  0.2 wa,  0.0 hi,  0.0 si,  0.2 st
    KiB Mem :  1016380 total,   171816 free,   586868 used,   257696 buff/cache
    KiB Swap:  2097148 total,  1733740 free,   363408 used.   182016 avail Mem
    
      PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
        9 root      20   0       0      0      0 R  6.2  0.0   0:49.83 rcu_sched
     4442 root      20   0       0      0      0 S  6.2  0.0   0:00.69 kworker/0:0
    14446 asterisk  20   0  731588  28916  11100 S  6.2  2.8   0:35.91 httpd
    14447 asterisk  20   0  744928  42548  22628 S  6.2  4.2   0:44.56 httpd
    21407 asterisk  20   0  657796  31416  13676 R  6.2  3.1   0:19.66 httpd
        1 root      20   0   43380   2448   1424 S  0.0  0.2   2:44.59 systemd
        2 root      20   0       0      0      0 S  0.0  0.0   0:00.09 kthreadd
        3 root      20   0       0      0      0 S  0.0  0.0   0:03.57 ksoftirqd/0
        7 root      rt   0       0      0      0 S  0.0  0.0   0:00.00 migration/0
        8 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcu_bh
       10 root      rt   0       0      0      0 S  0.0  0.0   0:02.96 watchdog/0
       12 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kdevtmpfs
       13 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 netns
       14 root      20   0       0      0      0 S  0.0  0.0   0:02.03 khungtaskd
       15 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 writeback
       16 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kintegrityd
       17 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
       18 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kblockd
       19 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 md
       25 root      20   0       0      0      0 S  0.0  0.0   0:23.46 kswapd0
       26 root      25   5       0      0      0 S  0.0  0.0   0:00.00 ksmd
       27 root      39  19       0      0      0 S  0.0  0.0   0:07.73 khugepaged
       28 root      20   0       0      0      0 S  0.0  0.0   0:00.00 fsnotify_mark
       29 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 crypto
       37 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kthrotld
       39 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kmpath_rdacd
       40 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kpsmoused
       41 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ipv6_addrconf
       60 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 deferwq
       92 root      20   0       0      0      0 S  0.0  0.0   0:03.09 kauditd
      264 root       0 -20       0      0      0 S  0.0  0.0   0:00.03 ata_sff
      280 root      20   0       0      0      0 S  0.0  0.0   0:00.00 scsi_eh_0
      281 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 scsi_tmf_0
      282 root      20   0       0      0      0 S  0.0  0.0   0:00.00 scsi_eh_1
      283 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 scsi_tmf_1
      285 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ttm_swap
      359 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kdmflush
      360 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
      371 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kdmflush
      372 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
      385 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfsalloc
      386 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs_mru_cache
      387 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-buf/dm-0
      388 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-data/dm-0
      389 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-conv/dm-0
      390 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-cil/dm-0
      391 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-reclaim/dm-
      392 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-log/dm-0
      393 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-eofblocks/d
      394 root      20   0       0      0      0 S  0.0  0.0   2:22.78 xfsaild/dm-0
      469 root      20   0   36120   2476   2336 S  0.0  0.2   1:38.96 systemd-journal
      484 root      20   0  127084      0      0 S  0.0  0.0   0:00.02 lvmetad
      498 root      20   0   43632    292    288 S  0.0  0.0   0:00.07 systemd-udevd
      516 root      20   0       0      0      0 S  0.0  0.0   0:00.00 vballoon
      517 root      20   0       0      0      0 R  0.0  0.0   0:03.62 hwrng
      572 root      20   0       0      0      0 S  0.0  0.0   0:00.20 kworker/u2:0
      581 root      20   0       0      0      0 S  0.0  0.0   0:00.00 jbd2/vda1-8
      582 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ext4-rsv-conver
      597 root      16  -4   55412    428    308 S  0.0  0.0   0:15.56 auditd
      619 avahi     20   0   27976    340    224 S  0.0  0.0   0:11.20 avahi-daemon
      620 dbus      20   0   24536    796    456 S  0.0  0.1   2:12.32 dbus-daemon
      623 avahi     20   0   27976     24      0 S  0.0  0.0   0:00.00 avahi-daemon
      627 root      20   0  105476    204    132 S  0.0  0.0   0:00.03 sshd
      629 root      20   0  553148    552    268 S  0.0  0.1   1:09.54 tuned
      630 root      20   0   24308    984    768 S  0.0  0.1   1:11.51 systemd-logind
      636 root      20   0    6964    144    104 S  0.0  0.0   0:00.15 mdadm
      637 root      20   0  262428   2980   2656 S  0.0  0.3   0:47.73 rsyslogd
      639 root      20   0  541036  18292  12560 S  0.0  1.8   0:30.62 httpd
      645 polkitd   20   0  527508    936    608 S  0.0  0.1   0:48.36 polkitd
      653 root      20   0   14988    420    288 S  0.0  0.0   0:00.03 incrond
      659 root      20   0   27100     24     24 S  0.0  0.0   0:00.00 xinetd
      661 chrony    20   0  115844    884    660 S  0.0  0.1   0:01.41 chronyd
      665 nobody    20   0   15540     96     92 S  0.0  0.0   0:00.00 dnsmasq
      697 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 cfg80211
      708 root      20   0   25840      0      0 S  0.0  0.0   0:00.02 atd
      721 root      20   0  126248    808    400 S  0.0  0.1   0:27.96 crond
      738 root      20   0  110032     64     60 S  0.0  0.0   0:00.00 agetty
      752 mysql     20   0  113252     52     52 S  0.0  0.0   0:00.01 mysqld_safe
      828 mongodb   20   0  483944   9632    608 R  0.0  0.9  22:58.14 mongod
      850 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 dio/dm-0
     1034 mysql     20   0 1175812 165616   2668 S  0.0 16.3  47:01.24 mysqld
     1148 root      20   0   89032    276    192 S  0.0  0.0   0:03.27 master
     1150 postfix   20   0   89312    496    372 S  0.0  0.0   0:00.65 qmgr
     1207 root      20   0  112876    536    304 S  0.0  0.1   0:00.05 dhclient
     2037 asterisk  20   0  390124  16108   9436 S  0.0  1.6   0:00.07 php
     2043 root      20   0       0      0      0 S  0.0  0.0   0:00.08 kworker/0:2
     2105 asterisk  20   0  541036   7228   1428 S  0.0  0.7   0:00.00 httpd
     2106 asterisk  20   0  541036   7228   1428 S  0.0  0.7   0:00.00 httpd
     2107 asterisk  20   0  541036   7228   1428 S  0.0  0.7   0:00.00 httpd
     2108 asterisk  20   0  541036   7228   1428 S  0.0  0.7   0:00.00 httpd
     2943 asterisk  20   0  747208  51848  31612 S  0.0  5.1   0:31.82 httpd
     2948 asterisk  20   0  746180  43004  22044 S  0.0  4.2   0:12.43 httpd
     2949 asterisk  20   0  655456  27236   9760 S  0.0  2.7   0:38.37 httpd
     2951 asterisk  20   0  657644  40812  21508 S  0.0  4.0   0:22.15 httpd
     3843 root       0 -20       0      0      0 S  0.0  0.0   0:00.74 kworker/0:2H
     4574 postfix   20   0   89136   3588   2592 S  0.0  0.4   0:00.02 pickup
     5837 root      20   0  115240     44     40 S  0.0  0.0   0:00.00 safe_asterisk
     5844 asterisk  20   0 1645160  51364   5172 S  0.0  5.1  60:03.66 asterisk
     9113 root      20   0  157568   2036   1500 R  0.0  0.2   0:00.00 top
     9131 asterisk  20   0       0      0      0 Z  0.0  0.0   0:00.00 grep
     9132 asterisk  20   0  744928  19920      0 R  0.0  2.0   0:00.00 httpd
     9133 asterisk  20   0  731588  17816      0 R  0.0  1.8   0:00.00 httpd
     9134 asterisk  20   0  390124   6672      0 R  0.0  0.7   0:00.00 php
     9135 asterisk  20   0  655456  17476      0 R  0.0  1.7   0:00.00 httpd
    14153 root       0 -20       0      0      0 S  0.0  0.0   0:02.08 kworker/0:1H
    14448 asterisk  20   0  659616  46120  26956 S  0.0  4.5   0:22.77 httpd
    14449 asterisk  20   0  745100  45528  27200 S  0.0  4.5   0:22.54 httpd
    16487 asterisk  20   0  928928  16416   2160 S  0.0  1.6   5:57.61 PM2 v2.7.1: God
    16503 asterisk  20   0 1214940  48888   1844 S  0.0  4.8   5:56.35 letschat
    16833 asterisk  20   0  195328    640    232 S  0.0  0.1   0:00.20 pnp_server
    20229 root      20   0       0      0      0 S  0.0  0.0   0:00.40 kworker/0:1
    26840 asterisk  20   0  650904  22204   9396 S  0.0  2.2   0:00.52 httpd
    26928 asterisk  20   0  657360  39272  20248 S  0.0  3.9   0:07.00 httpd
    30030 asterisk  20   0  417364  26320  10636 S  0.0  2.6   0:14.67 php
    30462 root      20   0  148248   4524   3180 S  0.0  0.4   0:00.09 sshd
    30530 root      20   0  115380   1808   1432 S  0.0  0.2   0:00.02 bash
    30857 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kworker/u2:2
    30959 asterisk  20   0 1084876  35248   4968 S  0.0  3.5   0:19.55 node /var/www/h
    31123 root      20   0  398024  11020   4956 S  0.0  1.1  10:52.11 php
    31176 root      20   0  185668   1504   1144 S  0.0  0.1   0:07.17 fail2ban-server
    31775 root      20   0  148384   5932   4584 S  0.0  0.6   0:00.05 sshd
    32603 root      20   0  115380   2024   1648 S  0.0  0.2   0:00.01 bash
    


  • Update: Looks like the spike has fixed itself and the episode lasted for about 20 minutes. The top data from above was about halfway through the episode.

    Current top readout.

    top - 11:23:00 up 4 days, 23:47,  2 users,  load average: 1.87, 0.80, 1.34
    Tasks: 112 total,   1 running, 111 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  4.9 us,  2.7 sy,  0.0 ni, 92.1 id,  0.2 wa,  0.0 hi,  0.0 si,  0.2 st
    KiB Mem :  1016380 total,   164472 free,   543580 used,   308328 buff/cache
    KiB Swap:  2097148 total,  1734024 free,   363124 used.   223956 avail Mem
    
      PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
    28491 root      20   0  157568   2036   1500 R  6.7  0.2   0:00.01 top
        1 root      20   0   43380   2448   1424 S  0.0  0.2   2:45.09 systemd
        2 root      20   0       0      0      0 S  0.0  0.0   0:00.09 kthreadd
        3 root      20   0       0      0      0 S  0.0  0.0   0:03.61 ksoftirqd/0
        7 root      rt   0       0      0      0 S  0.0  0.0   0:00.00 migration/0
        8 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcu_bh
        9 root      20   0       0      0      0 S  0.0  0.0   0:50.28 rcu_sched
       10 root      rt   0       0      0      0 S  0.0  0.0   0:02.97 watchdog/0
       12 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kdevtmpfs
       13 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 netns
       14 root      20   0       0      0      0 S  0.0  0.0   0:02.03 khungtaskd
       15 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 writeback
       16 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kintegrityd
       17 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
       18 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kblockd
       19 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 md
       25 root      20   0       0      0      0 S  0.0  0.0   0:23.46 kswapd0
       26 root      25   5       0      0      0 S  0.0  0.0   0:00.00 ksmd
       27 root      39  19       0      0      0 S  0.0  0.0   0:07.74 khugepaged
       28 root      20   0       0      0      0 S  0.0  0.0   0:00.00 fsnotify_mark
       29 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 crypto
       37 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kthrotld
       39 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kmpath_rdacd
       40 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kpsmoused
       41 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ipv6_addrconf
       60 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 deferwq
       92 root      20   0       0      0      0 S  0.0  0.0   0:03.11 kauditd
      264 root       0 -20       0      0      0 S  0.0  0.0   0:00.03 ata_sff
      280 root      20   0       0      0      0 S  0.0  0.0   0:00.00 scsi_eh_0
      281 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 scsi_tmf_0
      282 root      20   0       0      0      0 S  0.0  0.0   0:00.00 scsi_eh_1
      283 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 scsi_tmf_1
      285 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ttm_swap
      359 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kdmflush
      360 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
      371 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kdmflush
      372 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
      385 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfsalloc
      386 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs_mru_cache
      387 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-buf/dm-0
      388 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-data/dm-0
      389 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-conv/dm-0
      390 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-cil/dm-0
      391 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-reclaim/dm-
      392 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-log/dm-0
      393 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-eofblocks/d
      394 root      20   0       0      0      0 S  0.0  0.0   2:23.22 xfsaild/dm-0
      469 root      20   0   36120   3020   2880 S  0.0  0.3   1:39.27 systemd-journal
      484 root      20   0  127084      0      0 S  0.0  0.0   0:00.02 lvmetad
      498 root      20   0   43632    292    288 S  0.0  0.0   0:00.07 systemd-udevd
      516 root      20   0       0      0      0 S  0.0  0.0   0:00.00 vballoon
      517 root      20   0       0      0      0 S  0.0  0.0   0:03.66 hwrng
      581 root      20   0       0      0      0 S  0.0  0.0   0:00.00 jbd2/vda1-8
      582 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ext4-rsv-conver
      597 root      16  -4   55412    428    308 S  0.0  0.0   0:15.62 auditd
      619 avahi     20   0   27976    340    224 S  0.0  0.0   0:11.23 avahi-daemon
      620 dbus      20   0   24536    796    456 S  0.0  0.1   2:12.72 dbus-daemon
      623 avahi     20   0   27976     24      0 S  0.0  0.0   0:00.00 avahi-daemon
      627 root      20   0  105476    332    236 S  0.0  0.0   0:00.03 sshd
      629 root      20   0  553148    552    268 S  0.0  0.1   1:09.80 tuned
      630 root      20   0   24308    988    772 S  0.0  0.1   1:11.72 systemd-logind
      636 root      20   0    6964    180    140 S  0.0  0.0   0:00.15 mdadm
      637 root      20   0  262428   3476   3152 S  0.0  0.3   0:47.90 rsyslogd
      639 root      20   0  541036  18292  12560 S  0.0  1.8   0:30.71 httpd
      645 polkitd   20   0  527508    936    608 S  0.0  0.1   0:48.51 polkitd
      653 root      20   0   14988    420    288 S  0.0  0.0   0:00.03 incrond
      659 root      20   0   27100     24     24 S  0.0  0.0   0:00.00 xinetd
      661 chrony    20   0  115844    868    652 S  0.0  0.1   0:01.42 chronyd
      665 nobody    20   0   15540     96     92 S  0.0  0.0   0:00.00 dnsmasq
      697 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 cfg80211
      708 root      20   0   25840      0      0 S  0.0  0.0   0:00.02 atd
      721 root      20   0  126248    808    400 S  0.0  0.1   0:28.04 crond
      738 root      20   0  110032     64     60 S  0.0  0.0   0:00.00 agetty
      752 mysql     20   0  113252     52     52 S  0.0  0.0   0:00.01 mysqld_safe
      828 mongodb   20   0  483944   9632    608 S  0.0  0.9  23:02.65 mongod
      850 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 dio/dm-0
     1034 mysql     20   0 1175812 165620   2668 S  0.0 16.3  47:24.37 mysqld
     1148 root      20   0   89032    276    192 S  0.0  0.0   0:03.28 master
     1150 postfix   20   0   89312    496    372 S  0.0  0.0   0:00.65 qmgr
     1207 root      20   0  112876    536    304 S  0.0  0.1   0:00.05 dhclient
     2037 asterisk  20   0  390124  16112   9440 S  0.0  1.6   0:02.46 php
     2108 asterisk  20   0  649804  21732   9704 S  0.0  2.1   0:09.89 httpd
     2943 asterisk  20   0  747464  52088  31572 S  0.0  5.1   0:46.08 httpd
     2949 asterisk  20   0  651364  22804   9760 S  0.0  2.2   0:46.66 httpd
     2951 asterisk  20   0  657644  40948  21644 S  0.0  4.0   0:29.10 httpd
     3843 root       0 -20       0      0      0 S  0.0  0.0   0:00.74 kworker/0:2H
     4574 postfix   20   0   89136   3588   2592 S  0.0  0.4   0:00.03 pickup
     5837 root      20   0  115240     44     40 S  0.0  0.0   0:00.00 safe_asterisk
     5844 asterisk  20   0 1645160  51940   5540 S  0.0  5.1  60:21.48 asterisk
    14153 root       0 -20       0      0      0 S  0.0  0.0   0:02.17 kworker/0:1H
    14446 asterisk  20   0  725692  23224  11100 S  0.0  2.3   0:43.66 httpd
    14447 asterisk  20   0  745696  43244  22556 S  0.0  4.3   0:56.37 httpd
    14448 asterisk  20   0  659616  46120  26956 S  0.0  4.5   0:33.52 httpd
    14449 asterisk  20   0  745100  45560  27228 S  0.0  4.5   0:31.79 httpd
    16487 asterisk  20   0  928928  16728   2160 S  0.0  1.6   5:58.75 PM2 v2.7.1: God
    16503 asterisk  20   0 1214940  48604   1844 S  0.0  4.8   5:57.58 letschat
    16833 asterisk  20   0  195328    640    232 S  0.0  0.1   0:00.20 pnp_server
    20229 root      20   0       0      0      0 S  0.0  0.0   0:00.52 kworker/0:1
    21407 asterisk  20   0  652420  26280  13676 S  0.0  2.6   0:33.10 httpd
    23805 root      20   0       0      0      0 S  0.0  0.0   0:00.01 kworker/u2:0
    24324 root      20   0       0      0      0 S  0.0  0.0   0:00.57 kworker/0:0
    26812 root      20   0  148384   5936   4588 S  0.0  0.6   0:00.10 sshd
    26840 asterisk  20   0  651172  22260   9416 S  0.0  2.2   0:07.31 httpd
    26856 root      20   0  115380   2048   1656 S  0.0  0.2   0:00.01 bash
    27117 root      20   0       0      0      0 S  0.0  0.0   0:00.19 kworker/0:2
    30030 asterisk  20   0  417364  26320  10636 S  0.0  2.6   0:14.90 php
    30462 root      20   0  148248   4524   3180 S  0.0  0.4   0:00.09 sshd
    30530 root      20   0  115380   1808   1432 S  0.0  0.2   0:00.02 bash
    30857 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kworker/u2:2
    30959 asterisk  20   0 1085388  35716   5012 S  0.0  3.5   0:20.63 node /var/www/h
    31123 root      20   0  398024  11020   4956 S  0.0  1.1  10:54.80 php
    31176 root      20   0  185668   1504   1144 S  0.0  0.1   0:07.21 fail2ban-server
    


  • By default you will have about 8-12 httpd processes. How many did you have?



  • My guess is that you are waiting on disks. What does your SAR report say?



  • Using sar -d should tell you what @scottalanmiller is looking for. (I think)



  • @eddiejennings said in FreePBX - High Load Averages:

    This is related to another thread concerning what top is telling us.

    At the moment the FreePBX GUI is running super slow, so I ssh in, and run to top to find load averages of 8.74, 6.74, and 3.57. The last time this happened, I panicked and restarted. This time, I'm much more calm and am ready to figure out the problem.

    It looks like there are a number of httpd processes running; however, I don't know what I should consider "normal." Any verteran FreePBX folks see a smoking gun in the top output that could explain the load averages?

    top - 11:01:30 up 4 days, 23:26,  2 users,  load average: 8.74, 6.74, 3.57
    Tasks: 122 total,   9 running, 112 sleeping,   0 stopped,   1 zombie
    %Cpu(s):  4.9 us,  2.7 sy,  0.0 ni, 92.1 id,  0.2 wa,  0.0 hi,  0.0 si,  0.2 st
    KiB Mem :  1016380 total,   171816 free,   586868 used,   257696 buff/cache
    KiB Swap:  2097148 total,  1733740 free,   363408 used.   182016 avail Mem
    
      PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
        9 root      20   0       0      0      0 R  6.2  0.0   0:49.83 rcu_sched
     4442 root      20   0       0      0      0 S  6.2  0.0   0:00.69 kworker/0:0
    14446 asterisk  20   0  731588  28916  11100 S  6.2  2.8   0:35.91 httpd
    14447 asterisk  20   0  744928  42548  22628 S  6.2  4.2   0:44.56 httpd
    21407 asterisk  20   0  657796  31416  13676 R  6.2  3.1   0:19.66 httpd
        1 root      20   0   43380   2448   1424 S  0.0  0.2   2:44.59 systemd
        2 root      20   0       0      0      0 S  0.0  0.0   0:00.09 kthreadd
        3 root      20   0       0      0      0 S  0.0  0.0   0:03.57 ksoftirqd/0
        7 root      rt   0       0      0      0 S  0.0  0.0   0:00.00 migration/0
        8 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcu_bh
       10 root      rt   0       0      0      0 S  0.0  0.0   0:02.96 watchdog/0
       12 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kdevtmpfs
       13 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 netns
       14 root      20   0       0      0      0 S  0.0  0.0   0:02.03 khungtaskd
       15 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 writeback
       16 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kintegrityd
       17 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
       18 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kblockd
       19 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 md
       25 root      20   0       0      0      0 S  0.0  0.0   0:23.46 kswapd0
       26 root      25   5       0      0      0 S  0.0  0.0   0:00.00 ksmd
       27 root      39  19       0      0      0 S  0.0  0.0   0:07.73 khugepaged
       28 root      20   0       0      0      0 S  0.0  0.0   0:00.00 fsnotify_mark
       29 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 crypto
       37 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kthrotld
       39 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kmpath_rdacd
       40 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kpsmoused
       41 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ipv6_addrconf
       60 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 deferwq
       92 root      20   0       0      0      0 S  0.0  0.0   0:03.09 kauditd
      264 root       0 -20       0      0      0 S  0.0  0.0   0:00.03 ata_sff
      280 root      20   0       0      0      0 S  0.0  0.0   0:00.00 scsi_eh_0
      281 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 scsi_tmf_0
      282 root      20   0       0      0      0 S  0.0  0.0   0:00.00 scsi_eh_1
      283 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 scsi_tmf_1
      285 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ttm_swap
      359 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kdmflush
      360 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
      371 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kdmflush
      372 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
      385 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfsalloc
      386 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs_mru_cache
      387 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-buf/dm-0
      388 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-data/dm-0
      389 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-conv/dm-0
      390 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-cil/dm-0
      391 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-reclaim/dm-
      392 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-log/dm-0
      393 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-eofblocks/d
      394 root      20   0       0      0      0 S  0.0  0.0   2:22.78 xfsaild/dm-0
      469 root      20   0   36120   2476   2336 S  0.0  0.2   1:38.96 systemd-journal
      484 root      20   0  127084      0      0 S  0.0  0.0   0:00.02 lvmetad
      498 root      20   0   43632    292    288 S  0.0  0.0   0:00.07 systemd-udevd
      516 root      20   0       0      0      0 S  0.0  0.0   0:00.00 vballoon
      517 root      20   0       0      0      0 R  0.0  0.0   0:03.62 hwrng
      572 root      20   0       0      0      0 S  0.0  0.0   0:00.20 kworker/u2:0
      581 root      20   0       0      0      0 S  0.0  0.0   0:00.00 jbd2/vda1-8
      582 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ext4-rsv-conver
      597 root      16  -4   55412    428    308 S  0.0  0.0   0:15.56 auditd
      619 avahi     20   0   27976    340    224 S  0.0  0.0   0:11.20 avahi-daemon
      620 dbus      20   0   24536    796    456 S  0.0  0.1   2:12.32 dbus-daemon
      623 avahi     20   0   27976     24      0 S  0.0  0.0   0:00.00 avahi-daemon
      627 root      20   0  105476    204    132 S  0.0  0.0   0:00.03 sshd
      629 root      20   0  553148    552    268 S  0.0  0.1   1:09.54 tuned
      630 root      20   0   24308    984    768 S  0.0  0.1   1:11.51 systemd-logind
      636 root      20   0    6964    144    104 S  0.0  0.0   0:00.15 mdadm
      637 root      20   0  262428   2980   2656 S  0.0  0.3   0:47.73 rsyslogd
      639 root      20   0  541036  18292  12560 S  0.0  1.8   0:30.62 httpd
      645 polkitd   20   0  527508    936    608 S  0.0  0.1   0:48.36 polkitd
      653 root      20   0   14988    420    288 S  0.0  0.0   0:00.03 incrond
      659 root      20   0   27100     24     24 S  0.0  0.0   0:00.00 xinetd
      661 chrony    20   0  115844    884    660 S  0.0  0.1   0:01.41 chronyd
      665 nobody    20   0   15540     96     92 S  0.0  0.0   0:00.00 dnsmasq
      697 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 cfg80211
      708 root      20   0   25840      0      0 S  0.0  0.0   0:00.02 atd
      721 root      20   0  126248    808    400 S  0.0  0.1   0:27.96 crond
      738 root      20   0  110032     64     60 S  0.0  0.0   0:00.00 agetty
      752 mysql     20   0  113252     52     52 S  0.0  0.0   0:00.01 mysqld_safe
      828 mongodb   20   0  483944   9632    608 R  0.0  0.9  22:58.14 mongod
      850 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 dio/dm-0
     1034 mysql     20   0 1175812 165616   2668 S  0.0 16.3  47:01.24 mysqld
     1148 root      20   0   89032    276    192 S  0.0  0.0   0:03.27 master
     1150 postfix   20   0   89312    496    372 S  0.0  0.0   0:00.65 qmgr
     1207 root      20   0  112876    536    304 S  0.0  0.1   0:00.05 dhclient
     2037 asterisk  20   0  390124  16108   9436 S  0.0  1.6   0:00.07 php
     2043 root      20   0       0      0      0 S  0.0  0.0   0:00.08 kworker/0:2
     2105 asterisk  20   0  541036   7228   1428 S  0.0  0.7   0:00.00 httpd
     2106 asterisk  20   0  541036   7228   1428 S  0.0  0.7   0:00.00 httpd
     2107 asterisk  20   0  541036   7228   1428 S  0.0  0.7   0:00.00 httpd
     2108 asterisk  20   0  541036   7228   1428 S  0.0  0.7   0:00.00 httpd
     2943 asterisk  20   0  747208  51848  31612 S  0.0  5.1   0:31.82 httpd
     2948 asterisk  20   0  746180  43004  22044 S  0.0  4.2   0:12.43 httpd
     2949 asterisk  20   0  655456  27236   9760 S  0.0  2.7   0:38.37 httpd
     2951 asterisk  20   0  657644  40812  21508 S  0.0  4.0   0:22.15 httpd
     3843 root       0 -20       0      0      0 S  0.0  0.0   0:00.74 kworker/0:2H
     4574 postfix   20   0   89136   3588   2592 S  0.0  0.4   0:00.02 pickup
     5837 root      20   0  115240     44     40 S  0.0  0.0   0:00.00 safe_asterisk
     5844 asterisk  20   0 1645160  51364   5172 S  0.0  5.1  60:03.66 asterisk
     9113 root      20   0  157568   2036   1500 R  0.0  0.2   0:00.00 top
     9131 asterisk  20   0       0      0      0 Z  0.0  0.0   0:00.00 grep
     9132 asterisk  20   0  744928  19920      0 R  0.0  2.0   0:00.00 httpd
     9133 asterisk  20   0  731588  17816      0 R  0.0  1.8   0:00.00 httpd
     9134 asterisk  20   0  390124   6672      0 R  0.0  0.7   0:00.00 php
     9135 asterisk  20   0  655456  17476      0 R  0.0  1.7   0:00.00 httpd
    14153 root       0 -20       0      0      0 S  0.0  0.0   0:02.08 kworker/0:1H
    14448 asterisk  20   0  659616  46120  26956 S  0.0  4.5   0:22.77 httpd
    14449 asterisk  20   0  745100  45528  27200 S  0.0  4.5   0:22.54 httpd
    16487 asterisk  20   0  928928  16416   2160 S  0.0  1.6   5:57.61 PM2 v2.7.1: God
    16503 asterisk  20   0 1214940  48888   1844 S  0.0  4.8   5:56.35 letschat
    16833 asterisk  20   0  195328    640    232 S  0.0  0.1   0:00.20 pnp_server
    20229 root      20   0       0      0      0 S  0.0  0.0   0:00.40 kworker/0:1
    26840 asterisk  20   0  650904  22204   9396 S  0.0  2.2   0:00.52 httpd
    26928 asterisk  20   0  657360  39272  20248 S  0.0  3.9   0:07.00 httpd
    30030 asterisk  20   0  417364  26320  10636 S  0.0  2.6   0:14.67 php
    30462 root      20   0  148248   4524   3180 S  0.0  0.4   0:00.09 sshd
    30530 root      20   0  115380   1808   1432 S  0.0  0.2   0:00.02 bash
    30857 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kworker/u2:2
    30959 asterisk  20   0 1084876  35248   4968 S  0.0  3.5   0:19.55 node /var/www/h
    31123 root      20   0  398024  11020   4956 S  0.0  1.1  10:52.11 php
    31176 root      20   0  185668   1504   1144 S  0.0  0.1   0:07.17 fail2ban-server
    31775 root      20   0  148384   5932   4584 S  0.0  0.6   0:00.05 sshd
    32603 root      20   0  115380   2024   1648 S  0.0  0.2   0:00.01 bash
    

    What size instance did you go for on Vultr? Not relevant, but curious.



  • I failed to run sar in a timely fashion. When this spike occurs again, I'll know to run it.



  • @eddiejennings said in FreePBX - High Load Averages:

    I failed to run sar in a timely fashion. When this spike occurs again, I'll know to run it.

    If you don't have sar installed already, go ahead and install it so you know what "good" looks like too. 🙂



  • @dafyre said in FreePBX - High Load Averages:

    @eddiejennings said in FreePBX - High Load Averages:

    I failed to run sar in a timely fashion. When this spike occurs again, I'll know to run it.

    If you don't have sar installed already, go ahead and install it so you know what "good" looks like too. 🙂

    Yeah, I ran it and got a copy of the results. That how I knew I missed my opportunity.



  • @bigbear said in FreePBX - High Load Averages:

    What size instance did you go for on Vultr? Not relevant, but curious.

    The 1 CPU, 1 GB RAM, 25 GB SSD instance ($5 / month).



  • @scottalanmiller said in FreePBX - High Load Averages:

    By default you will have about 8-12 httpd processes. How many did you have?

    19 httpd processes during the problem time. 11 when the problem went away.



  • @eddiejennings said in FreePBX - High Load Averages:

    @dafyre said in FreePBX - High Load Averages:

    @eddiejennings said in FreePBX - High Load Averages:

    I failed to run sar in a timely fashion. When this spike occurs again, I'll know to run it.

    If you don't have sar installed already, go ahead and install it so you know what "good" looks like too. 🙂

    Yeah, I ran it and got a copy of the results. That how I knew I missed my opportunity.

    Did you check the logs (usually stored in /var/log/sysstat/)?

    Try sar -f /var/log/sysstat/sa02 to see the stats from yesterday.



  • In the middle of an episode now. I'm collecting data and will post in a moment.



  • Here are the sar -d results.

    For ease of reading, I put the window where the load average spike was at the top. Next is top from about 2/3 of the way through. Since this seems to solve itself, I'm more curious as to figure out what's causing tasks to be queued.

    Linux 3.10.0-514.26.2.el7.x86_64 (blah.domain.com)        10/10/2017      _x86_64_        (1 CPU)
    
    				 DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
    01:30:02 PM  dev252-0      9.94      0.74    385.64     38.87      0.01      0.86      0.18      0.18
    01:30:02 PM  dev253-0      9.81      0.66    385.64     39.40      0.01      0.90      0.19      0.18
    01:30:02 PM  dev253-1      0.01      0.08      0.00      8.00      0.00      1.00      0.17      0.00
    01:40:02 PM  dev252-0      9.34      6.86    354.74     38.70      0.01      0.81      0.17      0.16
    01:40:02 PM  dev253-0      9.18      5.87    354.74     39.28      0.01      0.84      0.18      0.16
    01:40:02 PM  dev253-1      0.12      0.99      0.00      8.00      0.00      0.56      0.10      0.00
    01:50:01 PM  dev252-0     12.80    251.00    333.08     45.64      0.01      1.16      0.30      0.38
    01:50:01 PM  dev253-0     12.25    238.55    313.44     45.07      0.01      1.16      0.28      0.34
    01:50:01 PM  dev253-1      4.01     12.45     19.65      8.00      0.07     18.65      0.12      0.05
    
    
    Linux 3.10.0-514.26.2.el7.x86_64 (blah.domain.com)        10/10/2017      _x86_64_        (1 CPU)
    
    12:00:02 AM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
    12:10:01 AM  dev252-0     12.24    233.79    399.69     51.75      0.02      1.27      0.31      0.38
    12:10:01 AM  dev253-0     12.07    233.47    379.27     50.76      0.01      1.14      0.27      0.33
    12:10:01 AM  dev253-1      2.59      0.33     20.42      8.00      0.13     49.31      0.26      0.07
    12:20:01 AM  dev252-0     11.28    129.02    334.39     41.08      0.01      1.19      0.30      0.33
    12:20:01 AM  dev253-0      9.72     80.71    328.20     42.05      0.01      1.27      0.26      0.26
    12:20:01 AM  dev253-1      6.81     48.31      6.19      8.00      0.03      4.17      0.12      0.08
    12:30:01 AM  dev252-0      8.89      0.37    339.30     38.20      0.01      1.01      0.20      0.18
    12:30:01 AM  dev253-0      8.75      0.00    339.30     38.77      0.01      1.04      0.20      0.18
    12:30:01 AM  dev253-1      0.05      0.37      0.00      8.00      0.00      0.67      0.11      0.00
    12:40:01 AM  dev252-0      7.81      0.45    297.80     38.19      0.01      0.83      0.19      0.15
    12:40:01 AM  dev253-0      7.66      0.03    294.60     38.46      0.01      0.87      0.19      0.15
    12:40:01 AM  dev253-1      0.45      0.42      3.20      8.00      0.00      1.93      0.04      0.00
    12:50:02 AM  dev252-0      9.27      0.34    362.79     39.18      0.01      0.69      0.16      0.15
    12:50:02 AM  dev253-0      9.13      0.09    361.99     39.66      0.01      0.72      0.16      0.15
    12:50:02 AM  dev253-1      0.13      0.24      0.80      8.00      0.00      1.35      0.12      0.00
    01:00:02 AM  dev252-0      8.93     21.75    326.97     39.07      0.01      0.77      0.16      0.15
    01:00:02 AM  dev253-0      8.78     21.45    326.97     39.69      0.01      0.80      0.17      0.15
    01:00:02 AM  dev253-1      0.04      0.30      0.00      8.00      0.00      0.41      0.09      0.00
    01:10:02 AM  dev252-0     12.50    256.42    408.88     53.23      0.01      1.09      0.27      0.33
    01:10:02 AM  dev253-0     12.31    256.17    378.13     51.53      0.01      0.95      0.20      0.25
    01:10:02 AM  dev253-1      3.88      0.26     30.74      8.00      0.20     51.04      0.24      0.09
    01:20:02 AM  dev252-0      9.64      6.37    345.35     36.49      0.01      0.79      0.18      0.18
    01:20:02 AM  dev253-0      9.47      6.15    336.18     36.14      0.01      0.82      0.19      0.18
    01:20:02 AM  dev253-1      1.17      0.22      9.16      8.00      0.00      2.71      0.02      0.00
    01:30:01 AM  dev252-0      9.63     10.99    366.75     39.22      0.01      0.89      0.17      0.16
    01:30:01 AM  dev253-0      9.49     10.78    366.75     39.79      0.01      0.92      0.17      0.16
    01:30:01 AM  dev253-1      0.03      0.20      0.00      8.00      0.00      0.87      0.20      0.00
    01:40:01 AM  dev252-0      9.94     40.40    369.56     41.23      0.01      0.88      0.18      0.18
    01:40:01 AM  dev253-0      9.78     40.21    363.37     41.28      0.01      0.90      0.18      0.18
    01:40:01 AM  dev253-1      0.80      0.19      6.18      8.00      0.01      7.92      0.08      0.01
    01:50:01 AM  dev252-0      9.05     39.06    333.04     41.10      0.01      0.85      0.17      0.15
    01:50:01 AM  dev253-0      8.88     38.42    327.97     41.26      0.01      0.88      0.17      0.15
    01:50:01 AM  dev253-1      0.72      0.64      5.09      8.00      0.00      6.84      0.06      0.00
    02:00:01 AM  dev252-0      8.61      0.20    330.00     38.33      0.01      0.84      0.17      0.15
    02:00:01 AM  dev253-0      8.42      0.01    329.98     39.17      0.01      0.87      0.18      0.15
    02:00:01 AM  dev253-1      0.02      0.19      0.00      8.00      0.00      0.57      0.14      0.00
    02:10:01 AM  dev252-0     10.85    178.08    373.68     50.83      0.01      1.00      0.24      0.26
    02:10:01 AM  dev253-0     10.66    177.83    364.17     50.85      0.01      0.95      0.19      0.20
    02:10:01 AM  dev253-1      1.22      0.24      9.51      8.00      0.05     42.90      0.51      0.06
    02:20:01 AM  dev252-0      8.84      5.29    330.72     38.02      0.01      0.80      0.15      0.14
    02:20:01 AM  dev253-0      8.68      4.91    330.72     38.68      0.01      0.83      0.16      0.14
    02:20:01 AM  dev253-1      0.05      0.38      0.00      8.00      0.00      0.71      0.14      0.00
    02:30:01 AM  dev252-0      8.63      0.92    329.72     38.31      0.01      0.78      0.16      0.14
    02:30:01 AM  dev253-0      8.48      0.00    329.72     38.89      0.01      0.81      0.17      0.14
    02:30:01 AM  dev253-1      0.12      0.92      0.00      8.00      0.00      0.50      0.09      0.00
    02:40:01 AM  dev252-0     10.39     18.80    401.01     40.39      0.01      0.86      0.17      0.18
    02:40:01 AM  dev253-0     10.24     18.65    401.01     40.98      0.01      0.88      0.18      0.18
    02:40:01 AM  dev253-1      0.02      0.15      0.00      8.00      0.00      1.00      0.18      0.00
    02:50:01 AM  dev252-0      9.64     79.81    346.50     44.24      0.01      0.90      0.19      0.19
    02:50:01 AM  dev253-0      9.32     75.35    328.13     43.29      0.01      0.89      0.18      0.17
    02:50:01 AM  dev253-1      2.85      4.46     18.38      8.00      0.06     19.69      0.07      0.02
    03:00:02 AM  dev252-0      8.91     16.85    323.36     38.17      0.01      0.77      0.17      0.15
    03:00:02 AM  dev253-0      8.74     16.46    323.36     38.89      0.01      0.81      0.18      0.15
    03:00:02 AM  dev253-1      0.05      0.39      0.00      8.00      0.00      0.28      0.03      0.00
    03:10:01 AM  dev252-0     11.32    259.35    366.04     55.27      0.02      1.95      0.30      0.34
    03:10:01 AM  dev253-0     11.09    258.06    337.96     53.72      0.02      1.96      0.30      0.33
    03:10:01 AM  dev253-1      3.67      1.29     28.08      8.00      0.04     10.70      0.05      0.02
    03:20:01 AM  dev252-0      8.81      8.00    322.37     37.49      0.01      0.98      0.19      0.17
    03:20:01 AM  dev253-0      8.64      5.87    322.37     37.98      0.01      1.02      0.20      0.17
    03:20:01 AM  dev253-1      0.27      2.13      0.00      8.00      0.00      0.55      0.08      0.00
    03:30:01 AM  dev252-0      8.83     25.89    326.49     39.90      0.01      0.80      0.18      0.16
    03:30:01 AM  dev253-0      8.69     22.91    326.49     40.23      0.01      0.83      0.19      0.16
    03:30:01 AM  dev253-1      0.37      2.98      0.00      8.00      0.00      0.52      0.08      0.00
    03:40:01 AM  dev252-0      8.53      0.14    322.66     37.84      0.01      0.80      0.17      0.15
    03:40:01 AM  dev253-0      8.40      0.00    322.66     38.43      0.01      0.83      0.18      0.15
    03:40:01 AM  dev253-1      0.02      0.14      0.00      8.00      0.00      0.80      0.10      0.00
    03:50:01 AM  dev252-0      9.30      0.00    359.01     38.61      0.01      0.77      0.17      0.15
    03:50:01 AM  dev253-0      9.17      0.00    359.01     39.15      0.01      0.80      0.17      0.16
    03:50:01 AM  dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
    04:00:01 AM  dev252-0      9.12      0.88    354.46     38.98      0.01      0.80      0.17      0.16
    04:00:01 AM  dev253-0      8.97      0.01    354.46     39.52      0.01      0.83      0.18      0.16
    04:00:01 AM  dev253-1      0.11      0.87      0.00      8.00      0.00      0.39      0.06      0.00
    04:10:01 AM  dev252-0     18.91    237.02    391.35     33.23      0.02      1.23      0.51      0.96
    04:10:01 AM  dev253-0     18.68    236.53    364.37     32.17      0.02      1.26      0.50      0.93
    04:10:01 AM  dev253-1      3.43      0.49     26.98      8.00      0.09     26.85      0.11      0.04
    04:20:01 AM  dev252-0      8.17     16.84    309.93     39.98      0.01      1.18      0.23      0.19
    04:20:01 AM  dev253-0      8.06     16.75    309.93     40.52      0.01      1.23      0.24      0.19
    04:20:01 AM  dev253-1      0.01      0.10      0.00      8.00      0.00      1.00      0.14      0.00
    04:30:02 AM  dev252-0      8.84      0.00    341.94     38.69      0.01      0.78      0.18      0.16
    04:30:02 AM  dev253-0      8.72      0.00    341.94     39.22      0.01      0.81      0.18      0.16
    04:30:02 AM  dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
    04:40:02 AM  dev252-0      8.16      0.11    306.14     37.52      0.01      0.86      0.17      0.14
    04:40:02 AM  dev253-0      8.02      0.00    306.14     38.19      0.01      0.90      0.17      0.14
    04:40:02 AM  dev253-1      0.01      0.11      0.00      8.00      0.00      0.12      0.12      0.00
    04:50:01 AM  dev252-0      7.09      9.28    251.23     36.76      0.01      0.82      0.16      0.11
    04:50:01 AM  dev253-0      7.00      9.28    251.23     37.24      0.01      0.86      0.17      0.12
    04:50:01 AM  dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
    05:00:02 AM  dev252-0      8.72      0.07    333.19     38.23      0.01      0.80      0.18      0.16
    05:00:02 AM  dev253-0      8.58      0.00    333.19     38.86      0.01      0.83      0.18      0.16
    05:00:02 AM  dev253-1      0.01      0.07      0.00      8.00      0.00      0.00      0.00      0.00
    05:10:02 AM  dev252-0     10.97    163.28    358.36     47.57      0.01      1.13      0.23      0.26
    05:10:02 AM  dev253-0     10.78    163.06    337.65     46.46      0.01      0.96      0.19      0.20
    05:10:02 AM  dev253-1      2.62      0.22     20.71      8.00      0.21     79.52      0.23      0.06
    05:20:01 AM  dev252-0      8.23      7.02    303.95     37.79      0.01      0.90      0.18      0.15
    05:20:01 AM  dev253-0      8.06      6.67    303.95     38.53      0.01      0.93      0.18      0.14
    05:20:01 AM  dev253-1      0.04      0.35      0.00      8.00      0.00      1.00      0.85      0.00
    05:30:01 AM  dev252-0      9.80      0.52    388.72     39.72      0.01      0.82      0.17      0.16
    05:30:01 AM  dev253-0      9.63      0.41    388.72     40.42      0.01      0.85      0.17      0.17
    05:30:01 AM  dev253-1      0.01      0.11      0.00      8.00      0.00      1.00      0.13      0.00
    05:40:01 AM  dev252-0      9.62      0.01    380.32     39.53      0.01      0.84      0.17      0.16
    05:40:01 AM  dev253-0      9.47      0.01    380.32     40.16      0.01      0.87      0.17      0.16
    05:40:01 AM  dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
    05:50:01 AM  dev252-0      7.96      0.11    309.45     38.89      0.01      0.87      0.17      0.14
    05:50:01 AM  dev253-0      7.84      0.00    309.45     39.49      0.01      0.90      0.18      0.14
    05:50:01 AM  dev253-1      0.01      0.11      0.00      8.00      0.00      1.00      0.12      0.00
    06:00:01 AM  dev252-0      8.85     15.84    336.92     39.88      0.01      0.86      0.16      0.15
    06:00:01 AM  dev253-0      8.71     15.84    336.92     40.48      0.01      0.90      0.17      0.15
    06:00:01 AM  dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
    06:10:02 AM  dev252-0     11.35    142.47    406.56     48.35      0.01      1.13      0.22      0.25
    06:10:02 AM  dev253-0     11.17    142.33    368.50     45.73      0.01      0.91      0.19      0.21
    06:10:02 AM  dev253-1      4.77      0.14     38.06      8.00      0.30     63.66      0.09      0.04
    06:20:01 AM  dev252-0      9.11      0.01    348.27     38.25      0.01      0.87      0.16      0.15
    06:20:01 AM  dev253-0      8.97      0.01    348.27     38.82      0.01      0.90      0.17      0.15
    06:20:01 AM  dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
    06:30:01 AM  dev252-0      9.57      0.00    370.63     38.74      0.01      0.79      0.17      0.16
    06:30:01 AM  dev253-0      9.46      0.00    370.63     39.19      0.01      0.82      0.17      0.16
    06:30:01 AM  dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
    06:40:01 AM  dev252-0      8.84      0.11    339.69     38.46      0.01      0.85      0.18      0.16
    06:40:01 AM  dev253-0      8.70      0.00    339.69     39.05      0.01      0.88      0.19      0.16
    06:40:01 AM  dev253-1      0.01      0.11      0.00      8.00      0.00      1.00      0.13      0.00
    06:50:02 AM  dev252-0      8.75      0.12    336.83     38.53      0.01      0.91      0.18      0.16
    06:50:02 AM  dev253-0      8.65      0.01    336.83     38.94      0.01      0.95      0.19      0.16
    06:50:02 AM  dev253-1      0.01      0.11      0.00      8.00      0.00      1.00      0.12      0.00
    07:00:01 AM  dev252-0      6.96      0.11    252.95     36.35      0.01      0.72      0.17      0.12
    07:00:01 AM  dev253-0      6.83      0.00    252.95     37.01      0.01      0.76      0.17      0.12
    07:00:01 AM  dev253-1      0.01      0.11      0.00      8.00      0.00      1.00      0.12      0.00
    07:10:01 AM  dev252-0      9.22     70.69    311.74     41.48      0.01      0.89      0.21      0.19
    07:10:01 AM  dev253-0      9.12     70.58    306.22     41.30      0.01      0.86      0.18      0.16
    07:10:01 AM  dev253-1      0.70      0.11      5.53      8.00      0.04     61.35      0.49      0.03
    07:20:01 AM  dev252-0      7.77      0.07    295.80     38.06      0.01      0.78      0.18      0.14
    07:20:01 AM  dev253-0      7.66      0.00    295.80     38.64      0.01      0.82      0.18      0.14
    07:20:01 AM  dev253-1      0.01      0.07      0.00      8.00      0.00      0.00      0.00      0.00
    07:30:01 AM  dev252-0      9.43      0.00    379.00     40.21      0.01      0.87      0.17      0.16
    07:30:01 AM  dev253-0      9.31      0.00    379.00     40.71      0.01      0.90      0.18      0.17
    07:30:01 AM  dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
    07:40:02 AM  dev252-0      9.80      0.50    384.10     39.24      0.01      0.88      0.17      0.17
    07:40:02 AM  dev253-0      9.67      0.50    384.10     39.78      0.01      0.91      0.17      0.17
    07:40:02 AM  dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
    07:50:01 AM  dev252-0     10.26      0.23    403.08     39.32      0.01      0.84      0.16      0.16
    07:50:01 AM  dev253-0     10.13      0.00    403.08     39.78      0.01      0.87      0.16      0.16
    07:50:01 AM  dev253-1      0.03      0.23      0.00      8.00      0.00      1.41      0.24      0.00
    08:00:01 AM  dev252-0      9.67      0.00    380.94     39.38      0.01      0.95      0.17      0.17
    08:00:01 AM  dev253-0      9.55      0.00    380.94     39.91      0.01      0.98      0.18      0.17
    08:00:01 AM  dev253-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
    08:10:01 AM  dev252-0      8.58     33.98    309.32     40.01      0.01      1.03      0.20      0.17
    08:10:01 AM  dev253-0      8.46     33.71    295.24     38.88      0.01      0.94      0.17      0.14
    08:10:01 AM  dev253-1      1.79      0.27     14.08      8.00      0.08     42.57      0.19      0.03
    08:20:01 AM  dev252-0      9.54      0.10    363.35     38.11      0.01      0.82      0.17      0.16
    08:20:01 AM  dev253-0      9.37      0.00    363.35     38.77      0.01      0.85      0.17      0.16
    08:20:01 AM  dev253-1      0.01      0.10      0.00      8.00      0.00      0.00      0.00      0.00
    08:30:01 AM  dev252-0      7.87      0.26    292.76     37.23      0.01      0.81      0.17      0.13
    08:30:01 AM  dev253-0      7.74      0.00    292.76     37.85      0.01      0.84      0.17      0.13
    08:30:01 AM  dev253-1      0.03      0.26      0.00      8.00      0.00      0.84      0.16      0.00
    08:40:01 AM  dev252-0     12.61    295.61    340.57     50.43      0.01      1.09      0.24      0.31
    08:40:01 AM  dev253-0     11.40    233.28    334.94     49.83      0.01      1.16      0.22      0.25
    08:40:01 AM  dev253-1      8.49     62.32      5.63      8.00      0.02      2.13      0.07      0.06
    08:50:01 AM  dev252-0     10.26     70.09    337.74     39.74      0.01      0.94      0.27      0.28
    08:50:01 AM  dev253-0      9.42     31.42    325.61     37.89      0.01      0.98      0.25      0.24
    08:50:01 AM  dev253-1      6.35     38.67     12.13      8.00      0.01      1.39      0.08      0.05
    09:00:01 AM  dev252-0      9.18      3.81    328.34     36.18      0.01      1.02      0.30      0.28
    09:00:01 AM  dev253-0      8.99      2.81    328.34     36.85      0.01      1.06      0.31      0.28
    09:00:01 AM  dev253-1      0.13      1.00      0.00      8.00      0.00      0.68      0.09      0.00
    09:10:02 AM  dev252-0     11.57    210.57    356.26     49.00      0.01      1.10      0.35      0.41
    09:10:02 AM  dev253-0     11.32    209.14    335.07     48.07      0.01      1.04      0.31      0.36
    09:10:02 AM  dev253-1      2.83      1.43     21.19      8.00      0.08     29.33      0.20      0.06
    09:20:01 AM  dev252-0      9.21     19.13    335.23     38.50      0.01      1.09      0.30      0.28
    09:20:01 AM  dev253-0      9.02     18.70    329.72     38.62      0.01      1.13      0.31      0.28
    09:20:01 AM  dev253-1      0.74      0.43      5.51      8.00      0.00      5.74      0.05      0.00
    09:30:02 AM  dev252-0     11.11    113.42    325.10     39.48      0.01      1.22      0.28      0.31
    09:30:02 AM  dev253-0     10.52    100.73    325.10     40.46      0.01      1.29      0.29      0.31
    09:30:02 AM  dev253-1      1.59     12.69      0.00      8.00      0.00      0.62      0.12      0.02
    09:40:01 AM  dev252-0      7.39      5.41    247.88     34.26      0.01      1.07      0.31      0.23
    09:40:01 AM  dev253-0      7.24      4.83    246.45     34.71      0.01      1.10      0.30      0.22
    09:40:01 AM  dev253-1      0.25      0.58      1.43      8.00      0.01     53.57      0.53      0.01
    09:50:02 AM  dev252-0      9.69      5.50    346.63     36.33      0.01      1.21      0.36      0.34
    09:50:02 AM  dev253-0      9.53      5.18    343.15     36.56      0.01      1.20      0.32      0.31
    09:50:02 AM  dev253-1      0.47      0.32      3.47      8.00      0.03     63.39      0.88      0.04
    10:00:02 AM  dev252-0      9.39     20.13    324.59     36.72      0.01      1.19      0.29      0.27
    10:00:02 AM  dev253-0      9.12     13.65    324.59     37.09      0.01      1.23      0.30      0.27
    10:00:02 AM  dev253-1      0.81      6.48      0.00      8.00      0.00      0.44      0.06      0.01
    10:10:01 AM  dev252-0     13.53    309.25    322.27     46.67      0.02      1.25      0.32      0.43
    10:10:01 AM  dev253-0     13.34    308.34    317.79     46.92      0.02      1.25      0.29      0.39
    10:10:01 AM  dev253-1      0.67      0.91      4.49      8.00      0.00      4.88      0.59      0.04
    10:20:02 AM  dev252-0      9.76     20.04    352.26     38.16      0.01      1.27      0.36      0.35
    10:20:02 AM  dev253-0      9.61     19.88    325.23     35.93      0.01      1.14      0.31      0.29
    10:20:02 AM  dev253-1      3.40      0.16     27.03      8.00      0.16     45.86      0.18      0.06
    10:30:01 AM  dev252-0      9.12     43.14    291.33     36.68      0.01      1.18      0.29      0.27
    10:30:01 AM  dev253-0      8.97     41.37    291.33     37.08      0.01      1.22      0.30      0.27
    10:30:01 AM  dev253-1      0.22      1.78      0.00      8.00      0.00      0.55      0.08      0.00
    10:40:01 AM  dev252-0     11.45     79.54    372.85     39.50      0.01      1.10      0.30      0.35
    10:40:01 AM  dev253-0     11.07     67.40    359.68     38.57      0.01      1.08      0.28      0.31
    10:40:01 AM  dev253-1      3.16     12.14     13.16      8.00      0.07     23.53      0.14      0.04
    10:50:01 AM  dev252-0      7.92      6.46    287.52     37.11      0.01      0.80      0.16      0.13
    10:50:01 AM  dev253-0      7.78      5.96    287.52     37.72      0.01      0.83      0.17      0.13
    10:50:01 AM  dev253-1      0.06      0.50      0.00      8.00      0.00      0.46      0.08      0.00
    11:00:01 AM  dev252-0     10.37      9.63    404.71     39.95      0.01      0.85      0.17      0.17
    11:00:01 AM  dev253-0     10.20      9.02    404.71     40.58      0.01      0.88      0.17      0.18
    11:00:01 AM  dev253-1      0.08      0.61      0.00      8.00      0.00      0.73      0.20      0.00
    11:10:01 AM  dev252-0      8.47    122.05    299.60     49.77      0.01      0.91      0.19      0.16
    11:10:01 AM  dev253-0      8.37    121.98    299.60     50.36      0.01      0.94      0.19      0.16
    11:10:01 AM  dev253-1      0.01      0.07      0.00      8.00      0.00      0.60      0.20      0.00
    11:20:01 AM  dev252-0      8.09      0.70    296.62     36.76      0.01      0.76      0.16      0.13
    11:20:01 AM  dev253-0      7.95      0.60    296.62     37.39      0.01      0.80      0.17      0.13
    11:20:01 AM  dev253-1      0.01      0.11      0.00      8.00      0.00      1.00      0.12      0.00
    11:30:01 AM  dev252-0      7.87      0.20    298.17     37.93      0.01      0.75      0.16      0.13
    11:30:01 AM  dev253-0      7.71      0.00    298.17     38.67      0.01      0.79      0.17      0.13
    11:30:01 AM  dev253-1      0.03      0.20      0.00      8.00      0.00      0.53      0.07      0.00
    11:40:01 AM  dev252-0      7.91      0.20    299.06     37.82      0.01      0.86      0.17      0.14
    11:40:01 AM  dev253-0      7.78      0.09    299.06     38.47      0.01      0.89      0.18      0.14
    11:40:01 AM  dev253-1      0.01      0.11      0.00      8.00      0.00      0.38      0.12      0.00
    11:50:01 AM  dev252-0      9.73      1.34    373.22     38.49      0.01      0.82      0.17      0.16
    11:50:01 AM  dev253-0      9.61      1.27    373.22     38.97      0.01      0.86      0.17      0.16
    11:50:01 AM  dev253-1      0.01      0.07      0.00      8.00      0.00      0.00      0.00      0.00
    12:00:01 PM  dev252-0      9.38      9.96    348.15     38.18      0.01      0.80      0.16      0.15
    12:00:01 PM  dev253-0      9.18      7.72    348.15     38.76      0.01      0.83      0.17      0.15
    12:00:01 PM  dev253-1      0.28      2.24      0.00      8.00      0.00      0.61      0.09      0.00
    12:10:01 PM  dev252-0     11.24     68.67    377.66     39.71      0.01      0.77      0.18      0.20
    12:10:01 PM  dev253-0     11.16     68.10    377.66     39.96      0.01      0.81      0.18      0.21
    12:10:01 PM  dev253-1      0.07      0.57      0.00      8.00      0.00      0.38      0.05      0.00
    12:20:01 PM  dev252-0     10.17     67.09    348.83     40.91      0.01      0.79      0.18      0.18
    12:20:01 PM  dev253-0     10.01     63.91    348.83     41.23      0.01      0.83      0.18      0.18
    12:20:01 PM  dev253-1      0.40      3.19      0.00      8.00      0.00      0.63      0.08      0.00
    12:30:01 PM  dev252-0     11.32     73.12    387.48     40.70      0.01      0.98      0.17      0.19
    12:30:01 PM  dev253-0     11.14     72.04    383.56     40.89      0.01      1.01      0.17      0.19
    12:30:01 PM  dev253-1      0.63      1.09      3.92      8.00      0.00      3.98      0.05      0.00
    12:40:01 PM  dev252-0      9.54     27.83    359.99     40.64      0.01      0.82      0.19      0.18
    12:40:01 PM  dev253-0      9.24     22.79    349.00     40.24      0.01      0.82      0.18      0.16
    12:40:01 PM  dev253-1      2.00      5.04     10.98      8.00      0.03     16.11      0.10      0.02
    12:50:01 PM  dev252-0     13.21    168.62    459.35     47.55      0.01      1.04      0.23      0.30
    12:50:01 PM  dev253-0     12.65    147.03    436.96     46.15      0.01      1.04      0.19      0.25
    12:50:01 PM  dev253-1      5.50     21.58     22.40      8.00      0.05      9.44      0.10      0.06
    01:00:01 PM  dev252-0      9.47     20.41    330.61     37.05      0.01      0.81      0.17      0.16
    01:00:01 PM  dev253-0      9.30     19.75    329.37     37.54      0.01      0.84      0.17      0.16
    01:00:01 PM  dev253-1      0.24      0.67      1.24      8.00      0.00      1.63      0.11      0.00
    01:10:02 PM  dev252-0     11.36    128.90    373.16     44.20      0.01      0.99      0.22      0.25
    01:10:02 PM  dev253-0     11.15    127.88    334.19     41.45      0.01      0.96      0.19      0.21
    01:10:02 PM  dev253-1      5.00      1.01     38.96      8.00      0.06     11.88      0.10      0.05
    01:20:01 PM  dev252-0     11.28     12.59    425.62     38.84      0.01      0.97      0.19      0.21
    01:20:01 PM  dev253-0     11.07     10.27    425.62     39.36      0.01      1.00      0.19      0.21
    01:20:01 PM  dev253-1      0.29      2.32      0.00      8.00      0.00      0.61      0.10      0.00
    

    top output

    top - 13:47:33 up 6 days,  2:12,  1 user,  load average: 6.47, 5.67, 2.87
    Tasks: 111 total,   1 running, 110 sleeping,   0 stopped,   0 zombie
    %Cpu(s):  5.2 us,  3.7 sy,  0.0 ni, 90.8 id,  0.1 wa,  0.0 hi,  0.0 si,  0.2 st
    KiB Mem :  1016380 total,   243200 free,   527096 used,   246084 buff/cache
    KiB Swap:  2097148 total,  1688092 free,   409056 used.   253448 avail Mem
    
      PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
     9871 root      20   0  157568   2072   1500 R  6.7  0.2   0:00.01 top
        1 root      20   0   43380   2452   1428 S  0.0  0.2   3:22.61 systemd
        2 root      20   0       0      0      0 S  0.0  0.0   0:00.10 kthreadd
        3 root      20   0       0      0      0 S  0.0  0.0   0:04.54 ksoftirqd/0
        7 root      rt   0       0      0      0 S  0.0  0.0   0:00.00 migration/0
        8 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcu_bh
        9 root      20   0       0      0      0 S  0.0  0.0   1:10.13 rcu_sched
       10 root      rt   0       0      0      0 S  0.0  0.0   0:03.73 watchdog/0
       12 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kdevtmpfs
       13 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 netns
       14 root      20   0       0      0      0 S  0.0  0.0   0:02.51 khungtaskd
       15 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 writeback
       16 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kintegrityd
       17 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
       18 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kblockd
       19 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 md
       25 root      20   0       0      0      0 S  0.0  0.0   0:26.06 kswapd0
       26 root      25   5       0      0      0 S  0.0  0.0   0:00.00 ksmd
       27 root      39  19       0      0      0 S  0.0  0.0   0:09.18 khugepaged
       28 root      20   0       0      0      0 S  0.0  0.0   0:00.00 fsnotify_mark
       29 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 crypto
       37 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kthrotld
       39 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kmpath_rdacd
       40 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kpsmoused
       41 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ipv6_addrconf
       60 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 deferwq
       92 root      20   0       0      0      0 S  0.0  0.0   0:03.85 kauditd
      264 root       0 -20       0      0      0 S  0.0  0.0   0:00.05 ata_sff
      280 root      20   0       0      0      0 S  0.0  0.0   0:00.00 scsi_eh_0
      281 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 scsi_tmf_0
      282 root      20   0       0      0      0 S  0.0  0.0   0:00.00 scsi_eh_1
      283 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 scsi_tmf_1
      285 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ttm_swap
      359 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kdmflush
      360 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
      371 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kdmflush
      372 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 bioset
      385 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfsalloc
      386 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs_mru_cache
      387 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-buf/dm-0
      388 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-data/dm-0
      389 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-conv/dm-0
      390 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-cil/dm-0
      391 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-reclaim/dm-
      392 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-log/dm-0
      393 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 xfs-eofblocks/d
      394 root      20   0       0      0      0 S  0.0  0.0   2:56.36 xfsaild/dm-0
      469 root      20   0   36120   1340   1200 S  0.0  0.1   2:01.94 systemd-journal
      484 root      20   0  127084      0      0 S  0.0  0.0   0:00.02 lvmetad
      498 root      20   0   43632    292    288 S  0.0  0.0   0:00.07 systemd-udevd
      516 root      20   0       0      0      0 S  0.0  0.0   0:00.00 vballoon
      517 root      20   0       0      0      0 S  0.0  0.0   0:06.08 hwrng
      581 root      20   0       0      0      0 S  0.0  0.0   0:00.00 jbd2/vda1-8
      582 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 ext4-rsv-conver
      597 root      16  -4   55412    432    312 S  0.0  0.0   0:19.40 auditd
      619 avahi     20   0   27976    428    300 S  0.0  0.0   0:13.94 avahi-daemon
      620 dbus      20   0   24536    788    456 S  0.0  0.1   2:42.62 dbus-daemon
      623 avahi     20   0   27976     24      0 S  0.0  0.0   0:00.00 avahi-daemon
      627 root      20   0  105476    160     92 S  0.0  0.0   0:00.04 sshd
      629 root      20   0  553148    552    268 S  0.0  0.1   1:26.93 tuned
      630 root      20   0   24308    984    768 S  0.0  0.1   1:27.94 systemd-logind
      636 root      20   0    6964    180    140 S  0.0  0.0   0:00.20 mdadm
      637 root      20   0  262428   1908   1608 S  0.0  0.2   0:59.10 rsyslogd
      639 root      20   0  541036  12456   9172 S  0.0  1.2   0:38.44 httpd
      645 polkitd   20   0  527508    912    608 S  0.0  0.1   0:59.68 polkitd
      653 root      20   0   14988    340    232 S  0.0  0.0   0:00.04 incrond
      661 chrony    20   0  115844    728    624 S  0.0  0.1   0:01.74 chronyd
      665 nobody    20   0   15540     96     92 S  0.0  0.0   0:00.00 dnsmasq
      697 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 cfg80211
      708 root      20   0   25840      0      0 S  0.0  0.0   0:00.02 atd
      721 root      20   0  126248    808    400 S  0.0  0.1   0:34.60 crond
      733 root      20   0   27100    620    616 S  0.0  0.1   0:00.00 xinetd
      738 root      20   0  110032     64     60 S  0.0  0.0   0:00.00 agetty
      746 asterisk  20   0  749892  49024  27388 S  0.0  4.8   1:35.82 httpd
      752 mysql     20   0  113252     52     52 S  0.0  0.0   0:00.01 mysqld_safe
      822 asterisk  20   0  748824  47468  27728 S  0.0  4.7   4:24.32 httpd
      828 mongodb   20   0  483944   9532    588 S  0.0  0.9  28:42.09 mongod
      850 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 dio/dm-0
     1034 mysql     20   0 1177564 165648   2656 S  0.0 16.3  59:20.55 mysqld
     1148 root      20   0   89032    600    516 S  0.0  0.1   0:04.09 master
     1207 root      20   0  112876    468    236 S  0.0  0.0   0:00.06 dhclient
     3395 asterisk  20   0  650608  20176   9308 S  0.0  2.0   0:00.04 httpd
     3758 asterisk  20   0  657232  38972  20556 S  0.0  3.8   0:45.70 httpd
     3843 root       0 -20       0      0      0 S  0.0  0.0   0:00.74 kworker/0:2H
     5837 root      20   0  115240     44     40 S  0.0  0.0   0:00.00 safe_asterisk
     5844 asterisk  20   0 1645160  41756   5708 S  0.0  4.1  76:50.41 asterisk
     6477 root      20   0       0      0      0 S  0.0  0.0   0:00.05 kworker/0:0
     7038 asterisk  20   0  746004  43712  23384 S  0.0  4.3   0:35.92 httpd
    10255 root      20   0       0      0      0 S  0.0  0.0   0:00.63 kworker/0:1
    10422 root      20   0  148384   5936   4588 S  0.0  0.6   0:00.08 sshd
    11069 root       0 -20       0      0      0 S  0.0  0.0   0:03.32 kworker/0:1H
    14137 asterisk  20   0  747104  47268  28128 S  0.0  4.7   0:10.80 httpd
    14138 asterisk  20   0  650384  20712   9536 S  0.0  2.0   0:00.10 httpd
    15724 root      20   0  115380   2076   1676 S  0.0  0.2   0:00.02 bash
    16487 asterisk  20   0 1005508  19880   4076 S  0.0  2.0   7:23.89 PM2 v2.7.1: God
    16503 asterisk  20   0 1214940  18216   1884 S  0.0  1.8   7:22.57 letschat
    16833 asterisk  20   0  195328    668    248 S  0.0  0.1   0:00.23 pnp_server
    17498 asterisk  20   0 1087448  37676   7188 S  0.0  3.7   0:09.16 node /var/www/h
    17615 root      20   0       0      0      0 S  0.0  0.0   0:00.19 kworker/u2:1
    19180 asterisk  20   0  741236  28020  13168 S  0.0  2.8   1:13.27 httpd
    19600 asterisk  20   0  659456  45256  26600 S  0.0  4.5   0:27.39 httpd
    20008 asterisk  20   0  650888  20976   9400 S  0.0  2.1   0:01.55 httpd
    20137 asterisk  20   0  390124  16112   9436 S  0.0  1.6   0:00.19 php
    20168 asterisk  20   0  650860  20816   9260 S  0.0  2.0   0:01.34 httpd
    27522 root      20   0       0      0      0 S  0.0  0.0   0:00.52 kworker/0:2
    30030 asterisk  20   0  417620  26224  10636 S  0.0  2.6   0:23.46 php
    30330 root      20   0       0      0      0 S  0.0  0.0   0:00.04 kworker/u2:0
    30730 postfix   20   0   89136   3572   2572 S  0.0  0.4   0:00.03 pickup
    30793 postfix   20   0   89312   2804   2660 S  0.0  0.3   0:00.15 qmgr
    31123 root      20   0  398792  11796   4960 S  0.0  1.2  14:21.45 php
    31176 root      20   0  185668   1504   1144 S  0.0  0.1   0:09.04 fail2ban-server
    


  • Ah. Reading through the man pages for sar. From sar -q looks like there were other spikes. Looks like I have some data collection and cross-referencing in my future. 😃



  • One more little tidbit of information. I can replicate the spike by viewing the Connectivity > Firewall page(s).