System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7
-
@wirestyle22 said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
@gjacobse So it's a memory leak?
No reason to think that. Used memory and memory leak are very different things. Also, the box is normal once his process is done. A leak keeps using memory more and more over time.
-
-
-
12:00:01 PM all 13.72 0.00 0.56 0.06 1.59 84.06 12:10:01 PM all 19.92 0.00 0.60 0.09 1.77 77.62 12:20:01 PM all 11.98 0.00 0.48 0.10 1.02 86.42 12:30:02 PM all 17.21 0.00 0.45 0.10 1.17 81.08 12:40:01 PM all 12.69 0.00 0.43 0.04 1.13 85.71 12:50:01 PM all 17.98 0.00 0.47 0.08 1.41 80.05 01:00:01 PM all 13.49 0.00 0.46 0.06 1.17 84.82 01:10:01 PM all 19.12 0.00 0.55 0.10 1.50 78.73 01:20:01 PM all 11.94 0.00 0.48 0.05 0.98 86.55 01:30:02 PM all 19.56 0.00 0.56 0.09 1.48 78.31 01:40:01 PM all 24.13 0.00 0.74 0.09 1.21 73.83 01:50:01 PM all 26.44 0.00 0.65 0.08 1.34 71.48 02:00:01 PM all 17.44 0.00 0.54 0.04 1.08 80.90 02:10:01 PM all 28.65 0.00 0.80 0.08 1.80 68.67 02:20:01 PM all 11.86 0.00 0.45 0.03 0.95 86.71 02:30:01 PM all 17.95 0.00 0.52 0.07 1.27 80.19 02:40:01 PM all 11.69 0.00 0.44 0.04 0.94 86.90 02:50:02 PM all 27.31 0.00 0.67 0.06 1.48 70.49 03:00:01 PM all 15.61 0.00 0.49 0.03 1.16 82.70 03:10:01 PM all 17.35 0.00 0.49 0.07 1.21 80.88 03:20:01 PM all 12.84 0.00 0.54 0.03 1.05 85.54 03:30:01 PM all 26.24 0.00 0.82 0.07 1.83 71.03 03:40:02 PM all 13.70 0.00 0.60 0.03 1.41 84.25 03:50:01 PM all 20.65 0.00 0.62 0.07 1.49 77.18 04:00:01 PM all 11.31 0.00 0.49 0.03 1.10 87.06 04:10:01 PM all 20.25 0.00 0.64 0.07 1.44 77.61 04:20:01 PM all 10.64 0.00 0.45 0.03 0.83 88.05 04:30:01 PM all 20.96 0.00 0.60 0.07 1.37 77.00 04:40:01 PM all 5.77 0.00 0.53 0.03 1.02 92.65 04:50:02 PM all 8.36 0.00 0.44 0.06 0.91 90.23 05:00:01 PM all 1.78 0.00 0.34 0.03 0.50 97.36 05:10:01 PM all 7.63 0.00 0.47 0.06 0.84 91.00 Average: all 9.29 0.00 0.36 0.06 0.98 89.30 -
I don't think it shows in the last few screen shots,.. but found a PID what was running for 4hr 54m,.. Talking to @scottalanmiller came up on killing that process to see if it would release memory and it didn't.
But, in that, I found another similar process that had been running for more than 10 hours, so I killed that one.
Memory is now down to 325M.
-
Tag 1 is the pid that seemed to have been the lynch pin
-
So what is routinely kicking off that problem child, though?
-
Sorted by Time:
-
Starting to look for differences since the DB Maint should have run about an hour ago.
]$ cd /opt/screenconnect//App_Data/ [ App_Data]$ ls -l total 101304 -rw-r--r-- 1 root root 454 Aug 25 19:15 ExtensionConfiguration.xml drwxr-xr-x 2 root root 4096 Nov 8 09:44 Helper -rw-r--r-- 1 root root 654 Aug 25 19:15 License.xml -rw-r--r-- 1 root root 23699 Dec 16 11:26 Role.xml -rw-r--r-- 1 root root 25210880 Dec 17 01:59 Session.db -rw-r--r-- 1 root root 32768 Dec 17 02:04 Session.db-shm -rw-r--r-- 1 root root 78403632 Dec 17 02:04 Session.db-wal -rw-r--r-- 1 root root 4030 Oct 25 08:55 SessionEventTrigger.xml -rw-r--r-- 1 root root 5178 Dec 2 08:13 SessionGroup.xml drwxr-xr-x 14 root root 4096 Dec 12 12:55 Toolbox -rw-r--r-- 1 root root 0 Dec 14 17:24 User.cd60043d-556e-4b23-9ef1-9959e6cea952.xml -rw-r--r-- 1 root root 21908 Dec 16 17:24 User.xml -
]$ sar Linux 3.10.0-327.10.1.el7.x86_64 (dny-lnx-sc) 12/17/2016 _x86_64_ (2 CPU) 12:00:02 AM CPU %user %nice %system %iowait %steal %idle 12:10:01 AM all 1.05 0.00 0.24 0.06 0.55 98.10 12:20:02 AM all 1.16 0.00 0.22 0.04 0.49 98.09 12:30:01 AM all 0.84 0.00 0.20 0.04 0.40 98.53 12:40:01 AM all 1.05 0.00 0.19 0.05 0.40 98.31 12:50:01 AM all 0.83 0.00 0.19 0.04 0.42 98.52 01:00:01 AM all 1.06 0.00 0.19 0.04 0.37 98.34 01:10:01 AM all 1.03 0.00 0.22 0.04 0.46 98.25 01:20:02 AM all 1.16 0.00 0.20 0.06 0.47 98.12 01:30:01 AM all 0.96 0.00 0.20 0.04 0.52 98.28 01:40:01 AM all 0.97 0.00 0.19 0.05 0.44 98.35 01:50:01 AM all 0.86 0.00 0.16 0.04 0.44 98.51 02:00:01 AM all 1.00 0.00 0.18 0.05 0.43 98.34 Average: all 1.00 0.00 0.20 0.04 0.45 98.31 -
-
Maybe reboot post maintenance so that we know that we have a fresh, clean system and see what it does.
-
What I see right off compairing:
https://mangolassi.it/topic/11869/system-lockups-with-screenconnect-6-0-11622-6115-on-centos-7/33
https://mangolassi.it/topic/11869/system-lockups-with-screenconnect-6-0-11622-6115-on-centos-7/56
is that Session.db is larger, but Session.db-shm is smaller.
-
@scottalanmiller said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
Maybe reboot post maintenance so that we know that we have a fresh, clean system and see what it does.
Done... awaiting system to come back online.
-
Session-db.wal went from 78,403,632 to 4,128,272 !!!
App_Data]$ ls -l total 28836 -rw-r--r-- 1 root root 454 Aug 25 19:15 ExtensionConfiguration.xml drwxr-xr-x 2 root root 4096 Nov 8 09:44 Helper -rw-r--r-- 1 root root 654 Aug 25 19:15 License.xml -rw-r--r-- 1 root root 23699 Dec 16 11:26 Role.xml -rw-r--r-- 1 root root 25284608 Dec 17 02:17 Session.db -rw-r--r-- 1 root root 32768 Dec 17 02:17 Session.db-shm -rw-r--r-- 1 root root 4128272 Dec 17 02:17 Session.db-wal -rw-r--r-- 1 root root 4030 Oct 25 08:55 SessionEventTrigger.xml -rw-r--r-- 1 root root 5178 Dec 2 08:13 SessionGroup.xml drwxr-xr-x 14 root root 4096 Dec 12 12:55 Toolbox -rw-r--r-- 1 root root 0 Dec 14 17:24 User.cd60043d-556e-4b23-9ef1-9959e 6cea952.xml -rw-r--r-- 1 root root 21908 Dec 16 17:24 User.xml -
That's quite a change.
-
No heavy CPU or memory on restart either.
-
Of note - the only 'difference' was the selection of Maintenance Plan Actions:
Previously only the Delete Ended Support Sessions and Compact Database Files were selected. The boxed items were not.
-
-rw-r--r-- 1 root root 244,855,752 Dec 14 16:25 Session.db-wal-rw-r--r-- 1 root root 78,403,632 Dec 16 08:33 Session.db-walrw-r--r-- 1 root root 4,128,272 Dec 17 02:17 Session.db-walGoing back and doing some comparisons and noticed this -
(commas added for 'dramatic effect'
-
SAR
r this morning - near as I can tell, no one is on SC currently.$ uptime 10:36:37 up 22:38, 1 user, load average: 0.05, 0.17, 0.22 [ ]$ sar Linux 3.10.0-327.10.1.el7.x86_64 ( ) 12/18/2016 _x86_64_ (1 CPU) 12:00:01 AM CPU %user %nice %system %iowait %steal %idle 12:10:01 AM all 1.89 0.00 0.36 0.09 0.06 97.60 12:20:01 AM all 1.03 0.00 0.33 0.03 0.03 98.57 12:30:01 AM all 1.83 0.00 0.36 0.07 0.05 97.68 12:40:01 AM all 0.89 0.00 0.29 0.03 0.05 98.73 12:50:01 AM all 1.68 0.00 0.35 0.08 0.04 97.85 01:00:01 AM all 1.03 0.00 0.34 0.07 0.03 98.54 01:10:01 AM all 2.30 0.00 0.45 0.13 0.06 97.05 01:20:01 AM all 0.79 0.00 0.29 0.05 0.03 98.84 01:30:01 AM all 1.82 0.00 0.36 0.09 0.08 97.65 01:40:01 AM all 0.71 0.00 0.28 0.02 0.03 98.97 01:50:01 AM all 1.78 0.00 0.35 0.11 0.07 97.70 02:00:01 AM all 0.74 0.00 0.29 0.06 0.03 98.88 02:10:02 AM all 1.76 0.00 0.36 0.09 0.07 97.72 02:20:01 AM all 1.16 0.00 0.33 0.04 0.04 98.43 02:30:01 AM all 1.90 0.00 0.35 0.08 0.11 97.56 02:40:01 AM all 0.76 0.00 0.28 0.02 0.05 98.88 02:50:01 AM all 1.80 0.00 0.36 0.24 0.08 97.52 03:00:01 AM all 0.69 0.00 0.28 0.03 0.05 98.95 03:10:01 AM all 1.86 0.02 0.36 0.08 0.07 97.61 03:20:01 AM all 0.82 0.00 0.32 0.02 0.04 98.79 03:30:01 AM all 1.84 0.00 0.34 0.09 0.05 97.67 03:40:01 AM all 0.74 0.00 0.28 0.03 0.04 98.91 03:50:01 AM all 1.81 0.00 0.35 0.09 0.06 97.70 04:00:01 AM all 0.82 0.00 0.29 0.03 0.04 98.82 04:10:01 AM all 2.42 0.00 0.44 0.09 0.10 96.96 04:20:01 AM all 0.77 0.00 0.29 0.03 0.05 98.86 04:30:01 AM all 1.84 0.00 0.37 0.09 0.10 97.61 04:40:01 AM all 0.75 0.00 0.30 0.06 0.06 98.83 04:50:01 AM all 1.84 0.00 0.36 0.09 0.08 97.63 05:00:01 AM all 0.81 0.00 0.29 0.02 0.05 98.83 05:10:01 AM all 2.17 0.00 0.43 0.09 0.11 97.21 05:20:01 AM all 0.85 0.00 0.29 0.16 0.04 98.67 05:30:01 AM all 1.79 0.00 0.38 0.09 0.07 97.67 05:40:01 AM all 0.86 0.00 0.32 0.03 0.08 98.72 05:50:01 AM all 1.94 0.00 0.46 0.16 0.12 97.32 06:00:01 AM all 0.80 0.00 0.33 0.02 0.05 98.80 06:10:01 AM all 1.79 0.00 0.37 0.33 0.08 97.42 06:20:01 AM all 0.93 0.00 0.34 0.02 0.09 98.62 06:30:01 AM all 1.82 0.00 0.36 0.08 0.09 97.65 06:40:02 AM all 0.76 0.00 0.29 0.03 0.06 98.87 06:50:01 AM all 1.87 0.00 0.37 0.09 0.11 97.56 07:00:01 AM all 0.82 0.00 0.30 0.03 0.05 98.79 07:10:02 AM all 2.23 0.00 0.46 0.08 0.15 97.09 07:20:01 AM all 0.82 0.00 0.32 0.03 0.09 98.74 07:30:01 AM all 2.84 0.00 0.75 0.09 0.12 96.20 07:40:01 AM all 3.05 0.00 0.30 0.03 0.09 96.54 07:50:01 AM all 24.73 0.32 0.62 0.12 0.15 74.07 08:00:01 AM all 6.52 0.00 0.34 0.04 0.07 93.03 08:00:01 AM CPU %user %nice %system %iowait %steal %idle 08:10:01 AM all 25.18 0.00 0.58 0.07 0.16 74.01 08:20:01 AM all 7.17 0.00 0.34 0.03 0.10 92.37 08:30:01 AM all 25.70 0.00 0.52 0.12 0.23 73.42 08:40:01 AM all 6.75 0.00 0.34 0.03 0.13 92.75 08:50:01 AM all 24.80 0.00 0.50 0.07 0.17 74.45 09:00:01 AM all 7.47 0.00 0.34 0.02 0.08 92.08 09:10:01 AM all 28.54 0.00 0.63 0.09 0.48 70.27 09:20:01 AM all 7.15 0.00 0.36 0.17 0.17 92.16 09:30:01 AM all 26.84 0.00 0.54 0.09 0.34 72.20 09:40:01 AM all 8.76 0.00 0.36 0.07 0.27 90.54 09:50:01 AM all 26.93 0.00 0.53 0.07 0.31 72.15 10:00:01 AM all 8.60 0.00 0.35 0.03 0.18 90.84 10:10:01 AM all 28.29 0.00 0.63 0.08 0.47 70.53 10:20:01 AM all 9.19 0.00 0.33 0.08 0.28 90.12 10:30:01 AM all 27.27 0.00 0.52 0.08 0.32 71.81 Average: all 5.79 0.01 0.38 0.07 0.11 93.64
