System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7
-
Of note - the only 'difference' was the selection of Maintenance Plan Actions:
Previously only the Delete Ended Support Sessions and Compact Database Files were selected. The boxed items were not.
-
-rw-r--r-- 1 root root 244,855,752 Dec 14 16:25 Session.db-wal-rw-r--r-- 1 root root 78,403,632 Dec 16 08:33 Session.db-walrw-r--r-- 1 root root 4,128,272 Dec 17 02:17 Session.db-walGoing back and doing some comparisons and noticed this -
(commas added for 'dramatic effect'
-
SAR
r this morning - near as I can tell, no one is on SC currently.$ uptime 10:36:37 up 22:38, 1 user, load average: 0.05, 0.17, 0.22 [ ]$ sar Linux 3.10.0-327.10.1.el7.x86_64 ( ) 12/18/2016 _x86_64_ (1 CPU) 12:00:01 AM CPU %user %nice %system %iowait %steal %idle 12:10:01 AM all 1.89 0.00 0.36 0.09 0.06 97.60 12:20:01 AM all 1.03 0.00 0.33 0.03 0.03 98.57 12:30:01 AM all 1.83 0.00 0.36 0.07 0.05 97.68 12:40:01 AM all 0.89 0.00 0.29 0.03 0.05 98.73 12:50:01 AM all 1.68 0.00 0.35 0.08 0.04 97.85 01:00:01 AM all 1.03 0.00 0.34 0.07 0.03 98.54 01:10:01 AM all 2.30 0.00 0.45 0.13 0.06 97.05 01:20:01 AM all 0.79 0.00 0.29 0.05 0.03 98.84 01:30:01 AM all 1.82 0.00 0.36 0.09 0.08 97.65 01:40:01 AM all 0.71 0.00 0.28 0.02 0.03 98.97 01:50:01 AM all 1.78 0.00 0.35 0.11 0.07 97.70 02:00:01 AM all 0.74 0.00 0.29 0.06 0.03 98.88 02:10:02 AM all 1.76 0.00 0.36 0.09 0.07 97.72 02:20:01 AM all 1.16 0.00 0.33 0.04 0.04 98.43 02:30:01 AM all 1.90 0.00 0.35 0.08 0.11 97.56 02:40:01 AM all 0.76 0.00 0.28 0.02 0.05 98.88 02:50:01 AM all 1.80 0.00 0.36 0.24 0.08 97.52 03:00:01 AM all 0.69 0.00 0.28 0.03 0.05 98.95 03:10:01 AM all 1.86 0.02 0.36 0.08 0.07 97.61 03:20:01 AM all 0.82 0.00 0.32 0.02 0.04 98.79 03:30:01 AM all 1.84 0.00 0.34 0.09 0.05 97.67 03:40:01 AM all 0.74 0.00 0.28 0.03 0.04 98.91 03:50:01 AM all 1.81 0.00 0.35 0.09 0.06 97.70 04:00:01 AM all 0.82 0.00 0.29 0.03 0.04 98.82 04:10:01 AM all 2.42 0.00 0.44 0.09 0.10 96.96 04:20:01 AM all 0.77 0.00 0.29 0.03 0.05 98.86 04:30:01 AM all 1.84 0.00 0.37 0.09 0.10 97.61 04:40:01 AM all 0.75 0.00 0.30 0.06 0.06 98.83 04:50:01 AM all 1.84 0.00 0.36 0.09 0.08 97.63 05:00:01 AM all 0.81 0.00 0.29 0.02 0.05 98.83 05:10:01 AM all 2.17 0.00 0.43 0.09 0.11 97.21 05:20:01 AM all 0.85 0.00 0.29 0.16 0.04 98.67 05:30:01 AM all 1.79 0.00 0.38 0.09 0.07 97.67 05:40:01 AM all 0.86 0.00 0.32 0.03 0.08 98.72 05:50:01 AM all 1.94 0.00 0.46 0.16 0.12 97.32 06:00:01 AM all 0.80 0.00 0.33 0.02 0.05 98.80 06:10:01 AM all 1.79 0.00 0.37 0.33 0.08 97.42 06:20:01 AM all 0.93 0.00 0.34 0.02 0.09 98.62 06:30:01 AM all 1.82 0.00 0.36 0.08 0.09 97.65 06:40:02 AM all 0.76 0.00 0.29 0.03 0.06 98.87 06:50:01 AM all 1.87 0.00 0.37 0.09 0.11 97.56 07:00:01 AM all 0.82 0.00 0.30 0.03 0.05 98.79 07:10:02 AM all 2.23 0.00 0.46 0.08 0.15 97.09 07:20:01 AM all 0.82 0.00 0.32 0.03 0.09 98.74 07:30:01 AM all 2.84 0.00 0.75 0.09 0.12 96.20 07:40:01 AM all 3.05 0.00 0.30 0.03 0.09 96.54 07:50:01 AM all 24.73 0.32 0.62 0.12 0.15 74.07 08:00:01 AM all 6.52 0.00 0.34 0.04 0.07 93.03 08:00:01 AM CPU %user %nice %system %iowait %steal %idle 08:10:01 AM all 25.18 0.00 0.58 0.07 0.16 74.01 08:20:01 AM all 7.17 0.00 0.34 0.03 0.10 92.37 08:30:01 AM all 25.70 0.00 0.52 0.12 0.23 73.42 08:40:01 AM all 6.75 0.00 0.34 0.03 0.13 92.75 08:50:01 AM all 24.80 0.00 0.50 0.07 0.17 74.45 09:00:01 AM all 7.47 0.00 0.34 0.02 0.08 92.08 09:10:01 AM all 28.54 0.00 0.63 0.09 0.48 70.27 09:20:01 AM all 7.15 0.00 0.36 0.17 0.17 92.16 09:30:01 AM all 26.84 0.00 0.54 0.09 0.34 72.20 09:40:01 AM all 8.76 0.00 0.36 0.07 0.27 90.54 09:50:01 AM all 26.93 0.00 0.53 0.07 0.31 72.15 10:00:01 AM all 8.60 0.00 0.35 0.03 0.18 90.84 10:10:01 AM all 28.29 0.00 0.63 0.08 0.47 70.53 10:20:01 AM all 9.19 0.00 0.33 0.08 0.28 90.12 10:30:01 AM all 27.27 0.00 0.52 0.08 0.32 71.81 Average: all 5.79 0.01 0.38 0.07 0.11 93.64 -
-
~]$ cd /opt/screenconnect//App_Data/ [gene@dny-lnx-sc App_Data]$ ls -l total 49048 -rw-r--r-- 1 root root 454 Aug 25 19:15 ExtensionConfiguration.xml drwxr-xr-x 2 root root 4096 Nov 8 09:44 Helper -rw-r--r-- 1 root root 654 Aug 25 19:15 License.xml -rw-r--r-- 1 root root 23699 Dec 16 11:26 Role.xml -rw-r--r-- 1 root root 24989696 Dec 18 10:26 Session.db -rw-r--r-- 1 root root 65536 Dec 18 10:39 Session.db-shm -rw-r--r-- 1 root root 25082592 Dec 18 10:39 Session.db-wal -rw-r--r-- 1 root root 4030 Oct 25 08:55 SessionEventTrigger.xml -rw-r--r-- 1 root root 5178 Dec 2 08:13 SessionGroup.xml drwxr-xr-x 14 root root 4096 Dec 12 12:55 Toolbox -rw-r--r-- 1 root root 0 Dec 14 17:24 User.cd60043d-556e-4b23-9ef1-9959e6cea952.xml -rw-r--r-- 1 root root 21908 Dec 17 10:41 User.xml -
@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
-rw-r--r-- 1 root root 244,855,752 Dec 14 16:25 Session.db-wal-rw-r--r-- 1 root root 78,403,632 Dec 16 08:33 Session.db-walrw-r--r-- 1 root root 4,128,272 Dec 17 02:17 Session.db-walGoing back and doing some comparisons and noticed this -
(commas added for 'dramatic effect'
-rw-r--r-- 1 root root 25,082,592 Dec 18 10:39 Session.db-wal -
So it is climbing back up and fast. Show that to support.
-
Memory is up to 398M used now.
SAR is bouncing around as well -
08:00:01 AM CPU %user %nice %system %iowait %steal %idle 08:10:01 AM all 25.18 0.00 0.58 0.07 0.16 74.01 08:20:01 AM all 7.17 0.00 0.34 0.03 0.10 92.37 08:30:01 AM all 25.70 0.00 0.52 0.12 0.23 73.42 08:40:01 AM all 6.75 0.00 0.34 0.03 0.13 92.75 08:50:01 AM all 24.80 0.00 0.50 0.07 0.17 74.45 09:00:01 AM all 7.47 0.00 0.34 0.02 0.08 92.08 09:10:01 AM all 28.54 0.00 0.63 0.09 0.48 70.27 09:20:01 AM all 7.15 0.00 0.36 0.17 0.17 92.16 09:30:01 AM all 26.84 0.00 0.54 0.09 0.34 72.20 09:40:01 AM all 8.76 0.00 0.36 0.07 0.27 90.54 09:50:01 AM all 26.93 0.00 0.53 0.07 0.31 72.15 10:00:01 AM all 8.60 0.00 0.35 0.03 0.18 90.84 10:10:01 AM all 28.29 0.00 0.63 0.08 0.47 70.53 10:20:01 AM all 9.19 0.00 0.33 0.08 0.28 90.12 10:30:01 AM all 27.27 0.00 0.52 0.08 0.32 71.81 10:40:01 AM all 8.15 0.00 0.33 0.07 0.23 91.22 10:50:01 AM all 25.58 0.00 0.46 0.07 0.27 73.61 11:00:01 AM all 41.85 0.00 0.66 0.14 0.29 57.06 11:10:01 AM all 26.54 0.00 0.62 0.10 0.47 72.27 11:20:01 AM all 8.44 0.00 0.39 0.03 0.21 90.92 11:30:01 AM all 43.08 0.00 0.68 0.15 0.58 55.50 11:40:01 AM all 27.83 0.00 0.52 0.11 0.47 71.07 11:50:01 AM all 24.17 0.00 0.53 0.07 0.38 74.84 12:00:01 PM all 8.93 0.00 0.42 0.04 0.31 90.30 12:10:01 PM all 26.29 0.00 0.60 0.10 0.40 72.61 12:20:01 PM all 9.36 0.00 0.61 0.05 0.17 89.80 12:30:01 PM all 23.48 0.00 0.66 0.07 0.19 75.60 12:40:01 PM all 9.61 0.00 0.50 0.06 0.21 89.63 12:50:01 PM all 23.44 0.00 0.62 0.06 0.18 75.70 01:00:01 PM all 8.63 0.00 0.48 0.03 0.10 90.76 01:10:01 PM all 24.10 0.00 0.70 0.07 0.21 74.93 Average: all 8.90 0.00 0.41 0.07 0.15 90.46 -
Yup that CPU is WAY too high.
-
Curious - system did a restart this evening.
05:05:15 PM LINUX RESTART 05:10:01 PM CPU %user %nice %system %iowait %steal %idle 05:20:01 PM all 3.56 0.00 0.15 0.02 0.09 96.19 05:30:01 PM all 14.36 0.00 0.23 0.09 0.07 85.25 05:40:01 PM all 4.74 0.00 0.16 0.03 0.09 94.99 05:50:01 PM all 16.36 0.00 0.26 0.14 0.11 83.13 06:00:01 PM all 3.68 0.00 0.13 0.02 0.06 96.12 06:10:01 PM all 16.09 0.00 0.27 0.10 0.13 83.41 06:20:01 PM all 4.60 0.00 0.24 0.13 0.09 94.96 06:30:01 PM all 15.20 0.00 0.24 0.32 0.11 84.14 06:40:01 PM all 3.54 0.00 0.13 0.01 0.05 96.27 06:50:01 PM all 15.19 0.00 0.24 0.09 0.07 84.40 07:00:01 PM all 5.10 0.00 0.14 0.02 0.07 94.68 07:10:01 PM all 15.11 0.00 0.29 0.13 0.11 84.36 07:20:01 PM all 3.91 0.00 0.12 0.03 0.06 95.89 07:30:01 PM all 14.58 0.00 0.24 0.08 0.08 85.02 07:40:01 PM all 4.20 0.00 0.18 0.02 0.05 95.55 07:50:01 PM all 14.34 0.00 0.26 0.07 0.08 85.25 Average: all 9.65 0.00 0.20 0.08 0.08 89.98DB is still on the upside for 25MB
]$ ls -l total 49596 -rw-r--r-- 1 root root 454 Aug 25 19:15 ExtensionConfiguration.xml drwxr-xr-x 2 root root 4096 Nov 8 09:44 Helper -rw-r--r-- 1 root root 654 Aug 25 19:15 License.xml -rw-r--r-- 1 root root 23699 Dec 16 11:26 Role.xml -rw-r--r-- 1 root root 25583616 Dec 18 19:48 Session.db -rw-r--r-- 1 root root 32768 Dec 18 19:50 Session.db-shm -rw-r--r-- 1 root root 25082592 Dec 18 19:50 Session.db-wal -rw-r--r-- 1 root root 4030 Oct 25 08:55 SessionEventTrigger.xml -rw-r--r-- 1 root root 5178 Dec 2 08:13 SessionGroup.xml drwxr-xr-x 14 root root 4096 Dec 12 12:55 Toolbox -rw-r--r-- 1 root root 0 Dec 14 17:24 User.cd60043d-556e-4b23-9ef1-9959e6cea952.xml -rw-r--r-- 1 root root 21908 Dec 17 10:41 User.xml -
After speaking with support this morning. Plan is to stop the SC service, rename the DB and start the service to allow the system to rebuild it.
-
@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
After speaking with support this morning. Plan is to stop the SC service, rename the DB and start the service to allow the system to rebuild it.
what data will you lose?
-
@Dashrender said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
After speaking with support this morning. Plan is to stop the SC service, rename the DB and start the service to allow the system to rebuild it.
what data will you lose?
We should not lose any data. Just some down time while the process is completeed.
-
Process done:
sudo service screenconnect stopmv /opt/screenconnect/Session.db /opt/screenconnect/Session.db.baksudo service screenconnect start -
]$ ls -l total 29724 -rw-r--r-- 1 root root 454 Aug 25 19:15 ExtensionConfiguration.xml drwxr-xr-x 2 root root 4096 Nov 8 09:44 Helper -rw-r--r-- 1 root root 654 Aug 25 19:15 License.xml -rw-r--r-- 1 root root 23699 Dec 16 11:26 Role.xml -rw-r--r-- 1 root root 1232896 Dec 19 20:14 Session.db -rw-r--r-- 1 root root 24956928 Dec 19 20:11 Session.db.bak -rw-r--r-- 1 root root 32768 Dec 19 20:18 Session.db-shm -rw-r--r-- 1 root root 4128272 Dec 19 20:18 Session.db-wal -rw-r--r-- 1 root root 4030 Oct 25 08:55 SessionEventTrigger.xml -rw-r--r-- 1 root root 5178 Dec 2 08:13 SessionGroup.xml drwxr-xr-x 14 root root 4096 Dec 12 12:55 Toolbox -rw-r--r-- 1 root root 0 Dec 14 17:24 User.cd60043d-556e-4b23-9ef1-9959e6cea952.xml -rw-r--r-- 1 root root 21908 Dec 19 20:13 User.xml -
@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
-rw-r--r-- 1 root root 244,855,752 Dec 14 16:25 Session.db-wal-rw-r--r-- 1 root root 78,403,632 Dec 16 08:33 Session.db-walrw-r--r-- 1 root root 4,128,272 Dec 17 02:17 Session.db-walGoing back and doing some comparisons and noticed this -
(commas added for 'dramatic effect'
-rw-r--r-- 1 root root 25,082,592 Dec 18 10:39 Session.db-walSession.db-wal is not the right file to be looking at... Should be looking at session.db
-
@Dashrender said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
After speaking with support this morning. Plan is to stop the SC service, rename the DB and start the service to allow the system to rebuild it.
what data will you lose?
The only thing we seem to have lost is Support Sessions, as they are temporary anyway.
-
@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
@Dashrender said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
@gjacobse said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
After speaking with support this morning. Plan is to stop the SC service, rename the DB and start the service to allow the system to rebuild it.
what data will you lose?
The only thing we seem to have lost is Support Sessions, as they are temporary anyway.
Taht would be expected.
-
So how are we looking after a few hours?
-
@scottalanmiller said in System Lockups with ScreenConnect 6.0.11622.6115 on CentOS 7:
So how are we looking after a few hours?
We are at 320 of the 540 devices previously logg d.
I expect this as some devices are off,...
