ScreenConnect on CentOS is sluggish



  • I know @gjacobse has been dealing with this for NTG also. But ScreenConnect on CentOS 7 is just slow and shitty.

    I have been using it for a week now and while it works well, it works like shit compared to how it ran on Windows.

              bnasc.ad.bundystl.com (CentOS Linux 7.3.1611 64bit / Linux 3.10.0-514.6.1.el7.x8              Uptime: 5 days, 16:55:35
    
    CPU       4.5%  steal:    0.0%   Load   2-core   Mem    95.5%  active:   1.09G   Swap   56.6%
    user:     1.1%  nice:     0.0%   1 min:   0.28   total: 1.70G  inactive:  413M   total: 1.75G
    system:   0.6%  iowait:   2.7%   5 min:   0.44   used:  1.63G  buffers:    12K   used:  1015M
    idle:    95.5%  irq:      0.0%   15 min:  0.24   free:  78.8M  cached:   10.5M   free:   777M
    
    Network    Rx/s    Tx/s   Tasks   96 (146 thr),  1 run,  95 slp,  0 oth  sorted automatically
    eth0        8Kb     4Kb
    lo          3Kb     3Kb    VIRT   RES  CPU%  MEM%   PID USER        NI S    TIME+ IOR/s IOW/s NAME
                               4.0G  1.4G   0.3  80.5 15855 root         0 S  0:54.34   73K   17K mono
    Disk I/O   In/s   Out/s    230M    5M   3.2   0.3 67135 root         0 R  0:07.75     0     0 /usr/bin/python /usr/bin/glances
    dm-0          0       0    427M    2M   0.0   0.1   659 root         0 S  0:00.96     0     0 NetworkManager
    dm-1          0     37K    320M    2M   0.0   0.1   655 root         0 S  0:07.97     0     0 firewalld
    sda1          0       0     36M    2M   0.0   0.1   476 root         0 S  0:01.57     0     0 /usr/lib/systemd/systemd-journald
    sda2          0       0    125M    1M   0.0   0.1     1 root         0 S  0:05.19     0     0 systemd
    sda3          0     40K     32M  772K   0.0   0.0   630 dbus         0 S  0:01.43     0     0 dbus-daemon
    sr0           0       0    516M  572K   0.0   0.0   623 polkitd      0 S  0:00.30     0     0 polkitd
                               110M  508K   0.0   0.0 20673 root         0 S  0:00.13     0     0 dhclient
    Mount      Used   Total    540M  496K   0.0   0.0   961 root         0 S  0:51.82     0     0 tuned
    /         1.55G   47.0G     24M  356K   0.0   0.0   626 root         0 S  0:01.52     0     0 /usr/lib/systemd/systemd-logind
    /boot      173M   1014M     19M  300K   0.0   0.0   629 root         0 S  0:32.14     0     0 /usr/sbin/irqbalance --foreground
    _oot/efi  9.45M    200M    279M  288K   0.0   0.0   967 root         0 S  0:00.34     0     0 /usr/sbin/rsyslogd -n
    /run      8.27M    872M    140M  192K   0.0   0.0 67114 root         0 S  0:00.30     0     0 sshd: [email protected]/0
    _/user/0      0    174M    123M  164K   0.0   0.0   644 root         0 S  0:01.60     0     0 /usr/sbin/crond -n
    _efivars      0       0    113M   80K   0.0   0.0   632 chrony       0 S  0:00.71     0     0 /usr/sbin/chronyd
    _selinux      0       0     81M   72K   0.0   0.0  1030 root         0 S  0:00.10     0     0 /usr/sbin/sshd
                                87M   72K   0.0   0.0  1922 root         0 S  0:02.61     0     0 /usr/libexec/postfix/master -w
                                54M   60K   0.0   0.0   605 root        -4 S  0:00.30     0     0 /sbin/auditd -n
                                87M   56K   0.0   0.0 67070 postfix      0 S  0:00.00     0     0 pickup -l -t unix -u
                                45M   16K   0.0   0.0   506 root         0 S  0:00.27     0     0 /usr/lib/systemd/systemd-udevd
                                90M   12K   0.0   0.0   649 root         0 S  0:00.22     0     0 login -- root
                               113M   12K   0.0   0.0 20335 root         0 S  0:00.20     0     0 -bash
                               110M    4K   0.0   0.0 15853 root         0 S  0:00.00     0     0 screenconnect
                               113M    4K   0.0   0.0 67118 root         0 S  0:00.20     0     0 -bash
                                  0     0   0.0   0.0     2 root         0 S  0:00.60     0     0 kthreadd
                                  0     0   0.0   0.0     3 root         0 S  0:01.98     0     0 ksoftirqd/0
                                  0     0   0.0   0.0     7 root         0 S  0:00.80     0     0 migration/0
                                  0     0   0.0   0.0     8 root         0 S  0:00.00     0     0 rcu_bh
                                  0     0   0.0   0.0     9 root         0 S  0:45.74     0     0 rcu_sched
                                  0     0   0.0   0.0    10 root         0 S  0:02.64     0     0 watchdog/0
                                  0     0   0.0   0.0    11 root         0 S  0:02.67     0     0 watchdog/1
    
    WARNING|CRITICAL logs (lasts 4 entries)
      2017-02-06 10:45:43 > 2017-02-06 10:45:46 CPU IOwait (63.5/63.5/63.5)
      2017-02-06 10:45:11 > 2017-02-06 10:45:14 CPU IOwait (60.1/60.1/60.1)
      2017-02-06 10:44:28 > 2017-02-06 10:44:31 CPU IOwait (65.7/65.7/65.7)
    ~ 2017-02-06 10:43:50 > ___________________ MEM real (1.59G/1.62G/1.64G) - Top process: mono
    


  • Starting a support chat to get things reported, we shall see where this goes.



  • What command is that?

    I'm running SC on Centos7 and not noticed any issues. Apart from a few when the other end is on a crap connection speed.



  • @hobbit666 said in ScreenConnect on CentOS is sluggish:

    What command is that?

    glances

    I'm running SC on Centos7 and not noticed any issues. Apart from a few when the other end is on a crap connection speed.

    IT certainly works fairly well. But compared to how it worked on a Windows instance, it is horrible.



  • WARNING|CRITICAL logs (lasts 6 entries)
      2017-02-06 10:52:17 > 2017-02-06 10:52:24 CPU IOwait (62.7/66.0/69.3)
      2017-02-06 10:47:53 > 2017-02-06 10:48:00 CPU IOwait (60.9/64.1/67.3)
      2017-02-06 10:45:43 > 2017-02-06 10:45:46 CPU IOwait (63.5/63.5/63.5)
      2017-02-06 10:45:11 > 2017-02-06 10:45:14 CPU IOwait (60.1/60.1/60.1)
      2017-02-06 10:44:28 > 2017-02-06 10:44:31 CPU IOwait (65.7/65.7/65.7)
    ~ 2017-02-06 10:43:50 > ___________________ MEM real (1.58G/1.61G/1.64G) - Top process: mono
    

    IOwait happening oftenish



  • @JaredBusch said in ScreenConnect on CentOS is sluggish:

    Starting a support chat to get things reported, we shall see where this goes.

    /sigh

    0_1486400256430_upload-4a63ad41-e237-42c5-b500-dcb9619370ac



  • deleted: mistake



  • How many clients are you running? We currently have 546 running without to much issue.

    Obivous questions:

    • centOS updated?
    • nGinix running?
    • System resources?

    As SAM is more on the OS side, I'd say this is more up his alley than mine. So much of this is still Japanese translated Greek...



  • Support Tech said:

    Ok, 6.1 has our improved video encoding routine.
    You may need to assign more CPU to that.
    Our server definitely does perform better on a Windows OS as we use .NET instead of Mono.



  • @JaredBusch said in ScreenConnect on CentOS is sluggish:

    Support Tech said:

    Ok, 6.1 has our improved video encoding routine.
    You may need to assign more CPU to that.
    Our server definitely does perform better on a Windows OS as we use .NET instead of Mono.

    Yes - They push Windows OS pretty hard. Which is 'okay' but dang it,.. we are on centOS and that is where we are staying.



  • @JaredBusch said in ScreenConnect on CentOS is sluggish:

    Support Tech said:
    Ok, 6.1 has our improved video encoding routine.
    You may need to assign more CPU to that.
    Our server definitely does perform better on a Windows OS as we use .NET instead of Mono.

    Really? I should definitely stop developing bus drivers which run perfectly fine on Mono. How could I even think about doing such a thing?



  • Hmm just clicked to a different group of machines and iotop shows this
    0_1486401728366_upload-c2105719-d794-4369-8fcf-632790c63440



  • @gjacobse said in ScreenConnect on CentOS is sluggish:

    @JaredBusch said in ScreenConnect on CentOS is sluggish:

    Support Tech said:

    Ok, 6.1 has our improved video encoding routine.
    You may need to assign more CPU to that.
    Our server definitely does perform better on a Windows OS as we use .NET instead of Mono.

    Yes - They push Windows OS pretty hard. Which is 'okay' but dang it,.. we are on centOS and that is where we are staying.

    Cheaper to push more CPU and RAM at it on Linux. We could double what we have and still be cheaper than where it used to be on Windows. Maybe even quadruple it. And it wasn't that much faster on Windows.



  • @JaredBusch said in ScreenConnect on CentOS is sluggish:

    Our server definitely does perform better on a Windows OS as we use .NET instead of Mono.

    While native .NET does have advantages over Mono, that statement alone doesn't make sense. That's only one piece of the puzzle.



  • @scottalanmiller said in ScreenConnect on CentOS is sluggish:

    @JaredBusch said in ScreenConnect on CentOS is sluggish:

    Our server definitely does perform better on a Windows OS as we use .NET instead of Mono.

    While native .NET does have advantages over Mono, that statement alone doesn't make sense. That's only one piece of the puzzle.

    Totally agree here. Mono does a pretty good job in most cases, even performance-wise.



  • @thwr said in ScreenConnect on CentOS is sluggish:

    @scottalanmiller said in ScreenConnect on CentOS is sluggish:

    @JaredBusch said in ScreenConnect on CentOS is sluggish:

    Our server definitely does perform better on a Windows OS as we use .NET instead of Mono.

    While native .NET does have advantages over Mono, that statement alone doesn't make sense. That's only one piece of the puzzle.

    Totally agree here. Mono does a pretty good job in most cases, even performance-wise.

    And why even use Mono? Native .NET is available. I wonder why they don't mention it?

    https://www.microsoft.com/net/core#linuxredhat



  • Current htop sorted by memory:

    0_1486401995311_2017-02-06 12_25_59-gene@dny-lnx-sc_~.png



  • @scottalanmiller said in ScreenConnect on CentOS is sluggish:

    @thwr said in ScreenConnect on CentOS is sluggish:

    @scottalanmiller said in ScreenConnect on CentOS is sluggish:

    @JaredBusch said in ScreenConnect on CentOS is sluggish:

    Our server definitely does perform better on a Windows OS as we use .NET instead of Mono.

    While native .NET does have advantages over Mono, that statement alone doesn't make sense. That's only one piece of the puzzle.

    Totally agree here. Mono does a pretty good job in most cases, even performance-wise.

    And why even use Mono? Native .NET is available. I wonder why they don't mention it?

    https://www.microsoft.com/net/core#linuxredhat

    That is quite new, and they probably have not spent time to change because they prefer windows.



  • @JaredBusch said in ScreenConnect on CentOS is sluggish:

    @scottalanmiller said in ScreenConnect on CentOS is sluggish:

    @thwr said in ScreenConnect on CentOS is sluggish:

    @scottalanmiller said in ScreenConnect on CentOS is sluggish:

    @JaredBusch said in ScreenConnect on CentOS is sluggish:

    Our server definitely does perform better on a Windows OS as we use .NET instead of Mono.

    While native .NET does have advantages over Mono, that statement alone doesn't make sense. That's only one piece of the puzzle.

    Totally agree here. Mono does a pretty good job in most cases, even performance-wise.

    And why even use Mono? Native .NET is available. I wonder why they don't mention it?

    https://www.microsoft.com/net/core#linuxredhat

    That is quite new, and they probably have not spent time to change because they prefer windows.

    I'm not surprised that they've not put time into it, but just saying that it's slow on Linux because of Mono needs a qualifier like "and we've decided to use Mono for now" or "Native .NET isn't ready yet and lacks something we need". The statement that they make is sensible only because we know Mono is being used by them, but on its own, the statement is weird because Linux is a fully viable Microsoft .NET platform now and it's been a big deal.

    I'm not upset that they haven't tested or ported yet, but they could present that better, I feel. And it suggests that moving to Windows for .NET isn't a long term thing as we already have it native on Linux now. So we just have to wait for them to start using it.



  • @scottalanmiller said in ScreenConnect on CentOS is sluggish:

    @JaredBusch said in ScreenConnect on CentOS is sluggish:

    @scottalanmiller said in ScreenConnect on CentOS is sluggish:

    @thwr said in ScreenConnect on CentOS is sluggish:

    @scottalanmiller said in ScreenConnect on CentOS is sluggish:

    @JaredBusch said in ScreenConnect on CentOS is sluggish:

    Our server definitely does perform better on a Windows OS as we use .NET instead of Mono.

    While native .NET does have advantages over Mono, that statement alone doesn't make sense. That's only one piece of the puzzle.

    Totally agree here. Mono does a pretty good job in most cases, even performance-wise.

    And why even use Mono? Native .NET is available. I wonder why they don't mention it?

    https://www.microsoft.com/net/core#linuxredhat

    That is quite new, and they probably have not spent time to change because they prefer windows.

    I'm not surprised that they've not put time into it, but just saying that it's slow on Linux because of Mono needs a qualifier like "and we've decided to use Mono for now" or "Native .NET isn't ready yet and lacks something we need". The statement that they make is sensible only because we know Mono is being used by them, but on its own, the statement is weird because Linux is a fully viable Microsoft .NET platform now and it's been a big deal.

    I'm not upset that they haven't tested or ported yet, but they could present that better, I feel. And it suggests that moving to Windows for .NET isn't a long term thing as we already have it native on Linux now. So we just have to wait for them to start using it.

    Looks like they are testing it. I asked about it.

    That was a subject of our meeting with development last week. They hit some kind of roadblock, but it's definitely being looked into.



  • @JaredBusch Cool, if they had it out on that in 6-9 months I'd be thrilled. Our performance has been okay with Mono, but the whole Mono-wrapper thing is an unnecessary bit of overhead. I'm guessing the Linux server will surge forward in speed once they get that working. Hopefully no major roadblocks.



  • Alright, upped the memory to 4gb for testing and wiped the sessions.db file.

    So far only a single high CPU warning right at boot time, so going to expect/accept that.

    2017-02-06 11:41:29 CPU user (82.1/84.6/87.0)
    


  • Session.db changed dramatically.

    [[email protected] ~]# ls -l /opt/screenconnect/App_Data/Session.*
    -rw-r--r--. 1 root root    794624 Feb  6 11:41 /opt/screenconnect/App_Data/Session.db
    -rw-r--r--. 1 root root 322928640 Feb  6 11:29 /opt/screenconnect/App_Data/Session.db.2017.02.06
    -rw-r--r--. 1 root root     32768 Feb  6 11:50 /opt/screenconnect/App_Data/Session.db-shm
    -rw-r--r--. 1 root root   4128272 Feb  6 11:50 /opt/screenconnect/App_Data/Session.db-wal
    


  • @JaredBusch

    Question on that - Do you have the system do DB maintenance?0_1486403889779_2017-02-06 12_57_57-NTG Remote Portal.png



  • @gjacobse said in ScreenConnect on CentOS is sluggish:

    @JaredBusch

    Question on that - Do you have the system do DB maintenance?!

    That answer, is no. My bad on that.



  • @JaredBusch said in ScreenConnect on CentOS is sluggish:

    @gjacobse said in ScreenConnect on CentOS is sluggish:

    @JaredBusch

    Question on that - Do you have the system do DB maintenance?!

    That answer, is no. My bad on that.

    It wasn't set on the NTG system at first either. But once set, and allowed to cycle it's kept the DB in check. If you set today, it's likely that it won't process until the next run time, which will be Tuesday Night / Wednesday Morning....

    At least that is what SC Support explained to me.



  • @gjacobse said in ScreenConnect on CentOS is sluggish:

    @JaredBusch said in ScreenConnect on CentOS is sluggish:

    @gjacobse said in ScreenConnect on CentOS is sluggish:

    @JaredBusch

    Question on that - Do you have the system do DB maintenance?!

    That answer, is no. My bad on that.

    It wasn't set on the NTG system at first either. But once set, and allowed to cycle it's kept the DB in check. If you set today, it's likely that it won't process until the next run time, which will be Tuesday Night / Wednesday Morning....

    At least that is what SC Support explained to me.

    I have a clean Session.db, so nothing there now anyway. Debating if I should put the old one back and let this run.



  • ok, just moved the new session DB out and copied the old one back in. going to setup the maintenance task and see what happens.

    [[email protected] ~]# ls -l /opt/screenconnect/App_Data/
    total 635964
    -rw-r--r--. 1 root root       803 Nov  4 12:41 ExtensionConfiguration.xml
    drwxr-xr-x. 2 root root        42 Jan 31 17:59 Helper
    -rw-r--r--. 1 root root       381 Aug 10 01:05 License.xml
    -rw-r--r--. 1 root root     16220 Jan 31 17:59 Role.xml
    -rw-r--r--. 1 root root 322928640 Feb  6 17:03 Session.db
    -rw-r--r--. 1 root root 322928640 Feb  6 11:29 Session.db.2017.02.06.old
    -rw-r--r--. 1 root root   1163264 Feb  6 16:30 Session.db.2017.02.06.new
    -rw-r--r--. 1 root root     32768 Feb  6 17:00 Session.db-shm
    -rw-r--r--. 1 root root   4132392 Feb  6 17:00 Session.db-wal
    -rw-r--r--. 1 root root      1045 Jul  1  2014 SessionEventTrigger.xml
    -rw-r--r--. 1 root root      3250 Jun 21  2016 SessionGroup.xml
    drwxr-xr-x. 2 root root        86 Jan 31 18:09 Toolbox
    -rw-r--r--. 1 root root      6001 Feb  6 12:07 User.xml
    


  • big ba-da boom

    0_1486422455491_upload-f82269d1-a1fe-4d52-bdfc-4ec92b42ad64



  • Put the new one back and it start up just fine /sigh


Log in to reply