NextCloud web interface painfully slow

bnrstnr

This is after reboot. It's still crazy slow. I rebooted a couple times yesterday, too. Even installing glances with DNF was crazy slow, and opening it takes about 20 seconds.

Glances had already been running for 28 min at this point, so I'm sure any spikes from opening had subsided by then.

0_1543595922876_12a57eb8-5b8b-4292-bbae-774a39e41422-image.png

bnrstnr

@dafyre said in NextCloud web interface painfully slow:

This. After you reboot, pay attention to the STEAL time. I've never noticed that in the display before... In your glances pic above, steal % is at 48%.
I had to go google it to see what it was, lol... The short version is that steal % is how long your VM has to wait until the Hypervisor gives it more CPU time.
This article, though dated, seems to give good info.
(Understanding CPU Steal Time)

This almost sounds like I have a noisy neighbor on my VPS?

bnrstnr

@black3dynamite said in NextCloud web interface painfully slow:

For the 504 Gateway Time-Out, and if you are using nginx has proxy, do you have this inside the location block in your nextcloud.conf file?
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;

I have nothing relating to timeouts in my nginx conf.

Should I add it? I'm assuming it's using whatever the nginx default is.

travisdh1

@bnrstnr said in NextCloud web interface painfully slow:

@dafyre said in NextCloud web interface painfully slow:

This. After you reboot, pay attention to the STEAL time. I've never noticed that in the display before... In your glances pic above, steal % is at 48%.
I had to go google it to see what it was, lol... The short version is that steal % is how long your VM has to wait until the Hypervisor gives it more CPU time.
This article, though dated, seems to give good info.
(Understanding CPU Steal Time)

This almost sounds like I have a noisy neighbor on my VPS?

Very well could be.

JaredBusch

@bnrstnr said in NextCloud web interface painfully slow:

@dafyre said in NextCloud web interface painfully slow:

This. After you reboot, pay attention to the STEAL time. I've never noticed that in the display before... In your glances pic above, steal % is at 48%.
I had to go google it to see what it was, lol... The short version is that steal % is how long your VM has to wait until the Hypervisor gives it more CPU time.
This article, though dated, seems to give good info.
(Understanding CPU Steal Time)

This almost sounds like I have a noisy neighbor on my VPS?

I would definitely submit a ticket to Vultr about it.

scottalanmiller

@bnrstnr said in NextCloud web interface painfully slow:

EDIT: This instance is actually Fedora 27, not 28.

Not your issue, but troubleshooting two versions behind is never ideal. Get up to Fedora 29 before investigating further.

But NC requires a lot of resources and is always decently slow on the web page. But not ten seconds slow.

Our NC is Fedora 29, heavily tuned, has like six cores, 12GB of RAM, doesn't have any noisy neighbours (not cloud) and you still feel the lag if doing the web interface.

black3dynamite

@bnrstnr said in NextCloud web interface painfully slow:

@black3dynamite said in NextCloud web interface painfully slow:
For the 504 Gateway Time-Out, and if you are using nginx has proxy, do you have this inside the location block in your nextcloud.conf file?
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;
I have nothing relating to timeouts in my nginx conf.

Should I add it? I'm assuming it's using whatever the nginx default is.

For Nextcloud, yes. It also helps with when updating Nextcloud via the web interface too.

scottalanmiller

@bnrstnr said in NextCloud web interface painfully slow:

We have WOW cable internet, like 240/20ish

I've poked around and found some other users NC installs and their login pages load almost instantly for me

Vultr storage instances are very slow, they are meant for capacity, not speed. This will definitely contribute to the issue.

Make sure you have swap available. 2GB isn't terrible, but not quite enough for even a tiny instance to run unimpeded.

bnrstnr

@scottalanmiller said in NextCloud web interface painfully slow:

Make sure you have swap available. 2GB isn't terrible, but not quite enough for even a tiny instance to run unimpeded.

I could definitely jack this up to whatever I want.

I will also work on getting this up to Fedora 29, I didn't even realize it was still on 27 :face_with_open_mouth_cold_sweat:

Also, I just submitted a ticket, we'll see what they say.

scottalanmiller

@bnrstnr said in NextCloud web interface painfully slow:

@dafyre said in NextCloud web interface painfully slow:

This. After you reboot, pay attention to the STEAL time. I've never noticed that in the display before... In your glances pic above, steal % is at 48%.
I had to go google it to see what it was, lol... The short version is that steal % is how long your VM has to wait until the Hypervisor gives it more CPU time.
This article, though dated, seems to give good info.
(Understanding CPU Steal Time)

This almost sounds like I have a noisy neighbor on my VPS?

always possible, but rarely seen as a CPU spike.

scottalanmiller

@bnrstnr said in NextCloud web interface painfully slow:

@scottalanmiller said in NextCloud web interface painfully slow:

Make sure you have swap available. 2GB isn't terrible, but not quite enough for even a tiny instance to run unimpeded.

I could definitely jack this up to whatever I want.

I will also work on getting this up to Fedora 29, I didn't even realize it was still on 27 :face_with_open_mouth_cold_sweat:

Also, I just submitted a ticket, we'll see what they say.

I'd add 2GB of swap.

JaredBusch

@scottalanmiller said in NextCloud web interface painfully slow:

@bnrstnr said in NextCloud web interface painfully slow:

@dafyre said in NextCloud web interface painfully slow:

This. After you reboot, pay attention to the STEAL time. I've never noticed that in the display before... In your glances pic above, steal % is at 48%.
I had to go google it to see what it was, lol... The short version is that steal % is how long your VM has to wait until the Hypervisor gives it more CPU time.
This article, though dated, seems to give good info.
(Understanding CPU Steal Time)

This almost sounds like I have a noisy neighbor on my VPS?

always possible, but rarely seen as a CPU spike.

It is not a spike. it is his system constantly waiting on CPU resources from the host.

bnrstnr

There were 260 failed login attempts since the last successful login.

I may be getting slammed by script kiddies here too... This is with fail2ban on. My last login was almost exactly 2 hours ago. I will change my SSH port and see if that helps anything.

DustinB3403

@bnrstnr said in NextCloud web interface painfully slow:

There were 260 failed login attempts since the last successful login.

I may be getting slammed by script kiddies here too... This is with fail2ban on. My last login was almost exactly 2 hours ago. I will change my SSH port and see if that helps anything.

Change your login method to use keys instead of U&P. . .

scottalanmiller

@bnrstnr said in NextCloud web interface painfully slow:

There were 260 failed login attempts since the last successful login.

I may be getting slammed by script kiddies here too... This is with fail2ban on. My last login was almost exactly 2 hours ago. I will change my SSH port and see if that helps anything.

That's totally normal traffic and not being slammed at all. That's light.

bnrstnr

@scottalanmiller said in NextCloud web interface painfully slow:

That's totally normal traffic and not being slammed at all. That's light.

I usually see that many in a couple hours with fail2ban off, with it on I might see 260 in a week.

jmoore

@bnrstnr i agree with Scott here, that is nothing in terms of attempts. My stuff will get thousands in a few minutes when it happens.

bnrstnr

@jmoore said in NextCloud web interface painfully slow:

@bnrstnr i agree with Scott here, that is nothing in terms of attempts. My stuff will get thousands in a few minutes when it happens.

Alright, that's good, one less thing to check off my list. Like I said, it's more than I normally see, but I definitely believe you guys. Thanks

Dashrender

@travisdh1 said in NextCloud web interface painfully slow:

@bnrstnr How's the client ISP connection? I've seen some major issues at handoff points to Level3 since their merger with Comcast.

Comcast? I thought Level 3 was bought by Century Link?

http://news.centurylink.com/2017-11-01-CenturyLink-completes-acquisition-of-Level-3

scottalanmiller

@Dashrender said in NextCloud web interface painfully slow:

@travisdh1 said in NextCloud web interface painfully slow:

@bnrstnr How's the client ISP connection? I've seen some major issues at handoff points to Level3 since their merger with Comcast.

Comcast? I thought Level 3 was bought by Century Link?

http://news.centurylink.com/2017-11-01-CenturyLink-completes-acquisition-of-Level-3

It was.