502 Bad Gateway; nginx/1.16.1

gjacobse

Seems that when I updated my NC last night, the nginx configuration was screwed. Now I'm left with trying to correct it. Simple enough I'm sure, except I've not done much configuration work with nginx. I installed NC a few months ago and accepted all the defaults.

Host: Vultr
OS Ver: Ubuntu 19.10

I'm sure @scottalanmiller or @JaredBusch can point out the error in seconds....

JaredBusch

Did you install Nextcloud with Nginx instead of Apche?

gjacobse

@JaredBusch said in 502 Bad Gateway; nginx/1.16.1:

Did you install Nextcloud with Nginx instead of Apche?

I do not recall - as that was months ago. I used the Vultr scripted base install. I've kept it update/upgraded but took the time to upgrade the distro from 18.04 to 19.10

gjacobse

root@TuDallasNC:~# netstat -lpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      759/nginx: master p
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      766/redis-server 12
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      759/nginx: master p
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      681/systemd-resolve
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      760/sshd
tcp        0      0 0.0.0.0:61209           0.0.0.0:*               LISTEN      733/python3
tcp6       0      0 :::443                  :::*                    LISTEN      759/nginx: master p
tcp6       0      0 ::1:6379                :::*                    LISTEN      766/redis-server 12
tcp6       0      0 :::80                   :::*                    LISTEN      759/nginx: master p
tcp6       0      0 :::22                   :::*                    LISTEN      760/sshd
udp        0      0 127.0.0.53:53           0.0.0.0:*                           681/systemd-resolve
udp        0      0 ServerIP:68        0.0.0.0:*                           656/systemd-network
raw6       0      0 :::58                   :::*                    7           656/systemd-network
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name     Path
unix  2      [ ACC ]     SEQPACKET  LISTENING     15426    1/init               /run/udev/control
unix  2      [ ACC ]     STREAM     LISTENING     23829    918/systemd          /run/user/0/systemd/private
unix  2      [ ACC ]     STREAM     LISTENING     15150    1/init               /run/systemd/private
unix  2      [ ACC ]     STREAM     LISTENING     23835    918/systemd          /run/user/0/gnupg/S.gpg-agent.ssh
unix  2      [ ACC ]     STREAM     LISTENING     23836    918/systemd          /run/user/0/snapd-session-agent.socket
unix  2      [ ACC ]     STREAM     LISTENING     23837    918/systemd          /run/user/0/gnupg/S.gpg-agent
unix  2      [ ACC ]     STREAM     LISTENING     15156    1/init               /run/systemd/journal/stdout
unix  2      [ ACC ]     STREAM     LISTENING     23838    918/systemd          /run/user/0/pk-debconf-socket
unix  2      [ ACC ]     STREAM     LISTENING     23839    918/systemd          /run/user/0/gnupg/S.gpg-agent.browser
unix  2      [ ACC ]     STREAM     LISTENING     23840    918/systemd          /run/user/0/gnupg/S.dirmngr
unix  2      [ ACC ]     STREAM     LISTENING     23841    918/systemd          /run/user/0/gnupg/S.gpg-agent.extra
unix  2      [ ACC ]     STREAM     LISTENING     15161    1/init               @/org/kernel/linux/storage/multipathd
unix  2      [ ACC ]     STREAM     LISTENING     15246    1/init               /run/systemd/fsck.progress
unix  2      [ ACC ]     STREAM     LISTENING     15420    1/init               /run/lvm/lvmpolld.socket
unix  2      [ ACC ]     STREAM     LISTENING     18729    1/init               @ISCSIADM_ABSTRACT_NAMESPACE
unix  2      [ ACC ]     STREAM     LISTENING     18724    1/init               /var/snap/lxd/common/lxd/unix.socket
unix  2      [ ACC ]     STREAM     LISTENING     18708    1/init               /run/uuidd/request
unix  2      [ ACC ]     STREAM     LISTENING     18710    1/init               /run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     18712    1/init               /run/snapd.socket
unix  2      [ ACC ]     STREAM     LISTENING     18714    1/init               /run/snapd-snap.socket
unix  2      [ ACC ]     STREAM     LISTENING     18720    1/init               /run/dbus/system_bus_socket

Edit: been so long since i've posted I had forgotten how to code blocking.

gjacobse

Nothing in the /var/log/nginx/error.log so I checked /var/log/nginx/nextcloud_https_error.log

cat nextcloud_https_error.log
2019/12/31 00:09:16 [error] 760#760: *281 connect() failed (111: Connection refused) while connecting to upstream, client: MY-IP, server: _, request: "GET /csrftoken HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "ServerIP"

gjacobse

Testing the nginx config:

root@TuDallasNC:/var/log/nginx# sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

scottalanmiller

Get us a...

netstat -tulpn

There are a lot of unknowns here. My guess is that whatever is running NextCloud, isn't.

gjacobse

@scottalanmiller said in 502 Bad Gateway; nginx/1.16.1:

netstat -tulpn

root@TuDallasNC:/var/log/nginx# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      759/nginx: master p
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      766/redis-server 12
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      759/nginx: master p
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      681/systemd-resolve
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      760/sshd
tcp        0      0 0.0.0.0:61209           0.0.0.0:*               LISTEN      733/python3
tcp6       0      0 :::443                  :::*                    LISTEN      759/nginx: master p
tcp6       0      0 ::1:6379                :::*                    LISTEN      766/redis-server 12
tcp6       0      0 :::80                   :::*                    LISTEN      759/nginx: master p
tcp6       0      0 :::22                   :::*                    LISTEN      760/sshd
udp        0      0 127.0.0.53:53           0.0.0.0:*                           681/systemd-resolve
udp        0      0 ServerIP:68        0.0.0.0:*                           656/systemd-network

gjacobse

Checking to see if nginx is running:

root@TuDallasNC:/var/log/nginx# systemctl status nginx
● nginx.service - nginx - high performance web server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-12-31 01:58:03 UTC; 40min ago
     Docs: http://nginx.org/en/docs/
  Process: 741 ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf (code=exited, status=0/SUCCESS)
 Main PID: 759 (nginx)
    Tasks: 2 (limit: 2287)
   Memory: 5.9M
   CGroup: /system.slice/nginx.service
           ├─759 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
           └─761 nginx: worker process

Dec 31 01:58:03 TuDallasNC systemd[1]: Starting nginx - high performance web server...
Dec 31 01:58:03 TuDallasNC systemd[1]: nginx.service: Can't open PID file /run/nginx.pid (yet?) after start: No such file or directory
Dec 31 01:58:03 TuDallasNC systemd[1]: Started nginx - high performance web server.

black3dynamite

@gjacobse said in 502 Bad Gateway; nginx/1.16.1:

Checking to see if nginx is running:

root@TuDallasNC:/var/log/nginx# systemctl status nginx
● nginx.service - nginx - high performance web server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-12-31 01:58:03 UTC; 40min ago
     Docs: http://nginx.org/en/docs/
  Process: 741 ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf (code=exited, status=0/SUCCESS)
 Main PID: 759 (nginx)
    Tasks: 2 (limit: 2287)
   Memory: 5.9M
   CGroup: /system.slice/nginx.service
           ├─759 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
           └─761 nginx: worker process

Dec 31 01:58:03 TuDallasNC systemd[1]: Starting nginx - high performance web server...
Dec 31 01:58:03 TuDallasNC systemd[1]: nginx.service: Can't open PID file /run/nginx.pid (yet?) after start: No such file or directory
Dec 31 01:58:03 TuDallasNC systemd[1]: Started nginx - high performance web server.

You can fix the nginx.service: Can't open PID file /run/nginx.pid (yet?) after start: No such file or directory by doing following:

sudo mkdir /etc/systemd/system/nginx.service.d
printf "[Service]\nExecStartPost=/bin/sleep 0.1\n" | \
sudo tee /etc/systemd/system/nginx.service.d/override.conf
sudo systemctl daemon-reload
sudo systemctl restart nginx
sudo systemctl stop nginx

https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1581864

https://medium.com/@parsher/nginx-service-failed-to-parse-pid-from-file-var-run-nginx-pid-invalid-argument-bugs-fda95eeeda73

gjacobse

I was reading that a few minutes ago, and search to see if the file was there,.. and it is.

However, Let me run through it.

gjacobse

Issue continues

JaredBusch

Nextcloud runs on Apache by default, or at least used to. So I have no idea what script you followed to get Nginx.

Emad R

@gjacobse said in 502 Bad Gateway; nginx/1.16.1:

@JaredBusch said in 502 Bad Gateway; nginx/1.16.1:

Did you install Nextcloud with Nginx instead of Apche?

I do not recall - as that was months ago. I used the Vultr scripted base install. I've kept it update/upgraded but took the time to upgrade the distro from 18.04 to 19.10

thats why i dont like the scripted installs, they are good and gets you going fast, but when times come to troubleshoot... cause you didnt do it, it will take you x100 time to solve it.

IRJ

@JaredBusch said in 502 Bad Gateway; nginx/1.16.1:

Nextcloud runs on Apache by default, or at least used to. So I have no idea what script you followed to get Nginx.

Hmm... Is apache running perhaps?

I've had the reverse of that issue before where apache was started and brought a site down

gjacobse

It was the Vultr stock server build for Nextcloud.

In trying to fix things, I think I have made matters worse, so at this point, I've downloaded the few files that where there, and will re-install the server nuking the current data.

Of note; the data was found at:

/var/www/html/data/user17049/files/SHARENAME

marcinozga

You have the answer in the logs you posted. Nginx can't connect to php handler daemon on port 9000, most likely php-fpm.