502 Bad Gateway; nginx/1.16.1



  • Seems that when I updated my NC last night, the nginx configuration was screwed. Now I'm left with trying to correct it. Simple enough I'm sure, except I've not done much configuration work with nginx. I installed NC a few months ago and accepted all the defaults.

    Host: Vultr
    OS Ver: Ubuntu 19.10

    I'm sure @scottalanmiller or @JaredBusch can point out the error in seconds....



  • Did you install Nextcloud with Nginx instead of Apche?



  • @JaredBusch said in 502 Bad Gateway; nginx/1.16.1:

    Did you install Nextcloud with Nginx instead of Apche?

    I do not recall - as that was months ago. I used the Vultr scripted base install. I've kept it update/upgraded but took the time to upgrade the distro from 18.04 to 19.10



  • [email protected]:~# netstat -lpn
    Active Internet connections (only servers)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
    tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      759/nginx: master p
    tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      766/redis-server 12
    tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      759/nginx: master p
    tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      681/systemd-resolve
    tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      760/sshd
    tcp        0      0 0.0.0.0:61209           0.0.0.0:*               LISTEN      733/python3
    tcp6       0      0 :::443                  :::*                    LISTEN      759/nginx: master p
    tcp6       0      0 ::1:6379                :::*                    LISTEN      766/redis-server 12
    tcp6       0      0 :::80                   :::*                    LISTEN      759/nginx: master p
    tcp6       0      0 :::22                   :::*                    LISTEN      760/sshd
    udp        0      0 127.0.0.53:53           0.0.0.0:*                           681/systemd-resolve
    udp        0      0 ServerIP:68        0.0.0.0:*                           656/systemd-network
    raw6       0      0 :::58                   :::*                    7           656/systemd-network
    Active UNIX domain sockets (only servers)
    Proto RefCnt Flags       Type       State         I-Node   PID/Program name     Path
    unix  2      [ ACC ]     SEQPACKET  LISTENING     15426    1/init               /run/udev/control
    unix  2      [ ACC ]     STREAM     LISTENING     23829    918/systemd          /run/user/0/systemd/private
    unix  2      [ ACC ]     STREAM     LISTENING     15150    1/init               /run/systemd/private
    unix  2      [ ACC ]     STREAM     LISTENING     23835    918/systemd          /run/user/0/gnupg/S.gpg-agent.ssh
    unix  2      [ ACC ]     STREAM     LISTENING     23836    918/systemd          /run/user/0/snapd-session-agent.socket
    unix  2      [ ACC ]     STREAM     LISTENING     23837    918/systemd          /run/user/0/gnupg/S.gpg-agent
    unix  2      [ ACC ]     STREAM     LISTENING     15156    1/init               /run/systemd/journal/stdout
    unix  2      [ ACC ]     STREAM     LISTENING     23838    918/systemd          /run/user/0/pk-debconf-socket
    unix  2      [ ACC ]     STREAM     LISTENING     23839    918/systemd          /run/user/0/gnupg/S.gpg-agent.browser
    unix  2      [ ACC ]     STREAM     LISTENING     23840    918/systemd          /run/user/0/gnupg/S.dirmngr
    unix  2      [ ACC ]     STREAM     LISTENING     23841    918/systemd          /run/user/0/gnupg/S.gpg-agent.extra
    unix  2      [ ACC ]     STREAM     LISTENING     15161    1/init               @/org/kernel/linux/storage/multipathd
    unix  2      [ ACC ]     STREAM     LISTENING     15246    1/init               /run/systemd/fsck.progress
    unix  2      [ ACC ]     STREAM     LISTENING     15420    1/init               /run/lvm/lvmpolld.socket
    unix  2      [ ACC ]     STREAM     LISTENING     18729    1/init               @ISCSIADM_ABSTRACT_NAMESPACE
    unix  2      [ ACC ]     STREAM     LISTENING     18724    1/init               /var/snap/lxd/common/lxd/unix.socket
    unix  2      [ ACC ]     STREAM     LISTENING     18708    1/init               /run/uuidd/request
    unix  2      [ ACC ]     STREAM     LISTENING     18710    1/init               /run/acpid.socket
    unix  2      [ ACC ]     STREAM     LISTENING     18712    1/init               /run/snapd.socket
    unix  2      [ ACC ]     STREAM     LISTENING     18714    1/init               /run/snapd-snap.socket
    unix  2      [ ACC ]     STREAM     LISTENING     18720    1/init               /run/dbus/system_bus_socket
    

    Edit: been so long since i've posted I had forgotten how to code blocking.



  • Nothing in the /var/log/nginx/error.log so I checked /var/log/nginx/nextcloud_https_error.log

    cat nextcloud_https_error.log
    2019/12/31 00:09:16 [error] 760#760: *281 connect() failed (111: Connection refused) while connecting to upstream, client: MY-IP, server: _, request: "GET /csrftoken HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "ServerIP"
    
    


  • Testing the nginx config:

    [email protected]:/var/log/nginx# sudo nginx -t
    nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
    nginx: configuration file /etc/nginx/nginx.conf test is successful
    
    


  • Get us a...

    netstat -tulpn

    There are a lot of unknowns here. My guess is that whatever is running NextCloud, isn't.



  • @scottalanmiller said in 502 Bad Gateway; nginx/1.16.1:

    netstat -tulpn

    [email protected]:/var/log/nginx# netstat -tulpn
    Active Internet connections (only servers)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
    tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      759/nginx: master p
    tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      766/redis-server 12
    tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      759/nginx: master p
    tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      681/systemd-resolve
    tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      760/sshd
    tcp        0      0 0.0.0.0:61209           0.0.0.0:*               LISTEN      733/python3
    tcp6       0      0 :::443                  :::*                    LISTEN      759/nginx: master p
    tcp6       0      0 ::1:6379                :::*                    LISTEN      766/redis-server 12
    tcp6       0      0 :::80                   :::*                    LISTEN      759/nginx: master p
    tcp6       0      0 :::22                   :::*                    LISTEN      760/sshd
    udp        0      0 127.0.0.53:53           0.0.0.0:*                           681/systemd-resolve
    udp        0      0 ServerIP:68        0.0.0.0:*                           656/systemd-network
    


  • Checking to see if nginx is running:

    [email protected]:/var/log/nginx# systemctl status nginx
    ● nginx.service - nginx - high performance web server
       Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
       Active: active (running) since Tue 2019-12-31 01:58:03 UTC; 40min ago
         Docs: http://nginx.org/en/docs/
      Process: 741 ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf (code=exited, status=0/SUCCESS)
     Main PID: 759 (nginx)
        Tasks: 2 (limit: 2287)
       Memory: 5.9M
       CGroup: /system.slice/nginx.service
               ├─759 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
               └─761 nginx: worker process
    
    Dec 31 01:58:03 TuDallasNC systemd[1]: Starting nginx - high performance web server...
    Dec 31 01:58:03 TuDallasNC systemd[1]: nginx.service: Can't open PID file /run/nginx.pid (yet?) after start: No such file or directory
    Dec 31 01:58:03 TuDallasNC systemd[1]: Started nginx - high performance web server.
    
    


  • @gjacobse said in 502 Bad Gateway; nginx/1.16.1:

    Checking to see if nginx is running:

    [email protected]:/var/log/nginx# systemctl status nginx
    ● nginx.service - nginx - high performance web server
       Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
       Active: active (running) since Tue 2019-12-31 01:58:03 UTC; 40min ago
         Docs: http://nginx.org/en/docs/
      Process: 741 ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf (code=exited, status=0/SUCCESS)
     Main PID: 759 (nginx)
        Tasks: 2 (limit: 2287)
       Memory: 5.9M
       CGroup: /system.slice/nginx.service
               ├─759 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
               └─761 nginx: worker process
    
    Dec 31 01:58:03 TuDallasNC systemd[1]: Starting nginx - high performance web server...
    Dec 31 01:58:03 TuDallasNC systemd[1]: nginx.service: Can't open PID file /run/nginx.pid (yet?) after start: No such file or directory
    Dec 31 01:58:03 TuDallasNC systemd[1]: Started nginx - high performance web server.
    
    

    You can fix the nginx.service: Can't open PID file /run/nginx.pid (yet?) after start: No such file or directory by doing following:

    sudo mkdir /etc/systemd/system/nginx.service.d
    printf "[Service]\nExecStartPost=/bin/sleep 0.1\n" | \
    sudo tee /etc/systemd/system/nginx.service.d/override.conf
    sudo systemctl daemon-reload
    sudo systemctl restart nginx
    sudo systemctl stop nginx
    

    https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1581864

    https://medium.com/@parsher/nginx-service-failed-to-parse-pid-from-file-var-run-nginx-pid-invalid-argument-bugs-fda95eeeda73



  • I was reading that a few minutes ago, and search to see if the file was there,.. and it is.

    However, Let me run through it.



  • Issue continues



  • Nextcloud runs on Apache by default, or at least used to. So I have no idea what script you followed to get Nginx.



  • @gjacobse said in 502 Bad Gateway; nginx/1.16.1:

    @JaredBusch said in 502 Bad Gateway; nginx/1.16.1:

    Did you install Nextcloud with Nginx instead of Apche?

    I do not recall - as that was months ago. I used the Vultr scripted base install. I've kept it update/upgraded but took the time to upgrade the distro from 18.04 to 19.10

    thats why i dont like the scripted installs, they are good and gets you going fast, but when times come to troubleshoot... cause you didnt do it, it will take you x100 time to solve it.



  • @JaredBusch said in 502 Bad Gateway; nginx/1.16.1:

    Nextcloud runs on Apache by default, or at least used to. So I have no idea what script you followed to get Nginx.

    Hmm... Is apache running perhaps?

    I've had the reverse of that issue before where apache was started and brought a site down



  • It was the Vultr stock server build for Nextcloud.

    In trying to fix things, I think I have made matters worse, so at this point, I've downloaded the few files that where there, and will re-install the server nuking the current data.

    Of note; the data was found at:

    /var/www/html/data/user17049/files/SHARENAME
    


  • You have the answer in the logs you posted. Nginx can't connect to php handler daemon on port 9000, most likely php-fpm.