NGINX Just Stop Working
-
certbot.timer failing?
https://stackoverflow.com/a/52967898
-
@black3dynamite I'm not seeing any evidence of this failing in the letsencrypt.log file syslog or nginx logs (both access and error). Would those logs be elsewhere? Obviously I don't want to have to manually renew certs.
-
@NashBrydges letsencrypt.log is the only one I'm aware of. Actually are using systemd to renew your certs or cronjob?
-
@black3dynamite systemd...
-
@NashBrydges said in NGINX Just Stop Working:
I also ran the NGINX test and all looks good.
If they weren't it wouldn't even start up.
-
@NashBrydges said in NGINX Just Stop Working:
Here is the only entry in the NGINX error log for the last time NGINX stopped.
The error log is where it records HTTP errors, not Nginx software errors.
-
@scottalanmiller Well at this point I'm looking at any log that has "error" in the name. Lol
-
@NashBrydges said in NGINX Just Stop Working:
@scottalanmiller Well at this point I'm looking at any log that has "error" in the name. Lol
This should show you what there is for Nginx itself....
grep nginx /var/log/messages -
@scottalanmiller said in NGINX Just Stop Working:
grep nginx /var/log/messages
/var/log/messagesDoes not exist.
-
@NashBrydges said in NGINX Just Stop Working:
@scottalanmiller said in NGINX Just Stop Working:
grep nginx /var/log/messages
/var/log/messagesDoes not exist.
Oh sorry, use Ubuntu's log. That's RHELs.
-
@scottalanmiller said in NGINX Just Stop Working:
@NashBrydges said in NGINX Just Stop Working:
Here is the only entry in the NGINX error log for the last time NGINX stopped.The error log is where it records HTTP errors, not Nginx software errors.
Which is useful for in a case I've seen where the service was started by other means, and showed all addresses were already in use.
-
@scottalanmiller said in NGINX Just Stop Working:
@NashBrydges said in NGINX Just Stop Working:
@scottalanmiller said in NGINX Just Stop Working:
grep nginx /var/log/messages
/var/log/messagesDoes not exist.
Oh sorry, use Ubuntu's log. That's RHELs.
I thought the modern distros these days were
journalctl -forjournalctl|grep blah|less -
@dafyre said in NGINX Just Stop Working:
I thought the modern distros these days were journalctl -f or journalctl|grep blah|less
Just another way to see the same thing.
-
@scottalanmiller Yeah, those log messages are in the syslog file in Ubuntu. The only log entries in syslog are related to when I rebooted the server. GREP output is too large for those related nginx entries in the log for me to post here but just reviewed every line and no errors that I could find.
-
@NashBrydges said in NGINX Just Stop Working:
@scottalanmiller Yeah, those log messages are in the syslog file in Ubuntu. The only log entries in syslog are related to when I rebooted the server. GREP output is too large for those related nginx entries in the log for me to post here but just reviewed every line and no errors that I could find.
Not necessarily looking for errors, just information as to what might have happened.
-
@scottalanmiller It's too large to upload the output here so uploaded a text file to Box and sharing link here. This is a simple txt file.
-
@NashBrydges said in NGINX Just Stop Working:
@scottalanmiller It's too large to upload the output here so uploaded a text file to Box and sharing link here. This is a simple txt file.
Well I see a major issue is that the hostname of the box appears to be nginx. So that essentially makes it unsearchable because you are trying to diagnose the nginx service. Start by giving the system a more appropriate name and that alone will clear up the logs so that you can search for services.
Rule of thumb - hostnames should be meaningful with information that you can't put elsewhere, but not overlap with things like service names.
Name the systems based on things like purpose, not code. So our standard here, if nginx is a proxy, the system would be named something that ends in rproxy to denote its role. If it was an Nginx based web server, it would be named lemp and so forth.
That's why you have a huge log that you can't sort through instead of a tiny one that's super easy to sort through.
-
@NashBrydges are you sure that you are on the right box? The nginx process has literally no logs, at all. The service has never logged that it started, stopped, ran or anything else.
Also, LXD is running on this system. It appears that this is a hypervisor host rather than a web server.
-
You can still search the logs, you just have to account for the name.
So instead of a standard search...
grep nginx mylogYou have to do it twice...
grep "nginx nginx" mylog -
At this point it's likely a more efficient use of time to migrate to a new VM if there's no indication of anything in any logs. Maybe debug logging could shine some light on something like you said, hopefully it does it again when expected, or maybe it won't help at all.... who knows. But being that it can be rebooted all the time, doesn't sound like a LoB service, so yeah, I'd just bring up a new VM and migrate as there is spare time and not waste so much time on further troubleshooting. Before you swap the traffic over to the new one, you can run tests on the new one to see if the same issue exists, too.
