NGINX Just Stop Working
-
@NashBrydges said in NGINX Just Stop Working:
@scottalanmiller said in NGINX Just Stop Working:
grep nginx /var/log/messages
/var/log/messagesDoes not exist.
Oh sorry, use Ubuntu's log. That's RHELs.
-
@scottalanmiller said in NGINX Just Stop Working:
@NashBrydges said in NGINX Just Stop Working:
Here is the only entry in the NGINX error log for the last time NGINX stopped.The error log is where it records HTTP errors, not Nginx software errors.
Which is useful for in a case I've seen where the service was started by other means, and showed all addresses were already in use.
-
@scottalanmiller said in NGINX Just Stop Working:
@NashBrydges said in NGINX Just Stop Working:
@scottalanmiller said in NGINX Just Stop Working:
grep nginx /var/log/messages
/var/log/messagesDoes not exist.
Oh sorry, use Ubuntu's log. That's RHELs.
I thought the modern distros these days were
journalctl -forjournalctl|grep blah|less -
@dafyre said in NGINX Just Stop Working:
I thought the modern distros these days were journalctl -f or journalctl|grep blah|less
Just another way to see the same thing.
-
@scottalanmiller Yeah, those log messages are in the syslog file in Ubuntu. The only log entries in syslog are related to when I rebooted the server. GREP output is too large for those related nginx entries in the log for me to post here but just reviewed every line and no errors that I could find.
-
@NashBrydges said in NGINX Just Stop Working:
@scottalanmiller Yeah, those log messages are in the syslog file in Ubuntu. The only log entries in syslog are related to when I rebooted the server. GREP output is too large for those related nginx entries in the log for me to post here but just reviewed every line and no errors that I could find.
Not necessarily looking for errors, just information as to what might have happened.
-
@scottalanmiller It's too large to upload the output here so uploaded a text file to Box and sharing link here. This is a simple txt file.
-
@NashBrydges said in NGINX Just Stop Working:
@scottalanmiller It's too large to upload the output here so uploaded a text file to Box and sharing link here. This is a simple txt file.
Well I see a major issue is that the hostname of the box appears to be nginx. So that essentially makes it unsearchable because you are trying to diagnose the nginx service. Start by giving the system a more appropriate name and that alone will clear up the logs so that you can search for services.
Rule of thumb - hostnames should be meaningful with information that you can't put elsewhere, but not overlap with things like service names.
Name the systems based on things like purpose, not code. So our standard here, if nginx is a proxy, the system would be named something that ends in rproxy to denote its role. If it was an Nginx based web server, it would be named lemp and so forth.
That's why you have a huge log that you can't sort through instead of a tiny one that's super easy to sort through.
-
@NashBrydges are you sure that you are on the right box? The nginx process has literally no logs, at all. The service has never logged that it started, stopped, ran or anything else.
Also, LXD is running on this system. It appears that this is a hypervisor host rather than a web server.
-
You can still search the logs, you just have to account for the name.
So instead of a standard search...
grep nginx mylogYou have to do it twice...
grep "nginx nginx" mylog -
At this point it's likely a more efficient use of time to migrate to a new VM if there's no indication of anything in any logs. Maybe debug logging could shine some light on something like you said, hopefully it does it again when expected, or maybe it won't help at all.... who knows. But being that it can be rebooted all the time, doesn't sound like a LoB service, so yeah, I'd just bring up a new VM and migrate as there is spare time and not waste so much time on further troubleshooting. Before you swap the traffic over to the new one, you can run tests on the new one to see if the same issue exists, too.
-
@NashBrydges said in NGINX Just Stop Working:
It runs nothing but NGINX and is hosted on hyper-v server.
The logs that you showed don't agree here. They show it running other things and not Nginx.
-
@Obsolesce said in NGINX Just Stop Working:
At this point it's likely a more efficient use of time to migrate to a new VM if there's no indication of anything in any logs. Maybe debug logging could shine some light on something like you said, hopefully it does it again when expected, or maybe it won't help at all.... who knows. But being that it can be rebooted all the time, doesn't sound like a LoB service, so yeah, I'd just bring up a new VM and migrate as there is spare time and not waste so much time on further troubleshooting. Before you swap the traffic over to the new one, you can run tests on the new one to see if the same issue exists, too.
And if you rebuild, I highly recommend doing so on the current OS version. This is in no way the issue here, but the more you want reliable, stable, easy to maintain, the more you should be always on the current releases. LTS releases are for very special "failure" situations where additional expertise, effort, and problems are accepted to deal with bad software.
-
What's the output of...
sudo netstat -tulpn -
@NashBrydges said in NGINX Just Stop Working:
The issue is that NGINX will randomly just stop routing requests.
So the proxy action stops, but the service does not? This could easily be something like a firewall or load balancing that is changing and being fixed by the reboot and not related to Nginx directly.
Given the description of the problem, we have several places to look. Nginx being only one of them.
-
Wanted to close the loop here. I've recreated the server and refetched all my certs. It's been working well for the last 2 days. Fingers crossed it stays that way.
Thanks to everyone who participated!
-
@NashBrydges said in NGINX Just Stop Working:
Thanks to everyone who participated!
Is this the age of "everyone gets a trophy" I want praise for sitting here watching with my
!Glad you got it working
-
@DustinB3403 said in NGINX Just Stop Working:
@NashBrydges said in NGINX Just Stop Working:
Thanks to everyone who participated!
Is this the age of "everyone gets a trophy" I want praise for sitting here watching with my
!Glad you got it working
Yes, I'm big on participation ribbons. Lol
-
@NashBrydges said in NGINX Just Stop Working:
Wanted to close the loop here. I've recreated the server and refetched all my certs. It's been working well for the last 2 days. Fingers crossed it stays that way.
Thanks to everyone who participated!
Ah, doing things the DevOps way. When it doubt, nuke and rebuild.
