Elastix reboot: cron job?
-
@thecreativeone91 said:
I personally would use your IT management server if you have one to restart the server remotely rather than on the server itself. Then you can configure checks to make sure it's back up within X mins and send an email alert if not.
I've worked in big environments that did this then moved off of it. It creates dependencies that need not exist. It is "easier to manage" in some ways but if the issue is resolving an SSH issue, for example, one can fix it automatically and the other just fails.
We found that moving from the jump servers doing it to the systems doing it themselves was a huge win. We had more control over the timing, far more reliable reboots (nothing missed because of a network blip), no extraneous dependencies, deals with SSH failures, etc.
-
@scottalanmiller said:
@thecreativeone91 said:
I personally would use your IT management server if you have one to restart the server remotely rather than on the server itself. Then you can configure checks to make sure it's back up within X mins and send an email alert if not.
I've worked in big environments that did this then moved off of it. It creates dependencies that need not exist. It is "easier to manage" in some ways but if the issue is resolving an SSH issue, for example, one can fix it automatically and the other just fails.
We found that moving from the jump servers doing it to the systems doing it themselves was a huge win. We had more control over the timing, far more reliable reboots (nothing missed because of a network blip), no extraneous dependencies, deals with SSH failures, etc.
Monitoring it is much easier when centralized. We'll never move to doing tasks on Servers individually.
-
@g.jacobse said:
@thecreativeone91 said:
It's good to build in checks to make sure there are no current connections (or in this case calls). We do this because we have so many people working after hours. But if they can't access a resource when they try, well that's part of working after hours. If they get disconnected or loss work because of that well then that is a problem.
I personally would use your IT management server if you have one to restart the server remotely rather than on the server itself. Then you can configure checks to make sure it's back up within X mins and send an email alert if not.
That would be my next question; can you configure the cron to do notification when it was about to restart, and once it came up.
I wouldn't send an email unless it fails others you're just turning alerts into noise.
Also relying on itself to tel you if it's successful or failed isn't that great, you really need another machine they connects to it to make sure it's up and then sends an email if it's not.
-
@thecreativeone91 said:
Monitoring it is much easier when centralized. We'll never move to doing tasks on Servers individually.
Monitoring should be centralized, agreed. But the actual maintenance?
-
@thecreativeone91 said:
Also relying on itself to tel you if it's successful or failed isn't that great, you really need another machine they connects to it to make sure it's up and then sends an email if it's not.
That's why you use the monitoring system for that, rather than the maintenance system. Let it handle its own maintenance and the centralized monitoring that should exist anyway can handle the monitoring.
-
@scottalanmiller said:
@thecreativeone91 said:
Monitoring it is much easier when centralized. We'll never move to doing tasks on Servers individually.
Monitoring should be centralized, agreed. But the actual maintenance?
Yes, By saying it's not you are suggestion Group Policy and WSUS are bad.
-
@thecreativeone91 said:
@scottalanmiller said:
@thecreativeone91 said:
Monitoring it is much easier when centralized. We'll never move to doing tasks on Servers individually.
Monitoring should be centralized, agreed. But the actual maintenance?
Yes, By saying it's not you are suggestion Group Policy and WSUS are bad.
Group Policy pushes out the changes, but runs them locally. It would be more like having a centralized script that set the crontabs to always be the same (or in a pattern or whatever.) I think that GPO is more as I have described, local maintenance jobs.
WSUS is patching which is a little different. I would prefer WSUS use localized patching and centralized reporting and filtering.
-
And don't translate "sub-optimal" as bad. But don't think of GPO as centralized pushes of management either.
I'm totally for the cronjobs being pushed out centrally, that makes sense.
-
@thecreativeone91 said:
Yes, By saying it's not you are suggestion Group Policy and WSUS are bad.
If you feel that that was what I was saying, are you implying that you feel that Chef, Puppet, cfEngine and other DevOps tools are bad? (They run locally and only pull centralized configurations.)
-
Thanks for the additional tags
As for e mail, it would be nice to get a health report on successful boot.
No report, something is wrong....
-
@g.jacobse said:
Thanks for the additional tags
As for e mail, it would be nice to get a health report on successful boot.
No report, something is wrong....
It sounds nicer than it is. But in reality you don't want emails saying that things are okay, that's fundamentally wrong. For a couple of reasons:
- At scale, you will be inundated with "everything is fine" emails. It just doesn't scale.
- It trains you to ignore your email and messages from the server. It's bad conditioning.
- You have to notice the absence of an alert to realize that something is wrong rather than the presence of it. You have no way to verify failure, you have to hope that you remember when success was supposed to, hope that you paid attention, etc.
- This doesn't give you a means of passing off to another person or connecting to other automation.
- You have to maintain a list of people to be alerted of "regular stuff."
- You are attempting to use your email as a logging system rather than as an alerting mechanism.
-
If you want this kind of information consider something like...
- Using proper logging and looking in the logs for this information rather than in email.
- Posting to a status feed (e.g. Twitter) with status info that anyone can look up if they care to.
-
@g.jacobse said:
Thanks for the additional tags
As for e mail, it would be nice to get a health report on successful boot.
No report, something is wrong....
That's backwards you want emails when something fails. I'd get way too many if I got once when things worked I easily get 200+ per day already.
-
@thecreativeone91 said:
@g.jacobse said:
Thanks for the additional tags
As for e mail, it would be nice to get a health report on successful boot.
No report, something is wrong....
That's backwards you want emails when something fails. I'd get way too many if I got once when things worked I easily get 200+ per day already.
I rant that over in my mind, and it seemed that if the reboot failed, you would not get any type of email.
-
@g.jacobse said:
@thecreativeone91 said:
@g.jacobse said:
Thanks for the additional tags
As for e mail, it would be nice to get a health report on successful boot.
No report, something is wrong....
That's backwards you want emails when something fails. I'd get way too many if I got once when things worked I easily get 200+ per day already.
I rant that over in my mind, and it seemed that if the reboot failed, you would not get any type of email.
If you have the device itself send the emails. But normally you'd have something else checking status of the network services that would email you after X number of failed attempts to contact the device. We use OpManager for this but there are free tools to do it, spiceworks though bloated can do it. And there's a million and one linux based network monitoring tools.
-
@thecreativeone91 said:
@g.jacobse said:
@thecreativeone91 said:
@g.jacobse said:
Thanks for the additional tags
As for e mail, it would be nice to get a health report on successful boot.
No report, something is wrong....
That's backwards you want emails when something fails. I'd get way too many if I got once when things worked I easily get 200+ per day already.
I rant that over in my mind, and it seemed that if the reboot failed, you would not get any type of email.
If you have the device itself send the emails. But normally you'd have something else checking status of the network services that would email you after X number of failed attempts to contact the device. We use OpManager for this but there are free tools to do it, spiceworks though bloated can do it. And there's a million and one linux based network monitoring tools.
Icinga is really good. I've got a Nagios server deployed right now to do this type of thing, that works fairly well although I really haven't dug too deeply into it.
-
@thecreativeone91 said:
@g.jacobse said:
Thanks for the additional tags
As for e mail, it would be nice to get a health report on successful boot.
No report, something is wrong....
That's backwards you want emails when something fails. I'd get way too many if I got once when things worked I easily get 200+ per day already.
Exactly. Think of emails as an alert. Logging success (somewhere, not email) is good so that you can always have that worried feeling and go look it up. Or just show someone why and when the reboot happened with some definitive info. But don't log to the alert system, save it for alerting. Make your emails matter.
-
@g.jacobse said:
I rant that over in my mind, and it seemed that if the reboot failed, you would not get any type of email.
If the reboot failed to run completely (like the system was down, cron had crashed, things were frozen) you would get no email either. You are making the system report on itself with is not good.
That's why the system should reboot itself (no way around that) but the monitoring of whether it is up or not should be done externally.
