Nginx Active-Passive HA

NashBrydges

I have a client who is about to migrate to using Let's Encrypt for SSL instead of their standard SSL issuer and fully manual process they had before. They host many hundreds of sites and manually updating certs was just ridiculously time consuming for them.

I'm looking to setup Nginx in Active-Passive HA mode so that when the cert update job takes Nginx offline for up to 15-20mins, the sites aren't taken offline.

I've found a couple tutorials that explain the setup process and will be testing this setup to death before it goes online (virtual IP, defining the master/passive node...etc) but I'm wondering if there is a best-practice for the SSL certs location. Should each Nginx instance host its own set of certs for the same domains? In this case, running the renew script on one would renew certs on only that instance (since Nginx has to reload to use the new certs) and then renew on the other node? I can't imagine I should save the certs on some network location because the remaining Nginx node would not be able to use the new certs until reload so in effect negating the HA setup. Should I simply have a script to copy the new certs to the other node after the master comes back online and then reload the other node's Nginx service?

The majority of these sites are low traffic (fewer than 100 visits a day) so offline sites for a few minutes a day or once a week during early morning hours isn't going to kill anyone but it's still a good plan to setup the HA proxies should one go down and a bonus if we can keep sites online while certs are getting renewed.

Thoughts? Recommendations? Gotchas?

NashBrydges

I'm guessing someone might suggest some automation method like SaltStack for this (not even sure if that's doable) but if you are going to suggest this, please provide a link to documentation where I can read up on it.

JaredBusch

Nginx does not have to go offline if you have the .well-known routed correctly.

It would still need to restart for the cert to be applied of course.

scottalanmiller

My Nginx doesn't go offline during a cert renewal, do them all of the time.

scottalanmiller

@jaredbusch said in Nginx Active-Passive HA:

It would still need to restart for the cert to be applied of course.

Just a reload, no downtime.

NashBrydges

Maybe I'm going renewals wrong or I'm misunderstanding the process but the renew script has the certbot renew --pre-hook "systemctl stop nginx" --post-hook "systemctl start nginx" line. Wouldn't that take Nginx offline, then renew certs, then restart Nginx? Maybe there's a better renewal method I'm not aware of.

Tbh, I've only assumed Nginx was going offline because of this line but only renewing a dozen or so certs only takes seconds so it isn't something I've actually had a chance to test.

JaredBusch

@scottalanmiller said in Nginx Active-Passive HA:

My Nginx doesn't go offline during a cert renewal, do them all of the time.

Mine does because I have not setup the .wellknown path as I do everything certonly when adding a cert. This means the certbot renew needs to shutdown nginx and run its own websesrver temporarily. It is all scripted with a pre-hook and post-hook to stop and start nginx though. so it is still fully automated.

I need to revisit this as cerbot is smarter now than it used to be.

NashBrydges

@jaredbusch said in Nginx Active-Passive HA:

@scottalanmiller said in Nginx Active-Passive HA:

My Nginx doesn't go offline during a cert renewal, do them all of the time.

Mine does because I have not setup the .wellknown path as I do everything certonly when adding a cert. This means the certbot renew needs to shutdown nginx and run its own websesrver temporarily. It is all scripted with a pre-hook and post-hook to stop and start nginx though. so it is still fully automated.

I need to revisit this as cerbot is smarter now than it used to be.

Yeah, this is the method I use as well.

scottalanmiller

@nashbrydges said in Nginx Active-Passive HA:

Maybe I'm going renewals wrong or I'm misunderstanding the process but the renew script has the certbot renew --pre-hook "systemctl stop nginx" --post-hook "systemctl start nginx" line.

I don't use this part: "--pre-hook "systemctl stop nginx"

JaredBusch

@nashbrydges said in Nginx Active-Passive HA:

Maybe I'm going renewals wrong or I'm misunderstanding the process but the renew script has the certbot renew --pre-hook "systemctl stop nginx" --post-hook "systemctl start nginx" line. Wouldn't that take Nginx offline, then renew certs, then restart Nginx? Maybe there's a better renewal method I'm not aware of.

Tbh, I've only assumed Nginx was going offline because of this line but only renewing a dozen or so certs only takes seconds so it isn't something I've actually had a chance to test.

Yes, that takes Nginx offline.

JaredBusch

@scottalanmiller said in Nginx Active-Passive HA:

@nashbrydges said in Nginx Active-Passive HA:

Maybe I'm going renewals wrong or I'm misunderstanding the process but the renew script has the certbot renew --pre-hook "systemctl stop nginx" --post-hook "systemctl start nginx" line.

I don't use this part: "--pre-hook "systemctl stop nginx"

You have to depending on how you got the cert to begin with.

black3dynamite

@scottalanmiller said in Nginx Active-Passive HA:

@jaredbusch said in Nginx Active-Passive HA:

It would still need to restart for the cert to be applied of course.

Just a reload, no downtime.

Is this what you mean?

certbot certonly --webroot -w /path/to/your/webroot -d example.com --post-hook="service nginx reload"

NashBrydges

@black3dynamite said in Nginx Active-Passive HA:

@scottalanmiller said in Nginx Active-Passive HA:

@jaredbusch said in Nginx Active-Passive HA:

It would still need to restart for the cert to be applied of course.

Just a reload, no downtime.

Is this what you mean?
certbot certonly --webroot -w /path/to/your/webroot -d example.com --post-hook="service nginx reload"

This will work if you define the webroot path which I don't. Separate Nginx server from web servers.

NashBrydges

My initial cert request process looks like this:

certbot certonly -d mydomain.com --pre-hook "systemctl stop nginx" --post-hook "systemctl start nginx" --preferred-challenges http

When prompted, I select 1 to spin up a temporary web server for the issuance and challenge. This as I understand it allows me to not have to name webroot folders anywhere. I've already defined the path of the certs because this is easy to figure out based on the command line that will save the certs in the location for the first named domain so when Nginx restarts, certs and domain are all good to go. I have a separate Nginx server that handles nothing but proxy and SSL services. All sites are hosted on their own Fedora, CentOS or Ubuntu servers. I don't use webroot authentication.

If I setup .well-known path, can this be setup globally for all cert issuances and renewals? I guess I would set this up in my config file for each domain.

scottalanmiller

Yeah, that's nothing like what my initial looks like.

black3dynamite

Using well-known path looks like a better approach.

https://community.letsencrypt.org/t/auto-renewal-with-nginx-without-downtime/7814/2

https://community.letsencrypt.org/t/auto-renewal-with-nginx-without-downtime/7814/4

https://github.com/mbrugger/letsencrypt-nginx-docker/blob/master/README.md

JaredBusch

@black3dynamite correct. this is what I need to setup on my system.

dafyre

server {
       listen         80;
       server_name    my.domain.com;
       return         301 https://$server_name$request_uri;

        location /.well-known/acme-challenge {
            root /var/www/letsencrypt;
         }
}

Is what an example I have on one of mine.

dafyre

Honest question... Why not just rsync /etc/letsencrypt from ServerA to ServerB after the certs are renewed?

JaredBusch

@dafyre said in Nginx Active-Passive HA:

Honest question... Why not just rsync /etc/letsencrypt from ServerA to ServerB after the certs are renewed?

There is not discussion about the second server at this point. it is all about the initial renew.