Handling DNS in a Single Active Directory Domain Controller Environment
-
@kelly said in Handling DNS in a Single Active Directory Domain Controller Environment:
So, backups. Here is where we may differ a bit. If there is a backup solution that is up to the task of backing up AD and restoring it without any additional issues then I would postulate that the business would have room on that hardware to run a second DC running as a Core install.
Many that I see use consumer or prosumer NAS devices for this, or even USB drives. AD doesn't have any special considerations for backup, it's a normal live/live database. The included Windows backup tools, free Veeam agent, etc. will back up AD no problem.
I don't know what average is here, but in companies that I've seen the majority have backups that will handle AD no problem, but many, but far from most, have redundant hardware to run on. But if they do, they could using just their simple backup mechanisms.
-
@kelly said in Handling DNS in a Single Active Directory Domain Controller Environment:
Edit: The $1,500 to $2,000 might seem like quite a bit, but those costs are pretty minimal compared to down time and repair costs if you do suffer a catastrophic failure.
A catastrophic DC failure in a single-DC environment means you have to spend 5-10 minutes restoring it from backup. Then you are done. As it's the only DC, you won't run into any of those weird multi-DC issues. That hardly justifies $2k to start for a SMB that has never pointed out a reason to need HA DC services.
-
@kelly said in Handling DNS in a Single Active Directory Domain Controller Environment:
If there aren't sufficient resources to spin up a second DC then there isn't anything to restore the back up to either. This means that there is no AD, DNS, or DHCP until something is put into place. Since every business class edge device can handle the latter two, we'll assume that they get moved over by our on site person. The time to do this is extremely variable. Let's grab an hour as the lower bound. So $25 to get DNS and DHCP up and running so that there is external access, plus employee labor impact. DHCP will continue to be fine during that one hour unless a lease comes up. DNS, however, means that most modern devices will stop functioning right away. A business of 10 employees averaging $15/hr means that there is a cost of $150 for that down time.
Correct, so we reasonably have to assume downtime until "new hardware is procured."
DNS can be pretty easily handled in a number of ways. Assuming you are a single AD shop, this should be done proactively, not reactively. So there should be no downtime or cost here, not even of labour. Whether you need internal DNS and your secondary is the edge device, or you have no internal needs and just go to Google or whatever, that should be "handled". Sure you can't update DNS during this time, but there is no downtime.
DHCP is similar. If you are running in a single DHCP environment, set leases to something long, say 72 hours, and you will always have a day or two to worry about a lease. We assume that you'll have new hardware up in under, say, 36 hours. Maybe that is a bad assumption, but there it is.
So DNS and DHCP should be "handled" with a little forethought. No more forethought than is required to put in a second AD server, so that's a draw IMHO.
I think we can agree... if someone screws up their environment, makes dependencies on things like DNS or DHCP being up continuously to do basic things like print, and let's them go down without any planning that money will be lost. But I think the idea here is that with both of those (DNS specific to this thread) that there are trivially easy, and free ways, to proactively ensure that enough DNS, which might be all of it or just some of it or even none in extreme cases, can be provided during an outage to ensure zero to nominal impact.
In a tiny environment, only ten people, you might even just use IP addresses for things like printers.
-
@kelly said in Handling DNS in a Single Active Directory Domain Controller Environment:
At this point, all internal services are down until AD is restored. Another variable that is difficult to account for.
Just to be clear, I work with many single AD environments and one of this happen. Most have some impact, none stop working. Each is different, some have no impact, for others it is pretty annoying. But none have large financial impacts for the first one to two days without AD working.
So while some setups that create a lot of AD dependencies could have this risk, it's not a given. And in all cases, we have to assume that the environment is set up appropriately for the situation chosen.
-
@kelly said in Handling DNS in a Single Active Directory Domain Controller Environment:
A business of 10 employees averaging $15/hr means that there is a cost of $150 for that down time.
This is a handy number to use because it is easy to price. But it's important to remember that it's kind of an average. Each business will be unique. One business will lose only $5/hr of productivity because their workers were idle anyway (I've literally had CEOs tell me this.) Another will lose $1,000/hr because while those workers only get paid $15, they earn $115 for the company. That sort of thing.
It's a handy way to discuss likely impacts quickly. Just don't want anyone to think that the per hourly price of an employee directly ties to impact numbers. It is sometimes lower, but often much higher, than their hourly rate, even their loaded cost.
I'd say more of an industry average that a $15/hr hour worker would actually cost a company more like $35/hr should they be forced to go truly idle for an extended period of time (but most can offset a few minutes of idleness by getting coffee or taking lunch early.)
-
@scottalanmiller said in Handling DNS in a Single Active Directory Domain Controller Environment:
@kelly said in Handling DNS in a Single Active Directory Domain Controller Environment:
A business of 10 employees averaging $15/hr means that there is a cost of $150 for that down time.
This is a handy number to use because it is easy to price. But it's important to remember that it's kind of an average. Each business will be unique. One business will lose only $5/hr of productivity because their workers were idle anyway (I've literally had CEOs tell me this.) Another will lose $1,000/hr because while those workers only get paid $15, they earn $115 for the company. That sort of thing.
It's a handy way to discuss likely impacts quickly. Just don't want anyone to think that the per hourly price of an employee directly ties to impact numbers. It is sometimes lower, but often much higher, than their hourly rate, even their loaded cost.
I'd say more of an industry average that a $15/hr hour worker would actually cost a company more like $35/hr should they be forced to go truly idle for an extended period of time (but most can offset a few minutes of idleness by getting coffee or taking lunch early.)
It depends, though... the scenario is the physical host going down (Kelly said "one VM host"). It's hard to assume a small SMB bought an entire extra server just to have a single VM host ONLY for AD. This being a small business means likely everything else is on that host too. What good is AD/DNS/DHCP if nothing else is working either?
-
@obsolesce that is just one possible outage, and still with warranties you'd be back up and running usually within 24 hours if not faster.
-
@kelly said in Handling DNS in a Single Active Directory Domain Controller Environment:
Yes, you can defray or reduce almost all of the points of impact, like configuring your DNS scenario the way @JaredBusch suggested, but that is not a common approach. Implementing it and other things to negate potential impacts of down time (scripting system capable of pushing out a new hosts file to clients reducing down time to less than 1 hour, etc.) have their own costs inherent to them, and are also not common in SMB, so I don't think that including them in the comparison of the two approaches.
I see where you are going, but this is something I disagree with. The problem here is that we assume having two (or more) AD servers will be done "correctly" and be set up to do what is needed. But then we assume that if we don't do that, that what is done will be done poorly. That's not a useful comparison mechanism.
This is the same problem with discussing college vs. going straight to the workforce. Most people use the alternative to going to college as just sitting idle for four years then entering the workforce at the same time. And while possible, that's not our reasonable alternative.
In all cases we have to consider "what good looks like." You are absolutely correct, any "average" shop will do things horribly and screw stuff up. It'll be configured wrong, passwords will be default, failover won't work, proactive work will be ignored, best practices won't even be considered, etc. Absolutely, 100% true. I agree, no question.
However, we can't consider that when comparing solutions. That's a false alternative. If we approached from the other view, we could say that the "alternative to setting up a good DNS failover in the router" was to pay for two AD DCs, but since most shops get that wrong their second DC won't work and so we should discount it.
Basically we are assuming that one solution will be done well, and one will be done poorly. Apples to oranges. To have a good decision system, we have to look at it in one of two ways...
-
The "average" of both and use this for statistical analysis, but not advice, as to how things work. In this case, reactive rather than proactive failover without dual AD, or poorly implemented and tested AD that is likely out of date and doesn't have proper backups.
-
The "advice" way, which is either have good, tested and working redundant AD or have a proactive single AD DC setup with failover or mitigation mechanisms to handle the scenario.
The goal in a discussion like this is the latter - to provide a "what's good" to provide good advice. The good advice is always here to do whatever you are doing well. No one would recommend either solution implemented inappropriately.
Does that make more sense?
-
-
If one hypervisor fails, IE a bad power supply. You order a replacement or have warranty come install a new one. If your hypervisor is built on commodity hardware (supermicro) you could likely put any power supply in from something you have on a shelf.
There are a lot of ways to correct downed hardware.
I would question how many systems have people had that have gone up in smoke and not had any other means of recovery at all?
-
@dustinb3403 said in Handling DNS in a Single Active Directory Domain Controller Environment:
@obsolesce that is just one possible outage, and still with warranties you'd be back up and running usually within 24 hours if not faster.
But you can restore a VM to any host. So you don't need the original hardware to restore a VM. Kelly said the SMB depends on AD, and has a dedicated host with a single VM on it for AD. This means there's other servers in play, to host the services that depend on AD. Likely also VM Hosts because, well, that's the proper way to do it.
-
@obsolesce said in Handling DNS in a Single Active Directory Domain Controller Environment:
@scottalanmiller said in Handling DNS in a Single Active Directory Domain Controller Environment:
@kelly said in Handling DNS in a Single Active Directory Domain Controller Environment:
A business of 10 employees averaging $15/hr means that there is a cost of $150 for that down time.
This is a handy number to use because it is easy to price. But it's important to remember that it's kind of an average. Each business will be unique. One business will lose only $5/hr of productivity because their workers were idle anyway (I've literally had CEOs tell me this.) Another will lose $1,000/hr because while those workers only get paid $15, they earn $115 for the company. That sort of thing.
It's a handy way to discuss likely impacts quickly. Just don't want anyone to think that the per hourly price of an employee directly ties to impact numbers. It is sometimes lower, but often much higher, than their hourly rate, even their loaded cost.
I'd say more of an industry average that a $15/hr hour worker would actually cost a company more like $35/hr should they be forced to go truly idle for an extended period of time (but most can offset a few minutes of idleness by getting coffee or taking lunch early.)
It depends, though... the scenario is the physical host going down (Kelly said "one VM host"). It's hard to assume a small SMB bought an entire extra server just to have a single VM host ONLY for AD. This being a small business means likely everything else is on that host too. What good is AD/DNS/DHCP if nothing else is working either?
Right, that's the problem with this kind of discussion... it's not "in this one scenario", but a massive range of different scenarios each with its own problems and solutions.
Now, that is always why we "always consider the unique situation" and never use a single approach. Each business is different.
In a business with only one piece of hardware for all services, chances are losing AD means all else is already lost, so having redundant AD might literally do nothing.
-
@obsolesce said in Handling DNS in a Single Active Directory Domain Controller Environment:
@dustinb3403 said in Handling DNS in a Single Active Directory Domain Controller Environment:
@obsolesce that is just one possible outage, and still with warranties you'd be back up and running usually within 24 hours if not faster.
But you can restore a VM to any host. So you don't need the original hardware to restore a VM. Kelly said the SMB depends on AD, and has a dedicated host with a single VM on it for AD. This means there's other servers in play, to host the services that depend on AD. Likely also VM Hosts because, well, that's the proper way to do it.
I would defer to @Kelly but going on out a limb I would assume she means there is a Single VM that is operating AD. And a single hypervisor that is hosting other services including this AD.
-
@dustinb3403 said in Handling DNS in a Single Active Directory Domain Controller Environment:
@obsolesce said in Handling DNS in a Single Active Directory Domain Controller Environment:
@dustinb3403 said in Handling DNS in a Single Active Directory Domain Controller Environment:
@obsolesce that is just one possible outage, and still with warranties you'd be back up and running usually within 24 hours if not faster.
But you can restore a VM to any host. So you don't need the original hardware to restore a VM. Kelly said the SMB depends on AD, and has a dedicated host with a single VM on it for AD. This means there's other servers in play, to host the services that depend on AD. Likely also VM Hosts because, well, that's the proper way to do it.
I would defer to @Kelly but going on out a limb I would assume she means there is a Single VM that is operating AD. And a single hypervisor that is hosting other services.
That's what I thought that he meant.
-
@obsolesce said in Handling DNS in a Single Active Directory Domain Controller Environment:
@dustinb3403 said in Handling DNS in a Single Active Directory Domain Controller Environment:
@obsolesce that is just one possible outage, and still with warranties you'd be back up and running usually within 24 hours if not faster.
But you can restore a VM to any host. So you don't need the original hardware to restore a VM. Kelly said the SMB depends on AD, and has a dedicated host with a single VM on it for AD. This means there's other servers in play, to host the services that depend on AD. Likely also VM Hosts because, well, that's the proper way to do it.
Once you have a single host only for AD, either you are so tiny that there is nothing else to affect, lol. Or you are likely so large as to not be an SMB. A full server only for your primary AD capacity would almost certainly imply one massive organization.
-
@scottalanmiller said in Handling DNS in a Single Active Directory Domain Controller Environment:
I see where you are going, but this is something I disagree with. The problem here is that we assume having two (or more) AD servers will be done "correctly" and be set up to do what is needed. But then we assume that if we don't do that, that what is done will be done poorly. That's not a useful comparison mechanism.
This is exactly what I keep seeing every time.
-
@obsolesce said in Handling DNS in a Single Active Directory Domain Controller Environment:
@scottalanmiller said in Handling DNS in a Single Active Directory Domain Controller Environment:
I see where you are going, but this is something I disagree with. The problem here is that we assume having two (or more) AD servers will be done "correctly" and be set up to do what is needed. But then we assume that if we don't do that, that what is done will be done poorly. That's not a useful comparison mechanism.
This is exactly what I keep seeing every time.
You keep people using this apples and oranges point? Or you see people set it up incorrectly?
I see both, but to the latter, what I see is are shops that aren't making a conscious effort to choose what is right for them (dual AD, or single AD with appropriate setup, or no AD at all) but are just doing random things that happen to result in a single AD server. While yes, that is common, they don't apply to any shop that is actually trying to do their job well - which might result in any one of those scenarios.
-
@scottalanmiller said in Handling DNS in a Single Active Directory Domain Controller Environment:
@obsolesce said in Handling DNS in a Single Active Directory Domain Controller Environment:
@scottalanmiller said in Handling DNS in a Single Active Directory Domain Controller Environment:
I see where you are going, but this is something I disagree with. The problem here is that we assume having two (or more) AD servers will be done "correctly" and be set up to do what is needed. But then we assume that if we don't do that, that what is done will be done poorly. That's not a useful comparison mechanism.
This is exactly what I keep seeing every time.
You keep people using this apples and oranges point? Or you see people set it up incorrectly?
I see both, but to the latter, what I see is are shops that aren't making a conscious effort to choose what is right for them (dual AD, or single AD with appropriate setup, or no AD at all) but are just doing random things that happen to result in a single AD server. While yes, that is common, they don't apply to any shop that is actually trying to do their job well - which might result in any one of those scenarios.
By what I said I meant that I keep seeing people using that argument approach. Exactly as you described. Everything is set up perfectly with 2 DCs, but with 1 DC, everything is set up poorly and to fail.
-
@obsolesce said in Handling DNS in a Single Active Directory Domain Controller Environment:
@scottalanmiller said in Handling DNS in a Single Active Directory Domain Controller Environment:
@obsolesce said in Handling DNS in a Single Active Directory Domain Controller Environment:
@scottalanmiller said in Handling DNS in a Single Active Directory Domain Controller Environment:
I see where you are going, but this is something I disagree with. The problem here is that we assume having two (or more) AD servers will be done "correctly" and be set up to do what is needed. But then we assume that if we don't do that, that what is done will be done poorly. That's not a useful comparison mechanism.
This is exactly what I keep seeing every time.
You keep people using this apples and oranges point? Or you see people set it up incorrectly?
I see both, but to the latter, what I see is are shops that aren't making a conscious effort to choose what is right for them (dual AD, or single AD with appropriate setup, or no AD at all) but are just doing random things that happen to result in a single AD server. While yes, that is common, they don't apply to any shop that is actually trying to do their job well - which might result in any one of those scenarios.
By what I said I meant that I keep seeing people using that argument approach. Exactly as you described. Everything is set up perfectly with 2 DCs, but with 1 DC, everything is set up poorly and to fail.
Ah yes. It's a natural thing and not intentional. It feels logical - "I see this bad thing all the time, I have to assume that's normal." And it is normal, I think. But no matter how common it is, it doesn't apply.
Like the average person not going to college literally does nothing for four years. That's pretty common. But not people who are choosing between college and an alternative approach for career advancement. Two different pools of people.
-
It's actually a form of the Monty Hall Problem.
-
@scottalanmiller said in Handling DNS in a Single Active Directory Domain Controller Environment:
@obsolesce said in Handling DNS in a Single Active Directory Domain Controller Environment:
@scottalanmiller said in Handling DNS in a Single Active Directory Domain Controller Environment:
@obsolesce said in Handling DNS in a Single Active Directory Domain Controller Environment:
@scottalanmiller said in Handling DNS in a Single Active Directory Domain Controller Environment:
I see where you are going, but this is something I disagree with. The problem here is that we assume having two (or more) AD servers will be done "correctly" and be set up to do what is needed. But then we assume that if we don't do that, that what is done will be done poorly. That's not a useful comparison mechanism.
This is exactly what I keep seeing every time.
You keep people using this apples and oranges point? Or you see people set it up incorrectly?
I see both, but to the latter, what I see is are shops that aren't making a conscious effort to choose what is right for them (dual AD, or single AD with appropriate setup, or no AD at all) but are just doing random things that happen to result in a single AD server. While yes, that is common, they don't apply to any shop that is actually trying to do their job well - which might result in any one of those scenarios.
By what I said I meant that I keep seeing people using that argument approach. Exactly as you described. Everything is set up perfectly with 2 DCs, but with 1 DC, everything is set up poorly and to fail.
Ah yes. It's a natural thing and not intentional. It feels logical - "I see this bad thing all the time, I have to assume that's normal." And it is normal, I think. But no matter how common it is, it doesn't apply.
Like the average person not going to college literally does nothing for four years. That's pretty common. But not people who are choosing between college and an alternative approach for career advancement. Two different pools of people.
The problem with this college example is - people believe (or want to believe) that the college grad will come out of college and start where the person who has been in the workforce for 4 years is at now. Just to toss some titles into it... let's say day one person 1 starts college, and person two starts cleaning rooms. After 4 years person 1 is graduating from college and person two is a night manager. The assuming by many is that person 1 will instantly be able to become a night manager.
I'm not saying it's right or wrong - I know Scott has given an example where after 4 years he was a hotel manager (day time) and now the college grad is lucky if he can become a night manager, and not have to start by cleaning rooms - which in some cases they might.
