Xen Server 6.5 + Xen Orchestra w. HA & SAN
-
More great info here, Thank you @scottalanmiller
-
Also... one of the best quotes ever in IT from a senior architect at VMware.... "HA is something that you do, not something that you buy."
That's great and you are approaching it well with not looking at just "buying your way out of it." but have you considered that....
The first thing for getting HA is moving your servers to a Tier IV datacenter? Your power supply and HVAC are the most critical components of your HA strategy. To look at HA you need things like dual generators, HA HVAC systems, dualing UPS systems that can be failover over, dual rail power supply, etc.
You get more uptime moving a single server to a good datacenter than you do putting HA servers on premises. The facilities matter a lot.
That's why we generally see six nines from our standard servers. Six nines!!!
-
Also, somewhat obvious things that often get lost when talking to management...
HA does not cover application errors, data corruption and the like. Those things replicate through the HA. Windows patching issues, downtime for upgrades... HA won't help with those and might make them more complicated.
In my experience, most IT outages are not addressed by HA. A big percentage are, maybe 30%, but only 30%. The other 70% you have to mitigate some other way.
And hardware issues, that HA protects against mostly, are the easier to remedy. How much does a spare part cost or does a Dell 6 hours to repair warranty cost?
-
I try to plan the environment similar to datacenter setup with N+1, although cant be the case without serious $$.
But yes, after your great detail and points of seeing the bigger picture. I've also now considered what it would cost to have a cold spare back-up server. Or just simply the 5-year NBD or as you said 6-hour repair warranty.
it all comes full circle as to what downtime costs the company and to what is worth to mitigate the downtime.
-
@ntoxicator said:
But yes, after your great detail and points of seeing the bigger picture. I've also now considered what it would cost to have a cold spare back-up server. Or just simply the 5-year NBD or as you said 6-hour repair warranty.
And, of course, look at lower cost providers like xByte who take a lot of the cost off of getting good gear. It lets you get better warranties and better equipment for a lower price. Better equipment means longer MTBF.
-
I have reached out to them recently
-
@ntoxicator said:
I have reached out to them recently
How quickly they can get you a replacement machine is a factor, too.
For us, our logistics partners can often get us a server in two hours. A full server. So the need for spare parts goes way down.
-
That's 'jimmy johns' fast. Nice!
-
-
@ntoxicator said:
Ok, why dont you come consult for us then? Explain why HA is not needed and list the negatives and upside.
I dont get why your so anti-HA?
So we get another single server, spec'd full of drives and hope that we dont have a hardware failure
What are chances of mobo dying on Dell R730? or integrated NIC card failing, etc? I suppose low percentage rate.
Of the approximately 100 servers that I personally have supported in the past 15 years I've seen exactly one motherboard failure, and zero RAID card failures. I've seen probably 5 Power Supply failures and perhaps 20-30 drives fail. This is a pretty low number of servers, but gives us easy things to make percentages out of.
So
motherboard failures = 1% over 15 years
RAID controllers = 0%
Power Supplies = 5%
drives - I can't give a real number here because I have no idea how many actual drives were on all these servers over the years. Assuming a minimum of 4, for a total of 400 drives (this is probably low) we'd be talking about a 4-5% failure rate.So looking at those numbers (as much BS as they really are) we can already see what we make certain parts redundant, and others not. Power Supplies and Drives fail often, at around 5% so we've setup multiple Power Supply systems and drives we've created RAID to keep the systems running in case of a failure.
But Mobo's and RAID cards fail so infrequently that we don't worry about it.
-
@Dashrender said:
I've seen probably 5 Power Supply failures and perhaps 20-30 drives fail.
And those are hot swap in any enterprise class server, even entry level. So they don't result in down time.
-
@scottalanmiller said:
@Dashrender said:
I've seen probably 5 Power Supply failures and perhaps 20-30 drives fail.
And those are hot swap in any enterprise class server, even entry level. So they don't result in down time.
Until they blow out the other drives!
-
@Dashrender said:
But Mobo's and RAID cards fail so infrequently that we don't worry about it.
High end server make those redundant too. they are the least likely to go bad and teh most expensive to make redundant so that is why they are avoided. But an Integrity, Oracle M or IBM i or z will all do redundant there.
-
@scottalanmiller said:
@ntoxicator tell them that without those numbers you have to assume that the losses would be minimal because of they were significant they would know how important it was for you to have them.
Instil in them that their actions are informing you where their words are not.
This is a hard thing to tell upper management without being fully prepared to be fired.
Sadly this feeling also proves that most businesses, even ones that appear successful, are really run poorly.
-
@scottalanmiller said:
You get more uptime moving a single server to a good datacenter than you do putting HA servers on premises. The facilities matter a lot.
That's why we generally see six nines from our standard servers. Six nines!!!
The problem with this is the cost of the Colo and the high speed internet to your main location are sometimes cost prohibitive, But that too also goes to Scott's point that HA isn't needed.
-
I didn't see it mentioned, if it was ignore this. Another way to look at it is the cost of the solution would directly be related to the amount of time saved by an HA solution. So for example:
You spend x number of dollars on an HA solution to fail over in 10 seconds. That means you would have to make x number of dollars every 10 seconds for it to be worth the cost.
-
For HA you would probably want 3 nodes also. If you are working on one node doing maintenance and it's offline, and the second node goes down you're out of luck.
-
@johnhooks said:
You spend x number of dollars on an HA solution to fail over in 10 seconds. That means you would have to make x number of dollars every 10 seconds for it to be worth the cost.
I think that you missed a number....
It would cost X to not have HA.
It would cost Y to have HA.
The downtime of X is Z
The downtime of Y is W
So the cost of HA is Y - X and the time to make up with Z - W.
So if R = Z - W and S = Y - X, then the cost S has to be justified in R downtime mitigated.
-
Eh I thought I read somewhere that you said something like that. I must have just misread one of your $1000 per minute posts.
-
@johnhooks said:
Eh I thought I read somewhere that you said something like that. I must have just misread one of your $1000 per minute posts.
Maybe it was the ten minute number. HA versus secondary server is a difference of normally about ten minutes of downtime.
