Domain Controller Down (VM)
-
@John-Nicholson said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
@DustinB3403 said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
If you're running on something using PV drivers that they don't understand...
Then your critical app vendor is below the home line. THAT'S how scary this should be to companies.
When your "business critical support" lacks the knowledge and skills of your first year help desk people, you need to be worried about their ability to support. Sure, when nothing goes wrong, everything is fine. But if anything goes wrong, you are suggesting these people don't have even the most rudimentary knowledge of systems today. That's worrisome. And it's why so many systems simply have no support options - relying on software and hardware that is out of support meaning that while the app might call itself supported, they depend on non-production systems making the whole thing out of support by extension.
So when running with a preallocated qcow2 image, which caching mode do you use for your disk? Writethrough, writeback, directsync, none?
What about IO mode? native, threads, default?
No one can support every hypervisor at that level.
Also, none of those things need to be supported by the app vendor. They just need to support the app and stop looking for meaningless excuses to block support. I understand some vendors want to support all the way down the stack, but if they don't know how to do that with virtualization, they don't know how to do it. The skills to support the stack would give them the skills to do it virtually even better (fewer variables.) So that logic doesn't hold up.
So they don't need to be fully supported, but let's say the IT guy down the street who's used Linux twice in his life installs the software in a VM with a non preallocated QCOW2 with an rtl8139 NIC. It's going to run slower than anything. So he calls the vendor for support and they try to help him. Nothing they are going to be able to tell him is going to help him, because it's nothing to do with their software. It's in their best interest to try to control what you're installing on to to mitigate stupid issues like that.
At least if the other end knew what he needed he could get some help. But now he might cancel his subscription and go somewhere else (which I believe is what they are trying to avoid). I can't imagine the amount of "IT Pros" that contact them looking for support for issues like that.
That is the issue of the IT Guy not understanding the system requirements, the fact that it is virtual means nothing. He could install that image to a bare metal system and have just as poor performance!
No, those are specific to a hypervisor. Bare metal would be much faster than that, you woudln't have those issues.
If bare metal was with a single ATA 66 drive, it might not be... Virtualization doesn't have a monopoly on stupid non-supported configurations.
Sorry, I meant bare metal using drivers with the OS. Not bad "try to support anything" drivers with the hypervisor.
This also assumed they were using real hardware.
-
@Dashrender said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@Dashrender said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@Dashrender said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
If you're running on something using PV drivers that they don't understand...
Then your critical app vendor is below the home line. THAT'S how scary this should be to companies.
When your "business critical support" lacks the knowledge and skills of your first year help desk people, you need to be worried about their ability to support. Sure, when nothing goes wrong, everything is fine. But if anything goes wrong, you are suggesting these people don't have even the most rudimentary knowledge of systems today. That's worrisome. And it's why so many systems simply have no support options - relying on software and hardware that is out of support meaning that while the app might call itself supported, they depend on non-production systems making the whole thing out of support by extension.
So when running with a preallocated qcow2 image, which caching mode do you use for your disk? Writethrough, writeback, directsync, none?
What about IO mode? native, threads, default?
No one can support every hypervisor at that level.
Also, none of those things need to be supported by the app vendor. They just need to support the app and stop looking for meaningless excuses to block support. I understand some vendors want to support all the way down the stack, but if they don't know how to do that with virtualization, they don't know how to do it. The skills to support the stack would give them the skills to do it virtually even better (fewer variables.) So that logic doesn't hold up.
You still haven't provided a single healthcare vendor that does any of what you say is appropriate.
I know Greenway didn't have a virtualization plan 3 years ago when we were looking at them. It's why I had to build a ridiculous $100K two server failover system. Today the performance needed could be done for $25k.
The sad thing is that the vendor could not provide any IOPs requirements, etc. They only had this generic hardware requirement.
SQL Dual Proc Xeon 4 cores each two drive boot, 4 drive RAID 10 SQL, 4 drive log
RDS single proc xeon 4 core 2 drive boot, 2 drive data
IIS application dual proc xeon 4 cores each, 2 drive boot, 6 drive RAID 10 data
etc
etcBecause... no support
eh? yeah Greenway didn't bother to do the right thing for their customers and have support for hypervisors! Shit, how can they really support their customers on bare metal if they don't know the IOPs requirements, etc? Just keep stabbing hardware until they "get lucky"?
That's my guess. Lacking support of VMs isn't exactly the big issue... it's WHY they lack that support that is the big issue.
LOL - Short of someone like Epic, from what I can tell, they are mostly software developers, who don't care about the hardware/VM it's running on. They don't approach the software holistically.
In healthcare you'll find a LOT that take this stance, for liability reasons (they want something they can provide support, or to reduce the chance of an SLA miss from something that their GSS isn't familiar with). Most healthcare systems are going hosted for this reason. I had a nice chat with the Cerner guys at VMworld and they mentioned that they offer SLA's all the way to how quick a patient note pulls up (7 seconds worst case I think). They in many cases actually take over on site support end to end (and act as a MSP in addition to an EMR). Realistically for EMR's given their horizontal integration of features, the next logical step is vertical integration of the hardware and end user computing support.
-
@stacksofplates said in Domain Controller Down (VM):
@DustinB3403 said in Domain Controller Down (VM):
Here is a physical system
Here is a virtual system.
Now tell me which system would you prefer to use if IOPS performance was an issue.
Ha. If you look at the timeline it's the same thing from the same dates (9-12:11 - 9-12:12). Good try.
This graph is also worthless as it doesn't show us latency or queue depth so we don't actually know if the app just doesn't do anything or if it has actual demands. This would be like me showing you how many RPM's I used on my car, and without any other context you don't know if I drove from Waco to Houston at 100MPH this weekend (I did) or if I just sat in a parking lot in neutral. RAWR IOPS GRAPH TIME!
-
@scottalanmiller said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
At least if the other end knew what he needed he could get some help. But now he might cancel his subscription and go somewhere else (which I believe is what they are trying to avoid). I can't imagine the amount of "IT Pros" that contact them looking for support for issues like that.
Same vein, how many avoid them because they don't provide ANY reasonable support options? I'm never asking anyone to support everything, but everyone needs to support something serious.
Right, and they do. VMware.
Oh okay, well that's fine then. Not the BEST option, but acceptable. And by BEST I don't mean that VMware is or isn't the best, I mean ONLY supporting that one is not as good as supported a few options.
Ya, this whole thing started because Dustin said he should drop them since they don't support anything else. That's ridiculous.
I see. Yeah that's going to far. That's lacking variety and options, but not lacking an enterprise deployment option. You have to figure the costs associated with VMware into the product's costs when decision making, but that's about it. VMware is very, very enterprise. It's a bit crappy that they don't offer ANY lower cost options for companies like this where VMware is way out of their league and crazy that they allow 100Mb/s Synology iSCSI but require VMware ESXi... so they have some clear problems in their thinking and requirements, but VMware itself is just fine.
This is another break down at the vendor end, most likely. The vendor probably only said - we only support ESXi as a hypervisor. Beyond that they probably don't say what server hardware they support/require, or the NICs or the Switches, or the SAN.
What they should be providing is minimum requirements in things like RAM and IOPs, then say - you must supply these, we really don't care how. Clearly if that had been done, it's likely that the synology SANs and the 100 Mb switches would have failed that test and other options would have had to be implemented.
-
@John-Nicholson said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
At least if the other end knew what he needed he could get some help. But now he might cancel his subscription and go somewhere else (which I believe is what they are trying to avoid). I can't imagine the amount of "IT Pros" that contact them looking for support for issues like that.
Same vein, how many avoid them because they don't provide ANY reasonable support options? I'm never asking anyone to support everything, but everyone needs to support something serious.
Right, and they do. VMware.
Oh okay, well that's fine then. Not the BEST option, but acceptable. And by BEST I don't mean that VMware is or isn't the best, I mean ONLY supporting that one is not as good as supported a few options.
Ya, this whole thing started because Dustin said he should drop them since they don't support anything else. That's ridiculous.
I see. Yeah that's going to far. That's lacking variety and options, but not lacking an enterprise deployment option. You have to figure the costs associated with VMware into the product's costs when decision making, but that's about it. VMware is very, very enterprise. It's a bit crappy that they don't offer ANY lower cost options for companies like this where VMware is way out of their league and crazy that they allow 100Mb/s Synology iSCSI but require VMware ESXi... so they have some clear problems in their thinking and requirements, but VMware itself is just fine.
To be clear, requiring VMware ESXi in a supported configuration is at odds with the 100Mb/s for vMotion and iSCSI (VMware does NOT support this abomination of a configuration).
I thought that I was stating that... that they had a mismatch, going for the biggest, baddest, most expensive enterprise hypervisor and then... don't care if it is set up in a viable way.
-
@scottalanmiller said in Domain Controller Down (VM):
@John-Nicholson said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
At least if the other end knew what he needed he could get some help. But now he might cancel his subscription and go somewhere else (which I believe is what they are trying to avoid). I can't imagine the amount of "IT Pros" that contact them looking for support for issues like that.
Same vein, how many avoid them because they don't provide ANY reasonable support options? I'm never asking anyone to support everything, but everyone needs to support something serious.
Right, and they do. VMware.
Oh okay, well that's fine then. Not the BEST option, but acceptable. And by BEST I don't mean that VMware is or isn't the best, I mean ONLY supporting that one is not as good as supported a few options.
Ya, this whole thing started because Dustin said he should drop them since they don't support anything else. That's ridiculous.
I see. Yeah that's going to far. That's lacking variety and options, but not lacking an enterprise deployment option. You have to figure the costs associated with VMware into the product's costs when decision making, but that's about it. VMware is very, very enterprise. It's a bit crappy that they don't offer ANY lower cost options for companies like this where VMware is way out of their league and crazy that they allow 100Mb/s Synology iSCSI but require VMware ESXi... so they have some clear problems in their thinking and requirements, but VMware itself is just fine.
To be clear, requiring VMware ESXi in a supported configuration is at odds with the 100Mb/s for vMotion and iSCSI (VMware does NOT support this abomination of a configuration).
I thought that I was stating that... that they had a mismatch, going for the biggest, baddest, most expensive enterprise hypervisor and then... don't care if it is set up in a viable way.
To be clear, he has Essentials Plus which is only 6K up front and $1200 a year for 24/7 support and free upgrades. This is the CHEAPEST hypervisor from an ongoing support for 24/7 support of 6 sockets, and a central management and monitoring solution. (Citrix for XenServer, and Red Hat cost more. Microsoft crazy more for SCCM-VMM and a support agreement).
-
@John-Nicholson said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@John-Nicholson said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
At least if the other end knew what he needed he could get some help. But now he might cancel his subscription and go somewhere else (which I believe is what they are trying to avoid). I can't imagine the amount of "IT Pros" that contact them looking for support for issues like that.
Same vein, how many avoid them because they don't provide ANY reasonable support options? I'm never asking anyone to support everything, but everyone needs to support something serious.
Right, and they do. VMware.
Oh okay, well that's fine then. Not the BEST option, but acceptable. And by BEST I don't mean that VMware is or isn't the best, I mean ONLY supporting that one is not as good as supported a few options.
Ya, this whole thing started because Dustin said he should drop them since they don't support anything else. That's ridiculous.
I see. Yeah that's going to far. That's lacking variety and options, but not lacking an enterprise deployment option. You have to figure the costs associated with VMware into the product's costs when decision making, but that's about it. VMware is very, very enterprise. It's a bit crappy that they don't offer ANY lower cost options for companies like this where VMware is way out of their league and crazy that they allow 100Mb/s Synology iSCSI but require VMware ESXi... so they have some clear problems in their thinking and requirements, but VMware itself is just fine.
To be clear, requiring VMware ESXi in a supported configuration is at odds with the 100Mb/s for vMotion and iSCSI (VMware does NOT support this abomination of a configuration).
I thought that I was stating that... that they had a mismatch, going for the biggest, baddest, most expensive enterprise hypervisor and then... don't care if it is set up in a viable way.
To be clear, he has Essentials Plus which is only 6K up front and $1200 a year for 24/7 support and free upgrades. This is the CHEAPEST hypervisor from an ongoing support for 24/7 support of 6 sockets, and a central management and monitoring solution. (Citrix for XenServer, and Red Hat cost more. Microsoft crazy more for SCCM-VMM and a support agreement).
I guess the difference there, at least with MS, is that you don't expect to get your expert support from MS directly, instead you get it from companies like NTG or those who know it.
But the bigger fail is - did they really need Essentials Plus in the first place? Could they afford near zero downtime? seems unlikely, unless they are a location that's open 24/7.
-
@Dashrender said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
At least if the other end knew what he needed he could get some help. But now he might cancel his subscription and go somewhere else (which I believe is what they are trying to avoid). I can't imagine the amount of "IT Pros" that contact them looking for support for issues like that.
Same vein, how many avoid them because they don't provide ANY reasonable support options? I'm never asking anyone to support everything, but everyone needs to support something serious.
Right, and they do. VMware.
Oh okay, well that's fine then. Not the BEST option, but acceptable. And by BEST I don't mean that VMware is or isn't the best, I mean ONLY supporting that one is not as good as supported a few options.
Ya, this whole thing started because Dustin said he should drop them since they don't support anything else. That's ridiculous.
I see. Yeah that's going to far. That's lacking variety and options, but not lacking an enterprise deployment option. You have to figure the costs associated with VMware into the product's costs when decision making, but that's about it. VMware is very, very enterprise. It's a bit crappy that they don't offer ANY lower cost options for companies like this where VMware is way out of their league and crazy that they allow 100Mb/s Synology iSCSI but require VMware ESXi... so they have some clear problems in their thinking and requirements, but VMware itself is just fine.
This is another break down at the vendor end, most likely. The vendor probably only said - we only support ESXi as a hypervisor. Beyond that they probably don't say what server hardware they support/require, or the NICs or the Switches, or the SAN.
What they should be providing is minimum requirements in things like RAM and IOPs, then say - you must supply these, we really don't care how. Clearly if that had been done, it's likely that the synology SANs and the 100 Mb switches would have failed that test and other options would have had to be implemented.
To be fair, its implied that you have at least GigE for iSCSI/NFS/vMotion. The Implementor had to have been either an idiot or greedy to deploy this. I got asked one time to do something like this and I just walked out and told sales to refund their money when they refused to get real gear. I couldn't risk my company name and professional reputation being attached to such clown car stuff.
Even personally (working in house IT) you have to put your foot down at some point, because otherwise the users will talk about how shitty your IT is, and it will impact your ability to get a job elsewhere when others hear about all the outage and performance issues. Even if you can tell in an interview why it was that bad, no one wants to hire someone who worked in a clown car for 3-5 years.
-
@Dashrender said in Domain Controller Down (VM):
unless they are a location that's open 24/7.
ya they are
-
@Dashrender said in Domain Controller Down (VM):
@John-Nicholson said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@John-Nicholson said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
@scottalanmiller said in Domain Controller Down (VM):
@stacksofplates said in Domain Controller Down (VM):
At least if the other end knew what he needed he could get some help. But now he might cancel his subscription and go somewhere else (which I believe is what they are trying to avoid). I can't imagine the amount of "IT Pros" that contact them looking for support for issues like that.
Same vein, how many avoid them because they don't provide ANY reasonable support options? I'm never asking anyone to support everything, but everyone needs to support something serious.
Right, and they do. VMware.
Oh okay, well that's fine then. Not the BEST option, but acceptable. And by BEST I don't mean that VMware is or isn't the best, I mean ONLY supporting that one is not as good as supported a few options.
Ya, this whole thing started because Dustin said he should drop them since they don't support anything else. That's ridiculous.
I see. Yeah that's going to far. That's lacking variety and options, but not lacking an enterprise deployment option. You have to figure the costs associated with VMware into the product's costs when decision making, but that's about it. VMware is very, very enterprise. It's a bit crappy that they don't offer ANY lower cost options for companies like this where VMware is way out of their league and crazy that they allow 100Mb/s Synology iSCSI but require VMware ESXi... so they have some clear problems in their thinking and requirements, but VMware itself is just fine.
To be clear, requiring VMware ESXi in a supported configuration is at odds with the 100Mb/s for vMotion and iSCSI (VMware does NOT support this abomination of a configuration).
I thought that I was stating that... that they had a mismatch, going for the biggest, baddest, most expensive enterprise hypervisor and then... don't care if it is set up in a viable way.
To be clear, he has Essentials Plus which is only 6K up front and $1200 a year for 24/7 support and free upgrades. This is the CHEAPEST hypervisor from an ongoing support for 24/7 support of 6 sockets, and a central management and monitoring solution. (Citrix for XenServer, and Red Hat cost more. Microsoft crazy more for SCCM-VMM and a support agreement).
I guess the difference there, at least with MS, is that you don't expect to get your expert support from MS directly, instead you get it from companies like NTG or those who know it.
But the bigger fail is - did they really need Essentials Plus in the first place? Could they afford near zero downtime? seems unlikely, unless they are a location that's open 24/7.
I can't disagree with you more.
Its a medical facility that has beds occupied 24/7 so yes. There is a bizarre assumption (That I used to be guilty to) that because you are small you don't need 24/7 availability and that is just changing. An increasing number of SMB's operate 3 shifts, or have customer expectations of availability 24/7. Its true you can have maintenance windows and things, and maybe we should blame google for it, but the game has changed. More mission critical systems have gone from pen and paper to the computers.
Spending 6K so you can get 24/7 support (That's not even an option of Essentials) is less on a per daily basis than my wife's star buck's addiction. That's not a big fail and nothing anyone should be shamed over especially one with no training and no backup (That's a bigger fail, but not a replacement for vendor support).
MSPs are NOT a replacement for a support agreement (In fact most REQUIRE you have them). If there is a driver issue someone has to stay on the phone and deal with it. Most MSP's worth a damn are going to charge you for 24/7 support of a hypervisor ~$150-250 per host. So the support costs for his 3 hosts from the MSP would actually be more even if you went Free Hyper-V. Given the MSP would need to manage patching, the costs for overtime to force it being done after hours disruptively would likely negate any savings from going local storage with no vMotion for patching.
I advocate having both. In house steady state IT should NOT be running outage's by them self's without the opportunity for a shift change. Also MSP's see every kind of outage and know how to isolate and react to them. In this case any normal MSP would have...
- Never agree'd to support this environment. They wouldn't have signed a contract after the discovery until this storage/networking mess was fixed.
- Mandated support remote monitoring (SNMP/Syslog) of the switch and detected the fault and isolated it. This would have cut the outage to a 1/3 of its length.
- If a HA cluster was deployed used, 2 switches would have been deployed so only a single one would have failed (no outage).
- Would be regularly patching the environment so he was on a mainstream supported release of vSphere.
- Would have demanded a replacement of the Synology.
- Would be actively managing proper backups (and not using an ancient version of ArcServe).
- Been on the phone handling the issue, handling updates to management to keep them out of the way, and bringing in specialists as needed (networking, storage, hypervisor) as well as used their partner relationships with the vendors involved (Cisco, HP, VMware) to get escalated tickets opened and tracked as needed.
A proper MSP is like having an enterprise support army in your back pocket for less than the cost of a FTE. Honestly as a SMB you shouldn't hire an in house resource before you hire a MSP first, and any shop that doesn't want to pay for a MSP but will pay for a FTE is a GIANT red flag that they lack any level of competence in IT governance, budgeting, or common sense.
-
I didn't know what kind of medical facility @wirestyle22 was in..
OK since the place is 24/7, he needs a higher than normal amount of uptime - fine. But real HA? really? I know XenServer and Hyper-V can both do storage motion while the system is running, so no shared storage is needed (granted XS is super slow, sooooo) so you don't need HA to do patches, you just need the storage motion options - I don't know if that's available in ESXi Essentials or not.
If HA is fully thought out and is felt is needed (don't forget about the power situation, and cooling, etc, etc, etc, - remember HA isn't a product, it's a process) then they should fully realize it. I'm guessing by the fact that the switches were 100 Mb that it really wasn't fully thought out, instead someone in the place of authority thought it sounded good and they tossed what they have in today in.
As for the rest, I generally agree with you. It shows the real costs of DOING IT RIGHT - but as most of us know - few SMBs are really willing to do what's right in IT.
Hell, just look at all of the threads in SW talking about print shops that couldn't upgrade their XP machines because their 10K+ printers didn't support anything newer. it's a never ending problem of knowing the real costs of doing something right.
-
@Dashrender said in Domain Controller Down (VM):
OK since the place is 24/7, he needs a higher than normal amount of uptime - fine. But real HA? really? I know XenServer and Hyper-V can both do storage motion while the system is running, so no shared storage is needed (granted XS is super slow, sooooo) so you don't need HA to do patches, you just need the storage motion options - I don't know if that's available in ESXi Essentials or not.
I can't disagree more. I've seen someone try to do this in a SMB and they got fired.
It is available in ESXi (its a bit faster in 5.5 ESXi has a proper IO mirror driver so you don't have helper snapshots in a never ending catch up process to handle the IO happening during the merge of snapshots).- Doing shared nothing migrations impacts performance (Seriously, look at the disk latency the next time you do it. Telling management "well we kicked off the migration 7 hours ago and we can't really stop it" is a great way to get shown the door.
- This doesn't scale, and can make patch windows take DAYS very quickly. No one would seriously consider this for monthly patching.
- If you have high enough IO and are using a hypervisor that lacks a mirror driver you end up with an never ending amount of snapshot merges.
-
@Dashrender said in Domain Controller Down (VM):
OK since the place is 24/7, he needs a higher than normal amount of uptime - fine. But real HA? really? I know XenServer and Hyper-V can both do storage motion while the system is running, so no shared storage is needed (granted XS is super slow, sooooo) so you don't need HA to do patches, you just need the storage motion options - I don't know if that's available in ESXi Essentials or not.
Storage motion is not for production hours. That's great if you have a greenzone, but if you have that you don't need the storage motion. Storage motion is mostly for migrations and one time, unavailable events. It's not something you do during production time unless you have no choice (dying storage system.)
-
@Dashrender said in Domain Controller Down (VM):
If HA is fully thought out and is felt is needed (don't forget about the power situation, and cooling, etc, etc, etc, - remember HA isn't a product, it's a process) then they should fully realize it. I'm guessing by the fact that the switches were 100 Mb that it really wasn't fully thought out, instead someone in the place of authority thought it sounded good and they tossed what they have in today in.
It's as simple as "there was no HA and no attempt made at it."
-
@Dashrender said in Domain Controller Down (VM):
I didn't know what kind of medical facility @wirestyle22 was in..
If HA is fully thought out and is felt is needed (don't forget about the power situation, and cooling, etc, etc, etc, - remember HA isn't a product, it's a process) then they should fully realize it. I'm guessing by the fact that the switches were 100 Mb that it really wasn't fully thought out, instead someone in the place of authority thought it sounded good and they tossed what they have in today in.
Medical facilities with beds have generators and fuel. HVAC for something this small can be covered for redundancy with a spot cooler (I have this in my own house for my lab, so If I can afford it, you have to be a tiny outfit to not be able to afford it). I agree its a process, and the biggest piece is having a MSP to back you up, and having 24/7 dispatched resources to help you with the persistent layer. Not having redundancy at the people level is the biggest issue to address. While I normally advocate some kind of offsite ready to fire off DR, in the case of a facility like this its not actually as important (beyond BC reasons) because if the whole facility blows up the need for the system goes with it. Still there are a bazillion Veeam/VCAN partners who can cover this piece for cheap so why not.
-
@scottalanmiller said in Domain Controller Down (VM):
@Dashrender said in Domain Controller Down (VM):
If HA is fully thought out and is felt is needed (don't forget about the power situation, and cooling, etc, etc, etc, - remember HA isn't a product, it's a process) then they should fully realize it. I'm guessing by the fact that the switches were 100 Mb that it really wasn't fully thought out, instead someone in the place of authority thought it sounded good and they tossed what they have in today in.
It's as simple as "there was no HA and no attempt made at it."
It would take me about 5 minutes to explain to a 3rd grader why the system he has isn't redundant is bad. The fact that it continues to exist shows that either...
- Management has the intellectual capacity below a 3rd grader (possible)
- No one in non-jargon english explained how bad this configuration was. (more likely).
-
@John-Nicholson said in Domain Controller Down (VM):
Its a medical facility that has beds occupied 24/7 so yes.
That doesn't mean that. We can equally say they didn't have 24x7 IT staff so they don't need it. What they need, we have no way of knowing. If we read back what we know about their environment, it tells us that they didn't think that they needed HA in any way whatsoever. But that's all we have to go on. They operate around the clock, but that isn't an HA concern. And they implemented something so far from HA that it is laughable. So all we know is that they implemented anti-HA and spent a lot to do it. That's it. We have no indication that HA is warranted in any way.
Just because a shop is 24x7 medical doesn't tell us that a specific system is needed 24x7 or that it needs to be available at all times. Those are very different requirements.
-
@Dashrender said in Domain Controller Down (VM):
As for the rest, I generally agree with you. It shows the real costs of DOING IT RIGHT - but as most of us know - few SMBs are really willing to do what's right in IT.
Hell, just look at all of the threads in SW talking about print shops that couldn't upgrade their XP machines because their 10K+ printers didn't support anything newer. it's a never ending problem of knowing the real costs of doing something right.The real cost of doing IT right is cheaper. Simply not having an onsite FTE, and having a MSP manage this stuff is likely cheaper (FTE's are expensive!). This outage might have been embarrassing enough for them to loose a patient or two (or worse someone die, and they get hit with a million wrongful death dollar lawsuit that spikes their premiums). Doing IT RIGHT includes understanding the capex and opex costs, and associated risks and external costs of doing IT right or wrong.
Doing IT Wrong means wasting tons of money and getting an output that causes other costs. IT budgets do NOT exist in a vacuum to the rest of the operations and their output (Especially in 2016!).
-
@John-Nicholson said in Domain Controller Down (VM):
A proper MSP is like having an enterprise support army in your back pocket for less than the cost of a FTE. Honestly as a SMB you shouldn't hire an in house resource before you hire a MSP first, and any shop that doesn't want to pay for a MSP but will pay for a FTE is a GIANT red flag that they lack any level of competence in IT governance, budgeting, or common sense.
I agree. Anyone going into an FTE role in an SMB should probably ask what their MSP ecosystem of support is like BEFORE accepting a position. That's something that we never talk about but is a great idea. They should either have a great answer (and the MSP should be likely part of the interview process) or they should be like "that's why we are bringing you in, to help us find those good resources."
-
@John-Nicholson said in Domain Controller Down (VM):
@Dashrender said in Domain Controller Down (VM):
As for the rest, I generally agree with you. It shows the real costs of DOING IT RIGHT - but as most of us know - few SMBs are really willing to do what's right in IT.
Hell, just look at all of the threads in SW talking about print shops that couldn't upgrade their XP machines because their 10K+ printers didn't support anything newer. it's a never ending problem of knowing the real costs of doing something right.The real cost of doing IT right is cheaper. Simply not having an onsite FTE, and having a MSP manage this stuff is likely cheaper (FTE's are expensive!). This outage might have been embarrassing enough for them to loose a patient or two (or worse someone die, and they get hit with a million wrongful death dollar lawsuit that spikes their premiums). Doing IT RIGHT includes understanding the capex and opex costs, and associated risks and external costs of doing IT right or wrong.
Doing IT Wrong means wasting tons of money and getting an output that causes other costs. IT budgets do NOT exist in a vacuum to the rest of the operations and their output (Especially in 2016!).
"IT Right" isn't even a thing. IT is just part of the business. It's "running the business right."
