Replacing the Dead IPOD, SAN Bit the Dust
-
Thanks @scottalanmiller for helping me out with this predicament.
Current status of SAN. Firmware is as updated as it can go right now. I have 2 drives that are rebuilding from a RAID6 array. I have one more drive that is warning me about potential failure but not going to replace it until the other 2 are done rebuilding. The SAN is a Dell EqualLogics PS5000X. Firmware of the controllers are second to the latest firmware.
Host is a Dell PowerEdge R610 with the 86 GB of RAM and 16 vCPUs with VMware ESXi 6.0. This host currently supports 3 VM's, totaling at about 350 GB of production data. 2 of these VM's is on the local datastore of the host, but 1 VM is actually on that SAN that we need. It totals at 220 GB of data. There are no backups (my mistake).
We've tried flipflop failovers with the controllers and it only lasts us so long. Long enough to boot the VM backup but not enough time to actually backup the data. The backplane has been replaced. We've tried replacing controllers and all of the disks turned orange instead of green. We went back with the original controller and array began to operate normally again.
Dell support has advised us to allow for the array to continue rebuilding which was at 17%. Once done, I'm going to attempt to connect to it again and try to pull off the data. Support guy thought that we were overtaxing the SAN and basically freezing it up.
Besides retiring the thing, are there any pointers that I should consider in order to ensure that the backup or migration is a success?
-
I'd say that there are probably three key options for this as broad stroke approaches, each is valuable for its own reasons:
-
Mainframe: Just put disks in the local machines and do away with the clustering. The clustering added cost and risk without any actual benefits in the past. So why carry any of that forward. Just put disks into the local machines for the lowest cost, simplest solution. Points of failure are reduced, overall risk is reduced, bottlenecks are removed, flexibility is increased all for the lowest cost of investment. Costs nearly nothing, very effective, no downsides compared to the old solution. All positive movement.
-
Self Made Cluster: Replicated Local Disks and a hypervisor with high availability like is in place today. This is more costly and likely means some hardware upgrades to get the two hosts closer together, but at two hosts is very low cost and will provide dramatically more protection than the old approach.
-
Hyperconvergence: Do a full update moving to a totally hyperconverged product that provides complete support top to bottom. This is the most costly but replaces all hardware, gets inclusive support and requires the least internal IT effort.
-
-
@NerdyDad said in Replacing the Dead IPOD, SAN Bit the Dust:
I have 2 drives that are rebuilding from a RAID6 array. I have one more drive that is warning me about potential failure but not going to replace it until the other 2 are done rebuilding.
Oh no, that isn't good. Two lost controllers and two lost drives on RAID 6? What's the projected drive replacement time, a week at least, I would guess. It's almost better to not bother replacing the drives and just take a backup.
-
@NerdyDad said in Replacing the Dead IPOD, SAN Bit the Dust:
Support guy thought that we were overtaxing the SAN and basically freezing it up.
He is likely correct. That is generally expected with a RAID 6 rebuild, especially with two drives rebuilding at once.
-
@scottalanmiller I likely won't put in my last spare drive unless I absolutely have to. My main end goal is to somehow migrate the data and retire the SAN. It went from 0-17% in about 3 hours. I'm going to let it continue and hopefully it will be done in the morning. I will check on it once I get back to the office.
-
@scottalanmiller I'm looking at proposals right now for new storage refresh which is consisting of currently at least 6 TB of production data and looking at a price somewhere in the vicinity of $35k to $55k. How much more would I be looking at spending for Hyperconvergence for the same setup? I also have concerns about it of course, such as how is hyperconvergence better than the current hosts/storage setup if its all in one box? Wouldn't it by nature be the worse single point of failure? How about backing up outside of the box, to say a local NAS box or a private cloud storage?
-
@NerdyDad said in Replacing the Dead IPOD, SAN Bit the Dust:
@scottalanmiller I likely won't put in my last spare drive unless I absolutely have to. My main end goal is to somehow migrate the data and retire the SAN. It went from 0-17% in about 3 hours. I'm going to let it continue and hopefully it will be done in the morning. I will check on it once I get back to the office.
That's not so bad, moving along really well. How small are the drives?
-
@NerdyDad Yikes! That is a ton of dough to get you into the same predicament.
-
@NerdyDad said in Replacing the Dead IPOD, SAN Bit the Dust:
@scottalanmiller I'm looking at proposals right now for new storage refresh which is consisting of currently at least 6 TB of production data and looking at a price somewhere in the vicinity of $35k to $55k. How much more would I be looking at spending for Hyperconvergence for the same setup?
You'd be spending LESS! LOL
I don't actually know that, but easily the same amount.
-
@wrx7m said in Replacing the Dead IPOD, SAN Bit the Dust:
@NerdyDad Yikes! That is a ton of dough to get you into the same predicament.
This ^^^
-
@NerdyDad said in Replacing the Dead IPOD, SAN Bit the Dust:
I also have concerns about it of course, such as how is hyperconvergence better than the current hosts/storage setup if its all in one box?
Because it is NOT all in one box. All in one box is what you have now. Hyperconvergence is the opposite.
DId you want the video from MangoCon? I diagram the difference there.
-
@scottalanmiller 1 TB drives.
-
@NerdyDad said in Replacing the Dead IPOD, SAN Bit the Dust:
How about backing up outside of the box, to say a local NAS box or a private cloud storage?
Same as anything else. Local, cloud, both. Whatever works for your needs. I like a local Synology, ReadyNAS or Exablox device (or SAM-SD, of course.)
-
@NerdyDad said in Replacing the Dead IPOD, SAN Bit the Dust:
@scottalanmiller 1 TB drives.
Bigger than I would have guessed.
-
@scottalanmiller Not yet, but plan to.
-
I have 2 Dell R720XD each with 10x1TB NLSAS in OBR10 and 1 older Dell R710 with 4 10K SAS drives in OBR10 running ESXi 6 (all installed on redundant SD card or USB flash). They are backed up to a Synology DS1813+ with 8x4TB Segate Constellation drives in OBR10 and then backups are uploaded throughout the day to Amazon S3 and Glacier.
Back in mid 2013, the cost for the R720xd servers was $7229 a piece with 4-hour Pro Support. The cost for the Synology was $999 (diskless) and the disks came to $2196. The total was $3195 for a backup target.
2- Dell R720XD servers and one loaded Synology NAS came to about $17,653 (USD), which is half of the lowest end of what you are looking at for a SAN.
-
Scale's entry level high availability cluster starts at $25K. That might actually be enough here, but I doubt it. But it gives you an idea of where things start.
-
Tagging @scale and @scale_alex, I'm betting they can get you a way better system than what you have now for the same price or less than what you're talking about for a replacement cost.
If not that, something like what wrx7m mentioned.
-
@wrx7m said in Replacing the Dead IPOD, SAN Bit the Dust:
I have 2 Dell R720XD each with 10x1TB NLSAS in OBR10 and 1 older Dell R710 with 4 10K SAS drives in OBR10 running ESXi 6 (all installed on redundant SD card or USB flash).
Doing anything like Starwind between them?
-
@wrx7m We have a couple of Synology's around our enterprise and am currently using Veeam to backup VMs from their respective local hosts. But I would also have the same concern about the synology that I am also having with this current SAN. It will eventually be the bottom part of the pyramid.
