RAID10 Expansion/Reconstruction
-
I'm doing a RAID10 expansion on a Hyper-V hypervisor. After a whole day of reconstructing, it's only at 3%. Looking at the lights (barely any activity) and testing read/write speeds (they're speedy), I can tell it's barely working. I was under the impression rebuilds/reconstructions happen at 30% I/O on Dell PERCs. It doesn't seem to be doing even that. Is there a way to turn it up? At this rate, it'll take just over a month... I suppose that's okay, but man... if I don't have to wait that I'd rather not.
== BACKGROUND INFO ==
(all drives are enterprise drives)It consists of an SSD RAID 6x 800GB (for things like SQL servers and other servers that benefit from high I/O), and also a 500GB partition that serves as an SSD cache for the HDD RAID.
I bought 4x 960GB SSDs (they are better and cheaper than the 800s already in it). I used those to expand the existing SSD RAID, it went well and took less than a day. I went in the Hypervisor and expanded the volume to use the newly available space, all is well, speedy, happy.
The HDD RAID10 consists of 4x 8TB for the bulk of VM data. I bought an additional 2x 8TB drives to expand the HDD RAID10. Threw the drives in, selected the new drives and started the RAID10 expansion/reconstruction. The reconstruction is going well, considering...
I tried disabling the SSD Cache for the HDD RAID for almost a day, and that made no difference, so I re-enabled the Cache. 99% of writes are going to the SSD Cache and not the HDD RAID, about 60% reads are coming from the SSD Cache and not the HDD RAID.
I guess all I'm looking for is a way to increase the speed in which the RAID reconstruction works? Probably not possible from my searches turning up nothing, but I thought I'd ask here.
-
Honestly I had no idea that the PERC allowed expansion.
-
@jaredbusch said in RAID10 Expansion/Reconstruction:
Honestly I had no idea that the PERC allowed expansion.
Yeah, more specifically, on-line expansion. I should have specified that above.
This is being done through Dell's OMSA.
-
@jaredbusch said in RAID10 Expansion/Reconstruction:
Honestly I had no idea that the PERC allowed expansion.
Same here.
-
@tim_g said in RAID10 Expansion/Reconstruction:
@jaredbusch said in RAID10 Expansion/Reconstruction:
Honestly I had no idea that the PERC allowed expansion.
Yeah, more specifically, on-line expansion. I should have specified that above.
This is being done through Dell's OMSA.
Keep in mind that mirrored RAID expansion is not reconstruction. Very different things.
Only Parity RAID has a reconstruction/expansion mixed operation.
For mirrored RAID to grow, it has to move ALL data off of the original array, rebuild it, and put it all back. It's an insanely enormous operation and there is little operating space for the system to do that. It's actually a pretty amazing technical feat. I can come up with ways to do it, but they are incredibly intensive and require special hardware support. Doable, but there is a reason it is slow.
-
@scottalanmiller said in RAID10 Expansion/Reconstruction:
@tim_g said in RAID10 Expansion/Reconstruction:
@jaredbusch said in RAID10 Expansion/Reconstruction:
Honestly I had no idea that the PERC allowed expansion.
Yeah, more specifically, on-line expansion. I should have specified that above.
This is being done through Dell's OMSA.
Keep in mind that mirrored RAID expansion is not reconstruction. Very different things.
Only Parity RAID has a reconstruction/expansion mixed operation.
For mirrored RAID to grow, it has to move ALL data off of the original array, rebuild it, and put it all back. It's an insanely enormous operation and there is little operating space for the system to do that. It's actually a pretty amazing technical feat. I can come up with ways to do it, but they are incredibly intensive and require special hardware support. Doable, but there is a reason it is slow.
I see, that makes sense.
I keep using "reconstruction" because that's the "state" of the virtual disk:
-
@tim_g said in RAID10 Expansion/Reconstruction:
@scottalanmiller said in RAID10 Expansion/Reconstruction:
@tim_g said in RAID10 Expansion/Reconstruction:
@jaredbusch said in RAID10 Expansion/Reconstruction:
Honestly I had no idea that the PERC allowed expansion.
Yeah, more specifically, on-line expansion. I should have specified that above.
This is being done through Dell's OMSA.
Keep in mind that mirrored RAID expansion is not reconstruction. Very different things.
Only Parity RAID has a reconstruction/expansion mixed operation.
For mirrored RAID to grow, it has to move ALL data off of the original array, rebuild it, and put it all back. It's an insanely enormous operation and there is little operating space for the system to do that. It's actually a pretty amazing technical feat. I can come up with ways to do it, but they are incredibly intensive and require special hardware support. Doable, but there is a reason it is slow.
I see, that makes sense.
I keep using "reconstruction" because that's the "state" of the virtual disk:
Right, well it IS reconstructing. But the term means something different than it does with RAID 5 or 6. With RAID F reconstruction is literally "we told the array it was broken, that the parity wasn't parity but actual data and told it to restore parity to the new drive." Which is just a fancy way to say "We downgraded this RAID 6 to a RAID 5, added a disk and told it to recover."
With RAID 10, it's "we are reconstructing the data from scratch in a way that is totally unlike any RAID operation."
So one is a reconstruction of your RAID array, the other is inherent RAID reconstruction.
-
Has the percentage increased?
-
Still at 3%... does about 3% per day so far.
-
@tim_g said in RAID10 Expansion/Reconstruction:
Still at 3%... does about 3% per day so far.
That feels pretty slow
-
@tim_g said in RAID10 Expansion/Reconstruction:
Still at 3%... does about 3% per day so far.
Ouch. Good luck.
-
-
-
It's at 7% now!
-
-
@scottalanmiller said in RAID10 Expansion/Reconstruction:
@tim_g said in RAID10 Expansion/Reconstruction:
It's at 7% now!
Speeding right along.
Yeah, it's going nowhere fast!
-
At least it is moving a little.
-
Oh I must not have refreshed the page or something... it says 11% now.
-
Either that or something caused a bump in percentage... no idea.
-
@tim_g That's much better
