RAID10 - Two Drive Failure
-
Some information I am reading says I need to take the drive "offline" first, is this true?
PowerEdge R720.
-
@aaronstuder What raid controller do you have?
-
I'm assuming @JaredBusch's qualifier of "Predictive failure is not failure" means that the could possibly change if this were a failed drive?
-
@gjacobse said in RAID10 - Two Drive Failure:
@wirestyle22 said in RAID10 - Two Drive Failure:
@gjacobse said in RAID10 - Two Drive Failure:
@JaredBusch said in RAID10 - Two Drive Failure:
Predictive failure is not failure. Replace one at a time. to give the RAID card the most power to work on the individual resilver.
in my experience - you
neverreplace more than one drive at a time...Ask me how I know.
That's very interesting. I have not really had to deal with drive failures actually.
I haven't kept up with the number of drive failures I have had over nearly 30 years. In all of my personal systems, I think I have had
one.Work related,.. maybe all of three.
Wow, that's pretty small.
Personally, I've probably lost 3-4 drives. In businesses - well over 10.
And Scott has probably seen hundreds fail. Of course it all boils down to how many systems you see/support.
-
@Dashrender said in RAID10 - Two Drive Failure:
@gjacobse said in RAID10 - Two Drive Failure:
@wirestyle22 said in RAID10 - Two Drive Failure:
@gjacobse said in RAID10 - Two Drive Failure:
@JaredBusch said in RAID10 - Two Drive Failure:
Predictive failure is not failure. Replace one at a time. to give the RAID card the most power to work on the individual resilver.
in my experience - you
neverreplace more than one drive at a time...Ask me how I know.
That's very interesting. I have not really had to deal with drive failures actually.
I haven't kept up with the number of drive failures I have had over nearly 30 years. In all of my personal systems, I think I have had
one.Work related,.. maybe all of three.
Wow, that's pretty small.
Personally, I've probably lost 3-4 drives. In businesses - well over 10.
And Scott has probably seen hundreds fail. Of course it all boils down to how many systems you see/support.
Your footprint is pretty small though so I think that's a pretty big number for you
-
Predictive failure is usually a report from SMART about the drives. It means that something isn't kosher.
-
@Dashrender said in RAID10 - Two Drive Failure:
@gjacobse said in RAID10 - Two Drive Failure:
@wirestyle22 said in RAID10 - Two Drive Failure:
@gjacobse said in RAID10 - Two Drive Failure:
@JaredBusch said in RAID10 - Two Drive Failure:
Predictive failure is not failure. Replace one at a time. to give the RAID card the most power to work on the individual resilver.
in my experience - you
neverreplace more than one drive at a time...Ask me how I know.
That's very interesting. I have not really had to deal with drive failures actually.
I haven't kept up with the number of drive failures I have had over nearly 30 years. In all of my personal systems, I think I have had
one.Work related,.. maybe all of three.
Wow, that's pretty small.
Personally, I've probably lost 3-4 drives. In businesses - well over 10.
And Scott has probably seen hundreds fail. Of course it all boils down to how many systems you see/support.
We have one or two fail every two-three months. Nothing crazy.
-
@wirestyle22 said in RAID10 - Two Drive Failure:
@Dashrender said in RAID10 - Two Drive Failure:
@gjacobse said in RAID10 - Two Drive Failure:
@wirestyle22 said in RAID10 - Two Drive Failure:
@gjacobse said in RAID10 - Two Drive Failure:
@JaredBusch said in RAID10 - Two Drive Failure:
Predictive failure is not failure. Replace one at a time. to give the RAID card the most power to work on the individual resilver.
in my experience - you
neverreplace more than one drive at a time...Ask me how I know.
That's very interesting. I have not really had to deal with drive failures actually.
I haven't kept up with the number of drive failures I have had over nearly 30 years. In all of my personal systems, I think I have had
one.Work related,.. maybe all of three.
Wow, that's pretty small.
Personally, I've probably lost 3-4 drives. In businesses - well over 10.
And Scott has probably seen hundreds fail. Of course it all boils down to how many systems you see/support.
Your footprint is pretty small though so I think that's a pretty big number for you
Remember, I used to be a consultant like JB - so I have a larger exposure than a single man SMB person.
-
@Dashrender said in RAID10 - Two Drive Failure:
@wirestyle22 said in RAID10 - Two Drive Failure:
@Dashrender said in RAID10 - Two Drive Failure:
@gjacobse said in RAID10 - Two Drive Failure:
@wirestyle22 said in RAID10 - Two Drive Failure:
@gjacobse said in RAID10 - Two Drive Failure:
@JaredBusch said in RAID10 - Two Drive Failure:
Predictive failure is not failure. Replace one at a time. to give the RAID card the most power to work on the individual resilver.
in my experience - you
neverreplace more than one drive at a time...Ask me how I know.
That's very interesting. I have not really had to deal with drive failures actually.
I haven't kept up with the number of drive failures I have had over nearly 30 years. In all of my personal systems, I think I have had
one.Work related,.. maybe all of three.
Wow, that's pretty small.
Personally, I've probably lost 3-4 drives. In businesses - well over 10.
And Scott has probably seen hundreds fail. Of course it all boils down to how many systems you see/support.
Your footprint is pretty small though so I think that's a pretty big number for you
Remember, I used to be a consultant like JB - so I have a larger exposure than a single man SMB person.
Yeah, I was thinking where you currently work. My bad
-
@coliver said in RAID10 - Two Drive Failure:
@Dashrender said in RAID10 - Two Drive Failure:
@gjacobse said in RAID10 - Two Drive Failure:
@wirestyle22 said in RAID10 - Two Drive Failure:
@gjacobse said in RAID10 - Two Drive Failure:
@JaredBusch said in RAID10 - Two Drive Failure:
Predictive failure is not failure. Replace one at a time. to give the RAID card the most power to work on the individual resilver.
in my experience - you
neverreplace more than one drive at a time...Ask me how I know.
That's very interesting. I have not really had to deal with drive failures actually.
I haven't kept up with the number of drive failures I have had over nearly 30 years. In all of my personal systems, I think I have had
one.Work related,.. maybe all of three.
Wow, that's pretty small.
Personally, I've probably lost 3-4 drives. In businesses - well over 10.
And Scott has probably seen hundreds fail. Of course it all boils down to how many systems you see/support.
We have one or two fail every two-three months. Nothing crazy.
How many drives do you have?
-
@Dashrender said in RAID10 - Two Drive Failure:
@coliver said in RAID10 - Two Drive Failure:
@Dashrender said in RAID10 - Two Drive Failure:
@gjacobse said in RAID10 - Two Drive Failure:
@wirestyle22 said in RAID10 - Two Drive Failure:
@gjacobse said in RAID10 - Two Drive Failure:
@JaredBusch said in RAID10 - Two Drive Failure:
Predictive failure is not failure. Replace one at a time. to give the RAID card the most power to work on the individual resilver.
in my experience - you
neverreplace more than one drive at a time...Ask me how I know.
That's very interesting. I have not really had to deal with drive failures actually.
I haven't kept up with the number of drive failures I have had over nearly 30 years. In all of my personal systems, I think I have had
one.Work related,.. maybe all of three.
Wow, that's pretty small.
Personally, I've probably lost 3-4 drives. In businesses - well over 10.
And Scott has probably seen hundreds fail. Of course it all boils down to how many systems you see/support.
We have one or two fail every two-three months. Nothing crazy.
How many drives do you have?
A few hundred for now. Should be under 100 at the end of summer.
-
@coliver said in RAID10 - Two Drive Failure:
@Dashrender said in RAID10 - Two Drive Failure:
@coliver said in RAID10 - Two Drive Failure:
@Dashrender said in RAID10 - Two Drive Failure:
@gjacobse said in RAID10 - Two Drive Failure:
@wirestyle22 said in RAID10 - Two Drive Failure:
@gjacobse said in RAID10 - Two Drive Failure:
@JaredBusch said in RAID10 - Two Drive Failure:
Predictive failure is not failure. Replace one at a time. to give the RAID card the most power to work on the individual resilver.
in my experience - you
neverreplace more than one drive at a time...Ask me how I know.
That's very interesting. I have not really had to deal with drive failures actually.
I haven't kept up with the number of drive failures I have had over nearly 30 years. In all of my personal systems, I think I have had
one.Work related,.. maybe all of three.
Wow, that's pretty small.
Personally, I've probably lost 3-4 drives. In businesses - well over 10.
And Scott has probably seen hundreds fail. Of course it all boils down to how many systems you see/support.
We have one or two fail every two-three months. Nothing crazy.
How many drives do you have?
A few hundred for now. Should be under 100 at the end of summer.
What are you guys changing to reduce that number by that much?
-
@wirestyle22 said in RAID10 - Two Drive Failure:
@coliver said in RAID10 - Two Drive Failure:
@Dashrender said in RAID10 - Two Drive Failure:
@coliver said in RAID10 - Two Drive Failure:
@Dashrender said in RAID10 - Two Drive Failure:
@gjacobse said in RAID10 - Two Drive Failure:
@wirestyle22 said in RAID10 - Two Drive Failure:
@gjacobse said in RAID10 - Two Drive Failure:
@JaredBusch said in RAID10 - Two Drive Failure:
Predictive failure is not failure. Replace one at a time. to give the RAID card the most power to work on the individual resilver.
in my experience - you
neverreplace more than one drive at a time...Ask me how I know.
That's very interesting. I have not really had to deal with drive failures actually.
I haven't kept up with the number of drive failures I have had over nearly 30 years. In all of my personal systems, I think I have had
one.Work related,.. maybe all of three.
Wow, that's pretty small.
Personally, I've probably lost 3-4 drives. In businesses - well over 10.
And Scott has probably seen hundreds fail. Of course it all boils down to how many systems you see/support.
We have one or two fail every two-three months. Nothing crazy.
How many drives do you have?
A few hundred for now. Should be under 100 at the end of summer.
What are you guys changing to reduce that number by that much?
Higher density drives.
-
@coliver said in RAID10 - Two Drive Failure:
@Dashrender said in RAID10 - Two Drive Failure:
@coliver said in RAID10 - Two Drive Failure:
@Dashrender said in RAID10 - Two Drive Failure:
@gjacobse said in RAID10 - Two Drive Failure:
@wirestyle22 said in RAID10 - Two Drive Failure:
@gjacobse said in RAID10 - Two Drive Failure:
@JaredBusch said in RAID10 - Two Drive Failure:
Predictive failure is not failure. Replace one at a time. to give the RAID card the most power to work on the individual resilver.
in my experience - you
neverreplace more than one drive at a time...Ask me how I know.
That's very interesting. I have not really had to deal with drive failures actually.
I haven't kept up with the number of drive failures I have had over nearly 30 years. In all of my personal systems, I think I have had
one.Work related,.. maybe all of three.
Wow, that's pretty small.
Personally, I've probably lost 3-4 drives. In businesses - well over 10.
And Scott has probably seen hundreds fail. Of course it all boils down to how many systems you see/support.
We have one or two fail every two-three months. Nothing crazy.
How many drives do you have?
A few hundred for now. Should be under 100 at the end of summer.
so you're losing around 1.5% of your drives per year... that seems a bit high, but my memory for the norm as published by google could be off. Plus your environment might not be as good as theirs.
-
@wirestyle22 said in RAID10 - Two Drive Failure:
@JaredBusch said in RAID10 - Two Drive Failure:
individual resilver.
Does this mean that the mileage is only applied to the new drive or it's just minimal in relation to the rest of the raid? Reason I ask is I always thought this put a lot of strain on the entire raid.
WTF? This is a nothing more than a single mirror pair. The "strain" here is only a copy operation. The least possible work.
The point of individual is because something like this is processed 100% by the CPU on the RAID card. So don't make it do more than one thing at a time.
A parity array is different.
-
@DustinB3403 said in RAID10 - Two Drive Failure:
@aaronstuder What raid controller do you have?
Exactly this. A real SMB system should be a hot plug. But we have no idea what you bought.
-
@gjacobse said in RAID10 - Two Drive Failure:
@JaredBusch said in RAID10 - Two Drive Failure:
Predictive failure is not failure. Replace one at a time. to give the RAID card the most power to work on the individual resilver.
in my experience - you
neverreplace more than one drive at a time...Ask me how I know.
I solemnly swear that I've pulled the wrong drive to replace before . Made a RAID6 rebuild take a lot longer, and a RAID 10 freak out till a reboot happened. Restoring from backup was always an option at least.
-
@aaronstuder said in RAID10 - Two Drive Failure:
Drive are 1 and 3 are in "predictive failure" , I am assuming the pairs are 0+1 and 2+3.
Why?
-
@gjacobse said in RAID10 - Two Drive Failure:
@JaredBusch said in RAID10 - Two Drive Failure:
Predictive failure is not failure. Replace one at a time. to give the RAID card the most power to work on the individual resilver.
in my experience - you
neverreplace more than one drive at a time...Ask me how I know.
In RAID 10, you always do if they are in different RAID 1 sets, always.
-
@scottalanmiller said in RAID10 - Two Drive Failure:
@aaronstuder said in RAID10 - Two Drive Failure:
Drive are 1 and 3 are in "predictive failure" , I am assuming the pairs are 0+1 and 2+3.
Why?
Why what? Assuming? Because he did not document and most hardware RAID controllers are not accessible except during the boot process.
