How would you build a File server with 170TB of Usable Storage?
-
@scottalanmiller said in How would you build a File server with 170TB of Usable Storage?:
For 170TB, if it were me, I'd have a RAIN cluster. You didn't go into all of your theoretical details, but assuming pretty common needs, Gluster might be a good choice across a handful of chassis.
I made it theoretical so you could use your imagination a little.
From the get-go, I expected somebody to come back with something like Gluster a little bit quicker. Yes, I realize that adds complexity and networking needs and such. It does solve the problem though. Building 3 nodes with enough storage and using Gluster to make the entire 170GB available for clients to use.
-
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
If I'm using storage shelves (drive enclosures), then a RAID card + 2 or 3 Shelves to get me up to enough drives to hold my data at RAID10 would be a more viable option. That way I could pretty much whatever I want as the base server and go from there.
Because they are a pain at this size. You can get more than enough internal storage for 170TB, no need for shelves. But yes, shelve work.
No need for a RAID card, software RAID will be better for this use case.
-
@Pete-S said in How would you build a File server with 170TB of Usable Storage?:
I'd just put twenty 15TB NVMe drives in my little 1U server and call it a day.
https://www.supermicro.com/a_images/products/views/1029UZ-TN20R25M_angle.jpgha ha ha.
That's actually a pretty neat setup. I configured it on their site for kicks & grins using 18 x 4TB NVME drives... Yours for only $60k... Gets you up to 36TB in RAID10...
-
If you need more storage density you could also go with another chassis using the NF1 format SSDs from Samsung.
That's a whopping 0.54 PB in 1U.@dafyre I checked our supplier and fully stocked with 36 15TB Samsung PM983 NVMe drives it would be priced at around $115K with 768GB RAM and two Xeon CPUs and a couple of 25Gb NICs.
https://www.supermicro.com/a_images/products/views/1029P-NMR36L_angle.jpg
-
I need to dig in and see what Chasis our new Nimble Storage units are (They're the new ones form HPE). I know those run about $60k each. I need to see how much storage we get in each.
-
@Pete-S said in How would you build a File server with 170TB of Usable Storage?:
I'd just put twenty 15TB NVMe drives in my little 1U server and call it a day.
https://www.supermicro.com/a_images/products/views/1029UZ-TN20R25M_angle.jpgThis is the best answer for the given question. Anything else is going into assumption land which was specifically not allowed.
-
@JaredBusch said in How would you build a File server with 170TB of Usable Storage?:
@Obsolesce said in How would you build a File server with 170TB of Usable Storage?:
Misread. So how much of the 170TB is used immediately and what's the projected growth rate?
Not relevant. The concept is that we need 170TB usable. Usable implies available now.
You are asking a different question to resolve a different problem.
Your own configuration does not provide 170TB usable. You calculated using unformatted capacity, and didn't take into account that even unformatted, a 14TB drive is only a decimal representation used for marketing, which takes another chunk out of the usable capacity
-
@Vlinderbeest said in How would you build a File server with 170TB of Usable Storage?:
@JaredBusch said in How would you build a File server with 170TB of Usable Storage?:
@Obsolesce said in How would you build a File server with 170TB of Usable Storage?:
Misread. So how much of the 170TB is used immediately and what's the projected growth rate?
Not relevant. The concept is that we need 170TB usable. Usable implies available now.
You are asking a different question to resolve a different problem.
Your own configuration does not provide 170TB usable. You calculated using unformatted capacity, and didn't take into account that even unformatted, a 14TB drive is only a decimal representation used for marketing, which takes another chunk out of the usable capacity
Plus RAID and FS overhead. With 14TB drives using mirroring, you'd need at least 26 drives, and 28 to be safe.
-
Just for the amusement factor, I just checked out one of the 90 bay SuperMicro chassis. With 14TB drives, that's 500TB usable in a single 4u chassis. All for around $60k
-
Was working with HPE today on a Nimble Storage controller... Turns out they use SuperMicro something or another. I forgot to bookmark the page before I logged out of the machine I was using. I'll find it tomorrow.
-
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
Was working with HPE today on a Nimble Storage controller... Turns out they use SuperMicro something or another. I forgot to bookmark the page before I logged out of the machine I was using. I'll find it tomorrow.
Tee hee. Now that's pretty funny. Even HPE uses SuperMicro!
-
@scottalanmiller said in How would you build a File server with 170TB of Usable Storage?:
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
Was working with HPE today on a Nimble Storage controller... Turns out they use SuperMicro something or another. I forgot to bookmark the page before I logged out of the machine I was using. I'll find it tomorrow.
Tee hee. Now that's pretty funny. Even HPE uses SuperMicro!
Don't know about the chasis itself, but the mobo's are, for sure.
I'll get a screen cap if I'm able.
-
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
@scottalanmiller said in How would you build a File server with 170TB of Usable Storage?:
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
Was working with HPE today on a Nimble Storage controller... Turns out they use SuperMicro something or another. I forgot to bookmark the page before I logged out of the machine I was using. I'll find it tomorrow.
Tee hee. Now that's pretty funny. Even HPE uses SuperMicro!
Don't know about the chasis itself, but the mobo's are, for sure.
I'll get a screen cap if I'm able.
Oh okay HP has all kinds of people make mobos.
-
Bringing something up from the "other" thread. Would RAID rebuild times play a factor into choosing a method of supplying this kind of storage amount?
-
@hobbit666 said in How would you build a File server with 170TB of Usable Storage?:
Bringing something up from the "other" thread. Would RAID rebuild times play a factor into choosing a method of supplying this kind of storage amount?
My first thought is YES! DEFINITELY! However, it actually may depend on how you set it up. Systems like Ceph and Gluster don't necessarily need RAID since they can work with different disks in the same node.
-
@hobbit666 said in How would you build a File server with 170TB of Usable Storage?:
Bringing something up from the "other" thread. Would RAID rebuild times play a factor into choosing a method of supplying this kind of storage amount?
Absolutely, that's why at scale you are generally limited to RAID 10. RAID 60 can work, but with so many RAID 6 arrays in the pool that you save extremely little compared to RAID 10, but you lose so much that the balance is rarely worth it.
RAID 10 is the only level that really scales because of the rebuild problems.
-
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
Systems like Ceph and Gluster don't necessarily need RAID since they can work with different disks in the same node.
They definitely don't need RAID. That's their point.
