How would you build a File server with 170TB of Usable Storage?
-
@scottalanmiller said in How would you build a File server with 170TB of Usable Storage?:
@DustinB3403 said in How would you build a File server with 170TB of Usable Storage?:
Which also raises the question is the 170TB 1 full backup, or some combination.
Not backup, the OP was file storage.
But @DustinB3403 was trying to drag this back to the original post, instead of reply appropriately to the call out of this topic.
-
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
As a response to the RAID10 Must use... thread, it got me to thinking.
[This topic isn't to rehash things stated in the other thread, so much as to hear how others would put something at this scale into production].
Business Goal: Build a production level storage system with 170TB of usable storage for SMB / NFS file shares for regular office workers up to power users and video editors.
- No NAS Products (Synology, QNAP), please.
- No Budget Constraints.
- RAIN Storage / multiple nodes can also be considered (CEPH, GLUSTER, etc).
My Build would use some flavor of Linux and based on the Storinator Q30 Enhanced and configured for RAID10.
Dual Power Supplies, No Setup / Configuration, and 24 x 14TB WD Purple Drives ~= $18,000.
(as we all know, the most significant cost will be the drives).
What would yours look like?
How much of the 170TB is video files, how much if it is hot? How big are the video files?
What are backup requirements?
Do you need triple the storage considerations for onprem backups plus a tape archive solution for off-site?
How quickly does that 170TB grow. You said 170TB of data, that tells me zero free space, so you already need more. How do you want to scale?
No budget? That makes no sense at all, Impossible.
It depends on many things.
Could it all live in the cloud with an onprem cache for hot or sticky data?
It's hard to give a real answer when the full scope and picture is not there.
-
Misread. So how much of the 170TB is used immediately and what's the projected growth rate?
-
@scottalanmiller said in How would you build a File server with 170TB of Usable Storage?:
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
My Build would use some flavor of Linux and based on the Storinator Q30 Enhanced and configured for RAID10.
This I would absolutely avoid. The Storinator has no place in a production SMB environment. It's exclusively designed for use as a RAIN node as part of an N+X cluster. It's disposable / ephemeral only. Any attempt to use it in another way, which would include any configuration with RAID, is misusing it by intent and will lead to support issues. It is not hot swap, it is not designed to be reliable. It's not meant to be serviced. All top considerations for a storage node. It's actually, other than capacity, the exact opposite of what you would seek in a storage device.
@scottalanmiller -- How would you build it? That is specifically my question or discussion point... How would you go about building a storage platform for 170TB usable data?
-
@Obsolesce said in How would you build a File server with 170TB of Usable Storage?:
Misread. So how much of the 170TB is used immediately and what's the projected growth rate?
40% Currently Used. 6% /year growth rate.
-
@Obsolesce said in How would you build a File server with 170TB of Usable Storage?:
Misread. So how much of the 170TB is used immediately and what's the projected growth rate?
Not relevant. The concept is that we need 170TB usable. Usable implies available now.
You are asking a different question to resolve a different problem.
-
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
@scottalanmiller said in How would you build a File server with 170TB of Usable Storage?:
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
My Build would use some flavor of Linux and based on the Storinator Q30 Enhanced and configured for RAID10.
This I would absolutely avoid. The Storinator has no place in a production SMB environment. It's exclusively designed for use as a RAIN node as part of an N+X cluster. It's disposable / ephemeral only. Any attempt to use it in another way, which would include any configuration with RAID, is misusing it by intent and will lead to support issues. It is not hot swap, it is not designed to be reliable. It's not meant to be serviced. All top considerations for a storage node. It's actually, other than capacity, the exact opposite of what you would seek in a storage device.
@scottalanmiller -- How would you build it? That is specifically my question or discussion point... How would you go about building a storage platform for 170TB usable data?
Assuming a single chassis, pick a linux distribution, and setup software raid, LVM, and XFS. My other two preferences would be ZFS on BSD, or hardware RAID.
Now if we start talking performance, backup or failover, it quickly becomes more difficult.
-
@travisdh1 said in How would you build a File server with 170TB of Usable Storage?:
Now if we start talking performance, backup or failover, it quickly becomes more difficult.
I'm specifically avoiding those topics for the sake of this post. I'm mainly curious as to what the hardware would look like... but more detailed than "buy a server and put hard drives in it" lol.
-
@travisdh1 said in How would you build a File server with 170TB of Usable Storage?:
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
@scottalanmiller said in How would you build a File server with 170TB of Usable Storage?:
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
My Build would use some flavor of Linux and based on the Storinator Q30 Enhanced and configured for RAID10.
This I would absolutely avoid. The Storinator has no place in a production SMB environment. It's exclusively designed for use as a RAIN node as part of an N+X cluster. It's disposable / ephemeral only. Any attempt to use it in another way, which would include any configuration with RAID, is misusing it by intent and will lead to support issues. It is not hot swap, it is not designed to be reliable. It's not meant to be serviced. All top considerations for a storage node. It's actually, other than capacity, the exact opposite of what you would seek in a storage device.
@scottalanmiller -- How would you build it? That is specifically my question or discussion point... How would you go about building a storage platform for 170TB usable data?
Assuming a single chassis, pick a linux distribution, and setup software raid, LVM, and XFS. My other two preferences would be ZFS on BSD, or hardware RAID.
Now if we start talking performance, backup or failover, it quickly becomes more difficult.
I'm not familiar with ZFS. I will stay far FAR FAAR FAAAAAAARRRRRRRR away from BTRFS. I've seen too many problems with it at my day job, lol.
I'd likely go with Linux + XFS for the file system.
-
LVM Thin Pool and XFS for a file server?
-
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
@travisdh1 said in How would you build a File server with 170TB of Usable Storage?:
Now if we start talking performance, backup or failover, it quickly becomes more difficult.
I'm specifically avoiding those topics for the sake of this post. I'm mainly curious as to what the hardware would look like... but more detailed than "buy a server and put hard drives in it" lol.
Dumb topic them.
-
@Obsolesce said in How would you build a File server with 170TB of Usable Storage?:
Dumb topic them.
:thumbs_up:
-
@JaredBusch said in How would you build a File server with 170TB of Usable Storage?:
@Obsolesce said in How would you build a File server with 170TB of Usable Storage?:
Misread. So how much of the 170TB is used immediately and what's the projected growth rate?
Not relevant. The concept is that we need 170TB usable. Usable implies available now.
You are asking a different question to resolve a different problem.
It is relevant because I have no idea what was put into the plan. For all I know, they only have 35TB of data that will only grow to 46TB over the next 5 years. If that's the case, it could change every aspect of the best solution.
If you are saying to just take the 170TB that they just want that amount and tjays that... Then I don't care to put any thought into this thread and will just say buy a server, some DAS, and move on.
-
@Obsolesce said in How would you build a File server with 170TB of Usable Storage?:
@JaredBusch said in How would you build a File server with 170TB of Usable Storage?:
@Obsolesce said in How would you build a File server with 170TB of Usable Storage?:
Misread. So how much of the 170TB is used immediately and what's the projected growth rate?
Not relevant. The concept is that we need 170TB usable. Usable implies available now.
You are asking a different question to resolve a different problem.
It is relevant because I have no idea what was put into the plan. For all I know, they only have 35TB of data that will only grow to 46TB over the next 5 years. If that's the case, it could change every aspect of the solution.
If you are saying to just take the 170TB that they just want that amount and tjays that... Then I don't care to put any thought into this thread and will just say buy a server, some DAS, and move on.
That is the point of the thread.. what hardware to buy to do that..
The point of the thread is not to analyze some made up needs. The original thread does that.
This thread is simply to say, I have this valid need for 170TB of storage. How do I actually build it with what hardware.
-
@JaredBusch said in How would you build a File server with 170TB of Usable Storage?:
@Obsolesce said in How would you build a File server with 170TB of Usable Storage?:
@JaredBusch said in How would you build a File server with 170TB of Usable Storage?:
@Obsolesce said in How would you build a File server with 170TB of Usable Storage?:
Misread. So how much of the 170TB is used immediately and what's the projected growth rate?
Not relevant. The concept is that we need 170TB usable. Usable implies available now.
You are asking a different question to resolve a different problem.
It is relevant because I have no idea what was put into the plan. For all I know, they only have 35TB of data that will only grow to 46TB over the next 5 years. If that's the case, it could change every aspect of the solution.
If you are saying to just take the 170TB that they just want that amount and tjays that... Then I don't care to put any thought into this thread and will just say buy a server, some DAS, and move on.
That is the point of the thread.. what hardware to buy to do that..
The point of the thread is not to analyze some made up needs. The original thread does that.
This thread is simply to say, I have this valid need for 170TB of storage. How do I actually build it with what hardware.
I see.
-
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
No Setup / Configuration
Not sure what you mean. Storinator doesn't ship any production ready configuration last I knew. ALL of the work has to be done by you. I don't think that they have any RAIN offerings or even support. Not that they would, they are a chassis vendor, not a storage vendor. But it would be a lot of work to do correctly.
-
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
@scottalanmiller said in How would you build a File server with 170TB of Usable Storage?:
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
My Build would use some flavor of Linux and based on the Storinator Q30 Enhanced and configured for RAID10.
This I would absolutely avoid. The Storinator has no place in a production SMB environment. It's exclusively designed for use as a RAIN node as part of an N+X cluster. It's disposable / ephemeral only. Any attempt to use it in another way, which would include any configuration with RAID, is misusing it by intent and will lead to support issues. It is not hot swap, it is not designed to be reliable. It's not meant to be serviced. All top considerations for a storage node. It's actually, other than capacity, the exact opposite of what you would seek in a storage device.
@scottalanmiller -- How would you build it? That is specifically my question or discussion point... How would you go about building a storage platform for 170TB usable data?
I would use a real server platform, no matter what the product need is, I use production gear. That's a starting point, even when you need it cheap, you still have to support it.
For 170TB, if it were me, I'd have a RAIN cluster. You didn't go into all of your theoretical details, but assuming pretty common needs, Gluster might be a good choice across a handful of chassis.
But you can do a single chassis, if I was doing that I would use RAID 10. Vendors like SuperMicro and Huawei make servers for this purpose (Storinator is a grid node, not a server on its own, it's a component of a larger system only) that act as servers with server class components and things like hot swap on the drives.
-
@dafyre said in How would you build a File server with 170TB of Usable Storage?:
I'm not familiar with ZFS.
ZFS is specifically the filesystem made by SUN for exactly this purpose. Before there was Storinator, before there were BackBlaze Pods, there was the SUN Thumper - an enterprise class, top loading, customer engineering storage server made to hold an insane number of drives and keep them safe and operational in a single chassis. All those others are attempts to replicate it without the enterprise components. The difference is, Thumper was designed to be a server, the Pods were not, but rather nodes.
The ZFS / Thumper team is who worked with me to design the first SAM-SD as well.
ZFS was super important to making Thumper work as it added RAID 7 (aka RAID 5-3) as an option, and all kinds of cache control, and a file system that could scale as big as Thumper could go.
-
@black3dynamite said in How would you build a File server with 170TB of Usable Storage?:
LVM Thin Pool and XFS for a file server?
Yes, XFS is super reliable and pretty fast.
-
I'd just put twenty 15TB NVMe drives in my little 1U server and call it a day.
https://www.supermicro.com/a_images/products/views/1029UZ-TN20R25M_angle.jpg
