Advice On a New Setup



  • Hi, i've been reading for last 3 weeks or so about sam-sd (open storage) but i can't decide which is best for my situation, and i would really appreciate your opinions on this setup as a (veeam backup, file archive and a lightly accessed file server) on site and an identical mirrored one off-site next building (1gb fiber)

    Current Setup
    2 old ibm x3650-7979 with raid 10 (6x 2tb wd black), please dont judge, it was already here when i started working here and we're moving forward 👍🏼
    but now it is so slow with backups and restores, so power hungry, raid is doing funny things and corruptions, low storage

    Goal
    Run a 'proper' storage server OS
    Upgrade storage capacity
    distributed file system between the two servers

    thought of Dell R510 (Acctually, it is the only one beside r710 with 3.5 we can find as a refurbished and more than 6 hd bays)
    single 6 core cpu
    32g Ram
    Raid H700
    12 (empty bay)

    What i could find in my county is these hard drives

    Seagate IronWolf NAS 4tb, 6tb, 8tb, 10tb
    Western Digital Gold 4tb, 6tb, 8tb, 10tb, 12tb
    Western Digital Red Pro 8tb
    Western Digital Red 4tb, 6tb, 8tb

    what we can afford is 4tb, 6tb or 8tb of any model, unfortunately the increase in price between 8tb to 10tb or 12tb is out of our reach,

    i would like your opinions on
    1- Which first choice and second choice drive model to buy,
    2- Which RAID setup if i want at least double the current setup capacity with the availability to expand the array later with just adding more drives and if it is OBR10, is it expandable on this setup or just the OBR6
    3- Which linux (Virtualized) OS to use? im familiar with most of them
    4- Which tool to use to mirror the two servers and distribute the load , can i use gluster or ceph or it is not worth it and just use DRDB

    thanks in advanced guys.



  • @GodfatherX64 said in Advice On a New Setup:

    What i could find in my county is these hard drives

    Seagate IronWolf NAS 4tb, 6tb, 8tb, 10tb
    Western Digital Gold 4tb, 6tb, 8tb, 10tb, 12tb
    Western Digital Red Pro 8tb
    Western Digital Red 4tb, 6tb, 8tb

    1- Which first choice and second choice drive model to buy,

    You likely want to rule out Seagate due to reliability concerns. See @CCWTech for info on why to avoid Seagate.

    Now between Red, RedPro, and Gold. Red you likely want to rule out because those are 5400 RPM Green drives with TLER blocking removed. They are okay to use, but their speed alone is likely a bit concern for you in the real world. Find for storing archives, but they are insanely slow overall.

    So that leaves Red Pro and Gold. Between these two, you have only the 8TB in the Red Pro, so almost certainly you will be going with the Gold just for availability reasons. Either is fine.



  • @GodfatherX64 said in Advice On a New Setup:

    2- Which RAID setup if i want at least double the current setup capacity with the availability to expand the array later with just adding more drives and if it is OBR10, is it expandable on this setup or just the OBR6

    OBR10 is almost never expandable. OBR6 generally is. OBR6 will be far less safe, especially as your array size gets bigger and bigger. And write speeds will be much lower. But it gives you more capacity at a lower price point.



  • @GodfatherX64 said in Advice On a New Setup:

    3- Which linux (Virtualized) OS to use? im familiar with most of them

    Most any are fine. I tend to use Fedora these days. Suse is best for the most storage options and updates, but that's not a big thing anymore. Ubuntu is fine.



  • @GodfatherX64 said in Advice On a New Setup:

    4- Which tool to use to mirror the two servers and distribute the load , can i use gluster or ceph or it is not worth it and just use DRDB

    This isn't something you would do. No one does this, not even in super high performance scenarios. Things like DRBD, Gluster, and CEPH are for use on the raw storage, not for use on top of your RAID.

    If you needed clustering here (trust me, you don't) then you'd do it lower in the stack with Starwind, not up on top in the VM.

    Keep in mind even big shops when they talk about clustering for file servers it is for failover, not load balancing.



  • @scottalanmiller said in Advice On a New Setup:

    @GodfatherX64 said in Advice On a New Setup:

    2- Which RAID setup if i want at least double the current setup capacity with the availability to expand the array later with just adding more drives and if it is OBR10, is it expandable on this setup or just the OBR6

    OBR10 is almost never expandable. OBR6 generally is. OBR6 will be far less safe, especially as your array size gets bigger and bigger. And write speeds will be much lower. But it gives you more capacity at a lower price point.

    You don't want do expand TBs of data. I've been through three of them, where one failed. The first took over a month, which was a big RAID 5... and afterwards a drive failed, but rebuilt again fine. I can only imagine a similar RAID 6 taking longer. The second was a few TB RAID 10 SSD array, which didn't take long and was fine. The third was a large RAID 10 which took forever and failed.

    Either plan ahead capacity wise, or have a way to offload your data somewhere safe and do a complete RAID rebuild instead of expansion.



  • @Obsolesce said in Advice On a New Setup:

    @scottalanmiller said in Advice On a New Setup:

    @GodfatherX64 said in Advice On a New Setup:

    2- Which RAID setup if i want at least double the current setup capacity with the availability to expand the array later with just adding more drives and if it is OBR10, is it expandable on this setup or just the OBR6

    OBR10 is almost never expandable. OBR6 generally is. OBR6 will be far less safe, especially as your array size gets bigger and bigger. And write speeds will be much lower. But it gives you more capacity at a lower price point.

    You don't want do expand TBs of data. I've been through three of them, where one failed. The first took over a month, which was a big RAID 5... and afterwards a drive failed, but rebuilt again fine. I can only imagine a similar RAID 6 taking longer. The second was a few TB RAID 10 SSD array, which didn't take long and was fine. The third was a large RAID 10 which took forever and failed.

    Either plan ahead capacity wise, or have a way to offload your data somewhere safe and do a complete RAID rebuild instead of expansion.

    Agreed, RAID growth is a form of induced failure and carries huge risks. It's almost never worth it. And even less worth it if you plan for it from the beginning. It's a bad fallback situation, not something to do intentionally from the beginning.



  • I'd say ditch the whole idea of file servers, especially if they are lightly used and move to cloud storage. $10 per user per month gets you unlimited storage from Google. You would only need local storage for Veeam backups, in one location, since you have gigabit fiber between sites. And for that you could just get Supermicro server, Asrock rack has some nice options too, or even build one. Synology or similar will work fine as a target for backups too.



  • @marcinozga said in Advice On a New Setup:

    I'd say ditch the whole idea of file servers, especially if they are lightly used and move to cloud storage. $10 per user per month gets you unlimited storage from Google.

    You sure that's the pricing?

    0_1543713126345_Screenshot from 2018-12-01 19-11-55.png



  • On GSuite, they claim unlimited if you have six users or more, 1TB limit if you don't.



  • 0_1543713806549_screen.png

    That's my account, G Suite business for $10, 1 user. They don't enforce 5 users minimum, but any business could afford 5 users.



  • I had to buy 4 drives few days ago, black friday sale, 4TB WD Red for $100 each, because all 10 of my drives are failing. That would have paid for 3 years and 4 months of G Suite, and with power cost it'd probably give me 4 years. Unfortunately I still need few TBs of local storage, but I just cannot justify the cost.



  • @marcinozga said in Advice On a New Setup:

    0_1543713806549_screen.png

    That's my account, G Suite business for $10, 1 user. They don't enforce 5 users minimum, but any business could afford 5 users.

    Odd that they price it so insanely at odds with their other plan.



  • (Ab)use it while you can 🙂



  • @marcinozga said in Advice On a New Setup:

    I had to buy 4 drives few days ago, black friday sale, 4TB WD Red for $100 each, because all 10 of my drives are failing. That would have paid for 3 years and 4 months of G Suite, and with power cost it'd probably give me 4 years. Unfortunately I still need few TBs of local storage, but I just cannot justify the cost.

    That works for one user, but for a business, it gets costly quickly. We certainly don't spend $120/year for storage for our users. We are a tiny business, but even so, we would be $2,280 per year for storage. Sure, if we used several PB of data, that would be hard to beat, but who does that?

    In reality, we use under 1TB. So that cost is absurd. Even if we used a lot more, four drives of 4TB each in RAID 10 is 8TB for just $400 in drives. Add a small server to run NextCloud and maybe we spend $1,400. Let's say we go with a collocation facility to put it in. That's another $900. That's only $2,300 for the first year, and $900 for each additional year.

    Over five years, we'd spend $5,900 instead of $11,400. And that's assuming we don't have any virtualization or shared computing at all, that's getting a dedicated server, dedicated drives, and dedicated colocation for that one box, and assuming we need 8TB of storage. For what we really need, we could easily make do with two 2TB drives taking that $400 cost down to just $165.

    Now imagine a larger business with 40 or 80 or even 200 users. The expected per user storage needs would plummet while the cost for Google storage would continue to grow exponentially.

    So it is anything but a slam dunk. For one or two users, that stuff is hard to beat. But for even a pretty small firm, it can be insanely expensive and limiting.



  • @marcinozga said in Advice On a New Setup:

    (Ab)use it while you can 🙂

    Really just shows their other plan as being crazy. Both seem to gouge on the pricing, though.



  • We just lost a customer who did a SAM-SD install a few months ago. They put in a server, I think it was about $6,000. It had over 48TB of storage on it. It's a lot of storage, but they have over 140 users.

    With G Suite, it would have cost them $16,800 for that same storage per year. So in a five year window the one used in house would remain at $6,000 while the G Suite approach would balloon to $84,000. A difference of $78,000 in a real world SAM-SD scenario that we just went through.

    That's not even considering that they don't have a network connection that could handle the traffic to have G Suite do the hosting. That's a separate, but very important issue. They need file transfer speeds at GigE and decently reliable. Their Internet (they are in the US) isn't very fast and definitely not stable.



  • If you had 6x2TB in RAID10 that's only 6TB of actual storage. That's very modest storage requirements.
    A simple RAID1 with two 12TB drives would double that. WD Ultrastar (former HGST Ultrastar) for instance.

    I'm sure you can find other brands of hard drives regardless of what country you are in. You just need to find the right suppliers.

    You handle storage expansion by adding a new array of disks.

    Regardless of what drives you want even 5400 rpm drives will saturate a gigabit link. I have used WD Red and they were fine but I would look for drives with a 5 year warranty, not 3 years as WD Red have. The price difference is not that big.

    Personally I wouldn't waste money on hardware raid but you need some kind of host adapter to access the drives. Software raid is very fast and RAID1 takes no cpu to speak of. With refurbished stuff you have to take what you can find though.



  • I'd also look for something newer than R510. Basically you want Ivy bridge or newer Xeon CPUs because there is a pretty big step down in power usage when Intel started with the 22nm process.

    R510 and R710 are too old IMHO. R720 will work though but with the E5-2600 V2 series processor for lower power consumption.



  • wow, thanks guys for your informative opinions

    so
    @scottalanmiller , WD Gold, OBR10 for performance, fedora (vm) with xfs maybe,

    @scottalanmiller @Obsolesce , ok , no RAID expansion, plan from the beginning , got it.

    @marcinozga , unfortunately, like @scottalanmiller said it will be more costly than local, and the gigabit fiber is between 2 buildings , not the internet connection

    @Pete-S , thanks for the 12tb suggestion, but i want performance too if i went for 10gb lan
    and unfortunately, i couldn't find any 3.5 servers except these 2, r710, r510



  • @scottalanmiller said in Advice On a New Setup:

    @GodfatherX64 said in Advice On a New Setup:
    Keep in mind even big shops when they talk about clustering for file servers it is for failover, not load balancing.

    what setup do you suggest to create a failover backup cluster?
    lets say i want to connect to the 2 on site servers with one connection or one path , an if one is down the second is up with the same data on it



  • @GodfatherX64 said in Advice On a New Setup:

    @scottalanmiller said in Advice On a New Setup:

    @GodfatherX64 said in Advice On a New Setup:
    Keep in mind even big shops when they talk about clustering for file servers it is for failover, not load balancing.

    what setup do you suggest to create a failover backup cluster?
    lets say i want to connect to the 2 on site servers with one connection or one path , an if one is down the second is up with the same data on it

    Starwind will probably work for this, in fact I had a failover cluster at work using Starwind few years ago, it wasn't worth it. Power loss was the weak point, Starwind had to sync storage after both nodes went down, so we were talking about 3-4 hours of downtime each time power went out for long enough to drain UPSes, no generators unfortunately. We had to spend additional few thousand on bigger UPS units.

    So do the math, if the server goes down, would the business lose so much money to justify buying 2nd server? And probably more equipment (UPS, etc.) You can have 4 hour onsite service, Dell I think even offers 2 hours onsite, so best case you could be up and running in 3-4 hours in case of a server failure.

    If you plan on going with 10gbit network, you really should be looking at SSD disks, spinning rust will have a hard time saturating that, probably 8-10 disks at minimum.



  • @GodfatherX64 said in Advice On a New Setup:

    @Pete-S , thanks for the 12tb suggestion, but i want performance too if i went for 10gb lan
    and unfortunately, i couldn't find any 3.5 servers except these 2, r710, r510

    If you're looking for servers with more 3.5" bay capability, have a look at Supermicro. They have a lot of different servers and sizes - many of them with 3.5" bays.

    They also have JBOD chassis which means you can take any standard 1U/2U server, put in a raid card that has external SAS ports and hook it up to the JBOD chassis. That way you can hook up how ever many disks you want to a simple 1U server.





  • @GodfatherX64 said in Advice On a New Setup:

    @Pete-S , thanks for the 12tb suggestion, but i want performance too if i went for 10gb lan
    and unfortunately, i couldn't find any 3.5 servers except these 2, r710, r510

    Dell 720xd - 12x3.5
    Dell 530 - 8x3.5
    Dell 730 - 8x2.5
    Dell 730xd - 12x3.5 + 4 non-hot swap 3.5

    There are plenty of other options. I would definitely check out xbyte. They might even have a precofigured 720xd "special" to your liking.



  • @bnrstnr said in Advice On a New Setup:

    @GodfatherX64 said in Advice On a New Setup:

    @Pete-S , thanks for the 12tb suggestion, but i want performance too if i went for 10gb lan
    and unfortunately, i couldn't find any 3.5 servers except these 2, r710, r510

    Dell 720xd - 12x3.5
    Dell 530 - 8x3.5
    Dell 730 - 8x2.5
    Dell 730xd - 12x3.5 + 4 non-hot swap 3.5

    There are plenty of other options. I would definitely check out xbyte. They might even have a precofigured 720xd "special" to your liking.

    xbyte normally only ships to the U.S., which @GodfatherX64 isn't in from what we can tell here.



  • @GodfatherX64 said in Advice On a New Setup:

    @scottalanmiller , WD Gold, OBR10 for performance, fedora (vm) with xfs maybe,

    That would be your best speed and reliability, yes. And definitely XFS.



  • @GodfatherX64 said in Advice On a New Setup:

    what setup do you suggest to create a failover backup cluster?
    lets say i want to connect to the 2 on site servers with one connection or one path , an if one is down the second is up with the same data on it

    If you want a failover cluster for a file server, use Starwind and do your failover at the platform level (hypervisor), not in the VM. For other workloads, like databases, you want the application to handle it.



  • @dave_c said in Advice On a New Setup:

    @GodfatherX64
    Entersource (among others) ships to many countries
    https://www.enterasource.com/dell-poweredge-r720-8-port-lff-2u-rackmount-server-configure-to-order

    xByte will ship to many countries, too. But the tariffs can be tough.



  • info for all : i live in Egypt

    @marcinozga , thanks for your suggestion , i'm starting to think about one big server instead of two.

    @Pete-S , unfortunately, supermicro is not available here and the shipping cost for a new server is massive.

    @dave_c , the cost to buy something from outside the country is tough, we cant do that right now.

    @bnrstnr , the r510 is the only server i can find here with 3.5 slots, either that or older,
    and we can't import from outside the country these days.


Log in to reply