Best way to backup big data...



  • Hi Guys,

    So I met someone recently who has an on-site server and NAS and have told me there is 45TB of data they want to backup. I understand they currently backup to a number of different external drives under a few different schedules but want to try streamline this. I havent handled this amount of data before (large media/raw video files etc) so wondered how best to backup 45TB's of growing data?

    Cloud would be best but think it could be expensive?



  • Do they have a 2nd building/office they could put a NAS into?

    Cloud could work, the initial upload would take time also depending on the amount of data that changes regularly.



  • No 2nd building but I'm sure they could find a home to host a secondary NAS somewhere. You think that's best option?



  • I like keep a copy in Tapes.

    If you can afford it, LTO-8 is a good option.



  • @joel it could work.

    Depends on the goals / business needs. Are they looking to just backup so if the primary fails the data is there, but what happens if the building burns down? Both sets will be lost.



  • We use Exagrids and tape. Though for only 45TB you could just build a box and Colo it. RHEL/CentOS now have VDO support so you get dedupe and compression on those volumes.

    A supermicro box with 24 8TB drives is around $13K. That's around 90TB in RAID 10. I don't know pricing for smaller because we build with those. But it shouldn't be too expensive to build your own and ship to it off-site.



  • You looking to backup the server and the NAS? What does the server do? Is it virtualized? What brand NAS? How much data is on each device?



  • My first question is ...

    Seriously? Forty-Five Terra Bytes?? good god - what are they storing?

    That seems like a huge amount of data... And you damn sure don't do that in a weekend...



  • the initial push of something of that size will suck.

    But what is their change rate?

    You could seed it manually and then resume sync.



  • I dont know anything as much as they want to backup 45TBs lol!
    I'm trying to schedule a meeting to have a chat and find out more so will come back once (if) they come back to me...It was someone I met over the weekend so I dont know so much just yet. Just making initial enquiry to suss out what others do with this amount of data.



  • Veeam to StarWind Virtual Tape Library (VTL) with a BackBlaze backend connection to back up all that data.

    We're getting ready to pilot this kind of setup. VTL integrates really nicely into a tape based infrastructure with little to no mods needed.

    Edit: For on-site storage we set up either a Storage Spaces Direct cluster or a Converged cluster. In this case it would be Converged with close to 1PB of available storage (2x nodes + 1x 102 bay JBOD)



  • @stacksofplates said in Best way to backup big data...:

    We use Exagrids and tape. Though for only 45TB you could just build a box and Colo it. RHEL/CentOS now have VDO support so you get dedupe and compression on those volumes.

    A supermicro box with 24 8TB drives is around $13K. That's around 90TB in RAID 10. I don't know pricing for smaller because we build with those. But it shouldn't be too expensive to build your own and ship to it off-site.

    If it's large files likes raw video then compression and deduplication is unfortunately of very limited use.

    We just use a standard supermicro 4U server with 24x3.5" drive bays. Running software RAID-6 with very modest hardware specs we have 250MB/s sustained write and 700MB/s read. More than enough to saturate a dual gigabit network link.

    Two RAID-6 arrays with twelve 3.5" 10TB enterprise drives in each will give you around 200TB of storage. Or perhaps three RAID-6 arrays with 8 drives in each giving you about 180TB.

    The most money in this type of config will be in the drives themselves. 10TB Seagate Exos X10 are about $330 each so 24 drives is $8K.


Log in to reply