Best way to backup big data...
-
Hi Guys,
So I met someone recently who has an on-site server and NAS and have told me there is 45TB of data they want to backup. I understand they currently backup to a number of different external drives under a few different schedules but want to try streamline this. I havent handled this amount of data before (large media/raw video files etc) so wondered how best to backup 45TB's of growing data?
Cloud would be best but think it could be expensive?
-
Do they have a 2nd building/office they could put a NAS into?
Cloud could work, the initial upload would take time also depending on the amount of data that changes regularly.
-
No 2nd building but I'm sure they could find a home to host a secondary NAS somewhere. You think that's best option?
-
I like keep a copy in Tapes.
If you can afford it, LTO-8 is a good option.
-
@joel it could work.
Depends on the goals / business needs. Are they looking to just backup so if the primary fails the data is there, but what happens if the building burns down? Both sets will be lost.
-
We use Exagrids and tape. Though for only 45TB you could just build a box and Colo it. RHEL/CentOS now have VDO support so you get dedupe and compression on those volumes.
A supermicro box with 24 8TB drives is around $13K. That's around 90TB in RAID 10. I don't know pricing for smaller because we build with those. But it shouldn't be too expensive to build your own and ship to it off-site.
-
You looking to backup the server and the NAS? What does the server do? Is it virtualized? What brand NAS? How much data is on each device?
-
My first question is ...
Seriously? Forty-Five Terra Bytes?? good god - what are they storing?
That seems like a huge amount of data... And you damn sure don't do that in a weekend...
-
the initial push of something of that size will suck.
But what is their change rate?
You could seed it manually and then resume sync.
-
I dont know anything as much as they want to backup 45TBs lol!
I'm trying to schedule a meeting to have a chat and find out more so will come back once (if) they come back to me...It was someone I met over the weekend so I dont know so much just yet. Just making initial enquiry to suss out what others do with this amount of data. -
Veeam to StarWind Virtual Tape Library (VTL) with a BackBlaze backend connection to back up all that data.
We're getting ready to pilot this kind of setup. VTL integrates really nicely into a tape based infrastructure with little to no mods needed.
Edit: For on-site storage we set up either a Storage Spaces Direct cluster or a Converged cluster. In this case it would be Converged with close to 1PB of available storage (2x nodes + 1x 102 bay JBOD)
-
@stacksofplates said in Best way to backup big data...:
We use Exagrids and tape. Though for only 45TB you could just build a box and Colo it. RHEL/CentOS now have VDO support so you get dedupe and compression on those volumes.
A supermicro box with 24 8TB drives is around $13K. That's around 90TB in RAID 10. I don't know pricing for smaller because we build with those. But it shouldn't be too expensive to build your own and ship to it off-site.
If it's large files likes raw video then compression and deduplication is unfortunately of very limited use.
We just use a standard supermicro 4U server with 24x3.5" drive bays. Running software RAID-6 with very modest hardware specs we have 250MB/s sustained write and 700MB/s read. More than enough to saturate a dual gigabit network link.
Two RAID-6 arrays with twelve 3.5" 10TB enterprise drives in each will give you around 200TB of storage. Or perhaps three RAID-6 arrays with 8 drives in each giving you about 180TB.
The most money in this type of config will be in the drives themselves. 10TB Seagate Exos X10 are about $330 each so 24 drives is $8K.
