Massive Storage Need for Video Project
-
I am a few months away from embarking on what will likely be a rather sizable video project. That's why the new GoPro
During this project I will be producing a lot of video, most of it while I am in Europe without necessarily a great Internet connection - at best it will vary month to month. This makes things very hard. I will have extremely little capacity to be moving around storage with me, at least storage to any real scale.So this is a big challenge. I need to get the video footage, which will likely be under 20GB per day but could, in theory, be up towards 128GB - 256GB per day on busy days. I need to not only store it but get it backed up in some way (just using a USB drive isn't going to cut it, I need to get it to something like Amazon S3 or OneDrive.)
Something like MS OneDrive is probably out because their storage limits, while listed as Unlimited, aren't predicable or available in the real world (I have OneDrive and there is certainly a limit for real subscribers and even the unlimited caps out way below 4TB which is nowhere near enough.)
Using some kind of drives that I ship to the US for uploading here is an option, but a painful one.
Ideally I want to sync files up while in Europe and not ship things physically back and forth unless absolutely necessary, especially since doing so would mean that I need a backup physically in Europe and someone testing media in the US and uploading on my behalf until I am able to get back and deal with it here.
A service like S3 would probably be fine, but I need a good way to deal with it if so. S3 is $28/TB with the data at rest, so that will get pretty expensive pretty quickly. Something more like OneDrive would be perfect.
Any ideas?
-
Should also be mentioned that just getting that data back out of S3 would cost $92.16 which is a ton to pay for this type of data transfer.
-
Glacier, in theory, would be enough as the time to get data out is not an issue.
-
How much over all data are you talking about? One to Two TB or more than 5TB?
-
1-4TB per month. So 12 - 48TB per year.
-
-
4K UHD raw video adds up FAST.
-
So, 50TB for a year,... how many years? Can you scale up to this limit?
If you will have limited bandwidth, shipping may be the 'best option' however that would mean needing TWO drives at a time - one to ship, one to retain as a safe guard...
And does all of this have to be online all the time? Or can you get away with the last 60 days being online...
-
I was thinking of a rack mounted server/NAS that you put in a portable rack with wheels/casters. Not unlike what you would put amps and other sound equipment when on the road.
You'd have to use a pile of SSDs, so it'd get expensive quickly.
That's all I've got.
-
@nadnerB said:
I was thinking of a rack mounted server/NAS that you put in a portable rack with wheels/casters. Not unlike what you would put amps and other sound equipment when on the road.
You'd have to use a pile of SSDs, so it'd get expensive quickly.
That's all I've got.
That is an option - However that would make travel rather interesting... Shipping and insurance could be cost preventative.
Not to mention the shock to the hardware could cause failure to any one of the drives.
-
@g.jacobse said:
So, 50TB for a year,... how many years? Can you scale up to this limit?
If you will have limited bandwidth, shipping may be the 'best option' however that would mean needing TWO drives at a time - one to ship, one to retain as a safe guard...
And does all of this have to be online all the time? Or can you get away with the last 60 days being online...
Needs to be online probably for a year or two at a time. Then can be archived. Might look at going to tape for long term archiving. Maybe even shorter term archiving. Maybe tape to ship is the right answer too. Tape is very resilient to shipping and storage. Speed of writing or recovery is a pretty minor thing. I would really prefer not to ship a tape drive around, though, nor to have to buy two high end ones.
-
@nadnerB said:
I was thinking of a rack mounted server/NAS that you put in a portable rack with wheels/casters. Not unlike what you would put amps and other sound equipment when on the road.
You'd have to use a pile of SSDs, so it'd get expensive quickly.
That's all I've got.
I'll be constantly on the move with only a few pieces of luggage, though, no way to have big equipment with me.
-
Something along these lines might be an answer but would only work once I have some way to stage ~3TB of data to store on the tape. Doesn't work as the intermediary format. Would work well for offloading older data from the "live" storage once it fills up.
-
You really know how to make things difficult don't you!?
Tape will definitely survive better than a HDD,.. SDD would be a better option, but they just are still a bit in the high cost range.
Something to consider is contacting a journalist and see what they might use... It is highly likely that someone else has already encountered this issue and found a solution which may fit your needs... Someone who does video-graphy for sports would be a first thought.
Shipping the tapes arent the issue,.. keeping the heads on the tape drive during transport would be. Unlikely that you will want to carry but so much in your carry-on bag...
One other thing of consideration would be electric - Though there are converters, one might know when they will have power....
-
Electric is generally easy these days. Everything is built to run internationally.
-
Something like the [Synology DS2413+](link url) might fit in well to a strategy. These hold some massive amounts of storage. With 6TB drives and RAID 6 that would be potentially 60TB without the expansion chassis. Although that is getting to be a crazy amount of storage on RAID 6 without going to enterprise class drives which drives the prices up a lot.
-
Yea,...I'd say that would do it.. With the option to expand out to 96TB that should cover it. Tape might still be an option for LTS.
Now - how do you plan to drag that around?
-
If you decide on the NAS, make sure the drives are easily accessible so that you can pull them out for transportation.
-
@nadnerB said:
If you decide on the NAS, make sure the drives are easily accessible so that you can pull them out for transportation.
Cannot transport a NAS. There is no possibility of moving a NAS around, it's a stationary target only.
-
Poking at Tape drives,.. wow.. I had not looked at them previously.. Not in a long time anyway. some of them are going for $1200 or more for a 1TB drive... and Tapes going for $25 each..
Tapes are cheap,.. but if the drive loses alignment that could be a serious issue.
To use SSD aren't going to be much cheaper.. and with limited and unreliable data connection, to try to dump a TB of data could be fun...
