Massive Storage Need for Video Project
-
@g.jacobse said:
So, 50TB for a year,... how many years? Can you scale up to this limit?
If you will have limited bandwidth, shipping may be the 'best option' however that would mean needing TWO drives at a time - one to ship, one to retain as a safe guard...
And does all of this have to be online all the time? Or can you get away with the last 60 days being online...
Needs to be online probably for a year or two at a time. Then can be archived. Might look at going to tape for long term archiving. Maybe even shorter term archiving. Maybe tape to ship is the right answer too. Tape is very resilient to shipping and storage. Speed of writing or recovery is a pretty minor thing. I would really prefer not to ship a tape drive around, though, nor to have to buy two high end ones.
-
@nadnerB said:
I was thinking of a rack mounted server/NAS that you put in a portable rack with wheels/casters. Not unlike what you would put amps and other sound equipment when on the road.
You'd have to use a pile of SSDs, so it'd get expensive quickly.
That's all I've got.
I'll be constantly on the move with only a few pieces of luggage, though, no way to have big equipment with me.
-
Something along these lines might be an answer but would only work once I have some way to stage ~3TB of data to store on the tape. Doesn't work as the intermediary format. Would work well for offloading older data from the "live" storage once it fills up.
-
You really know how to make things difficult don't you!?
Tape will definitely survive better than a HDD,.. SDD would be a better option, but they just are still a bit in the high cost range.
Something to consider is contacting a journalist and see what they might use... It is highly likely that someone else has already encountered this issue and found a solution which may fit your needs... Someone who does video-graphy for sports would be a first thought.
Shipping the tapes arent the issue,.. keeping the heads on the tape drive during transport would be. Unlikely that you will want to carry but so much in your carry-on bag...
One other thing of consideration would be electric - Though there are converters, one might know when they will have power....
-
Electric is generally easy these days. Everything is built to run internationally.
-
Something like the [Synology DS2413+](link url) might fit in well to a strategy. These hold some massive amounts of storage. With 6TB drives and RAID 6 that would be potentially 60TB without the expansion chassis. Although that is getting to be a crazy amount of storage on RAID 6 without going to enterprise class drives which drives the prices up a lot.
-
Yea,...I'd say that would do it.. With the option to expand out to 96TB that should cover it. Tape might still be an option for LTS.
Now - how do you plan to drag that around?
-
If you decide on the NAS, make sure the drives are easily accessible so that you can pull them out for transportation.
-
@nadnerB said:
If you decide on the NAS, make sure the drives are easily accessible so that you can pull them out for transportation.
Cannot transport a NAS. There is no possibility of moving a NAS around, it's a stationary target only.
-
Poking at Tape drives,.. wow.. I had not looked at them previously.. Not in a long time anyway. some of them are going for $1200 or more for a 1TB drive... and Tapes going for $25 each..
Tapes are cheap,.. but if the drive loses alignment that could be a serious issue.
To use SSD aren't going to be much cheaper.. and with limited and unreliable data connection, to try to dump a TB of data could be fun...
-
What is your expected time duration for each stop? Day, week, month....
-
@g.jacobse said:
What is your expected time duration for each stop? Day, week, month....
1-3 months. 2-2.5 months being likely average, but that will take time to determine. We have two 1.5 month stops lined up at the moment.
-
We are going to have a small scale, month long run through in San Francisco before being isolated in Europe.
-
Sounds like you have to work cut out... That is a lot of tech to try to stage before you head off.
If you know where you are staying, you could plan care packages with the supplies you need a head of time. Building the NAS up could be done over the course of the trip, With the biggest expense being the base unit.. Remote access would be nice to have after setup so that you can reach in and check on things.
Would you be able to purchase hardware while in Europe as to not ship from the US? How would that even work...
-
NAS has to be in the US. There is no possibility of storage devices larger than an external hard drive going to Europe. None.
All talk of storage devices like NAS, servers or cloud providers are on the US side. Only small scale devices like individual drives or possibly tapes at the extreme can be considered on the European side. Assume everything has to fit into a backpack along with all other gear.
-
@scottalanmiller said:
NAS has to be in the US. There is no possibility of storage devices larger than an external hard drive going to Europe. None.
All talk of storage devices like NAS, servers or cloud providers are on the US side. Only small scale devices like individual drives or possibly tapes at the extreme can be considered on the European side. Assume everything has to fit into a backpack along with all other gear.
agreed - And very understandable. NAS or other type of main storage would need to be US based.. But you also don't want to carry all the drives to off load the camera with you either as it adds to much weight, space and well.. insurance.
-
Services like DropBox don't even offer a service big enough, not even at $750/year (where they top out) does it provide what is needed.
-
I wonder if a huge ownCloud system would make sense.
-
@scottalanmiller does not even have to be huge. It at a minimum would only need to be big enough to handle the uploads.
Once data is uploaded, it could be offloaded elsewhere is that was more economical.
But, an ownCloud system would be my suggestion because you will not have artificial limits set on sizes by a third party.
-
ownCloud might be the answer on a SAM-SD. That way I can easily control the storage pools, use low cost where it makes sense and the only thing that it needs to do is deal with the sync process from ownCloud copying the files.
