Marketing - Video Editing Storage
-
Hi all,
I am being asked to find a storage solution for our video editing department. They use various Adobe tools on Mac clients.
They have around 40TB of files they use, with the largest files being RAW files of 300 GB - 400 GB. Average file sizes are 10 GB - 20 GB.
1/2 of this is archive and can be stored on cloud storage for pull down should they require again. Leaving around 20TB live data and 20TB in archive.
Originally, I was looking at proposing a 20 - 30 TB NAS populated with SSDs in the local office, with 10 Gbps NIC. This would provide high speed local access over the LAN to 6 marketing users.
Marketing said speed was the main concern. Accessing the large files is currently slow causing many delays. They have so far been sharing USB 3.0 devices between each other, without backups.
Our CIO is now pushing for cloud only solutions for the storage where marketing can check in/out the files they need, killing my NAS idea. I have concerns on this but am open, so would like some advice. What have you used or what would you suggest to use to provide this?
I am concerned that when an editor wants to do something with the 300 GB file, that will have to pull down from the cloud, bringing our WAN link to a crawl until the download finishes. Then, once they have finished editing, they have to upload the end product, again causing bandwidth issues. This is exacerbated shoudl multiple editors be pulling multiple files at the same time. Even at 500 Mbps with no overhead that is around 1.5h of time to sit and wait for a 300 GB file whilst everybody else is affected on the same WAN link.
I assume the'video editing' cloud storage providers are like OneDrive or DropBox, where each machine has a local cache so do not have to pull from cloud each time. But, wouldnt that mean that our video editing workstations all need to have a 20 TB local drive to store this local cache?
Kind of all over the place on this one, any ideas folks?
-
Taking into account that you must opt for a cloud solution
The best thing would be to go for Private Cloud.
You may also consider:
https://www.blackbird.video/
https://www.adobe.com/products/premiere-rush.html -
@Jimmy9008 said in Marketing - Video Editing Storage:
Hi all,
I am being asked to find a storage solution for our video editing department. They use various Adobe tools on Mac clients.
They have around 40TB of files they use, with the largest files being RAW files of 300 GB - 400 GB. Average file sizes are 10 GB - 20 GB.
1/2 of this is archive and can be stored on cloud storage for pull down should they require again. Leaving around 20TB live data and 20TB in archive.
Originally, I was looking at proposing a 20 - 30 TB NAS populated with SSDs in the local office, with 10 Gbps NIC. This would provide high speed local access over the LAN to 6 marketing users.
Marketing said speed was the main concern. Accessing the large files is currently slow causing many delays. They have so far been sharing USB 3.0 devices between each other, without backups.
Our CIO is now pushing for cloud only solutions for the storage where marketing can check in/out the files they need, killing my NAS idea. I have concerns on this but am open, so would like some advice. What have you used or what would you suggest to use to provide this?
I am concerned that when an editor wants to do something with the 300 GB file, that will have to pull down from the cloud, bringing our WAN link to a crawl until the download finishes. Then, once they have finished editing, they have to upload the end product, again causing bandwidth issues. This is exacerbated shoudl multiple editors be pulling multiple files at the same time. Even at 500 Mbps with no overhead that is around 1.5h of time to sit and wait for a 300 GB file whilst everybody else is affected on the same WAN link.
I assume the'video editing' cloud storage providers are like OneDrive or DropBox, where each machine has a local cache so do not have to pull from cloud each time. But, wouldnt that mean that our video editing workstations all need to have a 20 TB local drive to store this local cache?
Kind of all over the place on this one, any ideas folks?
You need to present the math for the CIO.
It's unrealistic to assume you can use your entire bandwith for one download. It's also completely unrealistic to assume you can get 500 Mbps sustained from whatever cloud storage you have.
I would say you're lucky to have 100 Mbit/s sustained. So 10MB/s of data and a little over 8 hours for a 300GB file. 50 Mbit/s is however a more realistic number.
Next up is the maximum file size limits on cloud storage. Microsoft for example is 250GB maximum on Onedrive and Sharepoint. In other words - you can't store your video files there.
Using cloud for archival storage you may never need is fine. But for working files that the editing team is sharing, you must have those files on the LAN.
You should run a test. Sign up for a cloud storage solution and take one of those 400GB RAW files and upload it. And then download it. See what happens and how long it takes.
-
@Jimmy9008 said in Marketing - Video Editing Storage:
I assume the'video editing' cloud storage providers are like OneDrive or DropBox, where each machine has a local cache so do not have to pull from cloud each time. But, wouldnt that mean that our video editing workstations all need to have a 20 TB local drive to store this local cache?
When you use on-demand sync the cache is based on files you need access to. So local storage on the workstation only has to be big enough to hold the files you are currently working on.
-
The only way to make a cloud only video editing solution work is to use a VDI solution.
Basically you have a LAN in the cloud where your editing workstations, storage and backups are located. Your editors remote control their workstations to do the work.
All files that are shot or produced need to be uploaded first before any editing can be done.
-
Sounds like all the bases have been covered.
It's unrealistic to work with files of that size (even 10 GB files are unrealistic to work on from cloud).
As Pete said - show the owners the time it takes to move the data, then show them the cost of a potential VDI solution with hosted storage - your solution of a NAS will quickly be back on the table with backups to cloud.
-
What kind of marketing department is this? A movie firm? It seems insane to have video files average 10-20gb. Even high budget commercials are probably only that size.. Is this a bunch of templates or something?
I would be interested in understanding their work flow, as it seems very extravagant to have such large files and so many of them?
-
@IRJ said in Marketing - Video Editing Storage:
What kind of marketing department is this? A movie firm? It seems insane to have video files average 10-20gb. Even high budget commercials are probably only that size.. Is this a bunch of templates or something?
I would be interested in understanding their work flow, as it seems very extravagant to have such large files and so many of them?
Raw video is easily that large. It's not final cut and transcode that it becomes manageable.
-
@Jimmy9008 said in Marketing - Video Editing Storage:
I am being asked to find a storage solution for our video editing department. They use various Adobe tools on Mac clients.
Sounds like my every day life. I use MacOS and Apple editing products and produce an insane amount of material daily.
-
@Jimmy9008 said in Marketing - Video Editing Storage:
Originally, I was looking at proposing a 20 - 30 TB NAS populated with SSDs in the local office, with 10 Gbps NIC. This would provide high speed local access over the LAN to 6 marketing users.
Marketing said speed was the main concern. Accessing the large files is currently slow causing many delays. They have so far been sharing USB 3.0 devices between each other, without backups.Traditionally, this is one of the use cases for SAN. Large post processing shops (this is marketing, but more or less the same use case) typically use high speed SAN with clustered file systems to deal with this issue.
-
@Jimmy9008 said in Marketing - Video Editing Storage:
Our CIO is now pushing for cloud only solutions for the storage where marketing can check in/out the files they need, killing my NAS idea. I have concerns on this but am open, so would like some advice. What have you used or what would you suggest to use to provide this?
I am concerned that when an editor wants to do something with the 300 GB file, that will have to pull down from the cloud, bringing our WAN link to a crawl until the download finishes. Then, once they have finished editing, they have to upload the end product, again causing bandwidth issues. This is exacerbated shoudl multiple editors be pulling multiple files at the same time. Even at 500 Mbps with no overhead that is around 1.5h of time to sit and wait for a 300 GB file whilst everybody else is affected on the same WAN link.Yes, that would be a huge performance bottleneck. I have a similar situation here. I do this a lot and it is only me, a single user. I end up waiting most of a day for files to upload and download. I literally have my other thread on here discussing how to move files to prep them for final publication because this issue is so extreme.
-
@Jimmy9008 said in Marketing - Video Editing Storage:
I assume the'video editing' cloud storage providers are like OneDrive or DropBox, where each machine has a local cache so do not have to pull from cloud each time. But, wouldnt that mean that our video editing workstations all need to have a 20 TB local drive to store this local cache?
Kind of all over the place on this one, any ideas folks?Yes, in order to have the files ready all of the machines would need to keep a copy AND would need to download changes as they happen.
-
@Pete-S said in Marketing - Video Editing Storage:
It's unrealistic to assume you can use your entire bandwith for one download. It's also completely unrealistic to assume you can get 500 Mbps sustained from whatever cloud storage you have.
My cloud storage from a video provider gives me about 65Mb/s in real world bandwidth, if that helps.
-
@IRJ said in Marketing - Video Editing Storage:
What kind of marketing department is this? A movie firm? It seems insane to have video files average 10-20gb. Even high budget commercials are probably only that size.. Is this a bunch of templates or something?
Check your Olympus EM-1 MK2 for video, it'll produce that amount fast (but breaks up the files, but just imagine that it kept them as a single file like newer cameras do.)
If you are shooting with a Fuji XH2 or similar, you'll produce 20GB files every few minutes.
I'm just on a GoPro 11 and all my files are 10GB because that's its file limit. If the GoPro kept the files together, I'd routinely have 20GB - 40GB files. And that's low bit rate compared to serious cameras. Imagine if you shot on Panasonic GH6 or better, especially if you go to 6K or 8K! And I'm using H.265 which is tiny. If you shot on ProRes, which is what most do, the file sizes will explode quickly. 100GB or more would be nothing.
-
@Jimmy9008 said in Marketing - Video Editing Storage:
Originally, I was looking at proposing a 20 - 30 TB NAS populated with SSDs in the local office, with 10 Gbps NIC. This would provide high speed local access over the LAN to 6 marketing users.
If their PCs accessing a NAS at 1-10Gbps isn't good enough because their primary concern is speed, why would they push for way slower cloud storage, assuming no on-prem cache?
-
@Obsolesce said in Marketing - Video Editing Storage:
@Jimmy9008 said in Marketing - Video Editing Storage:
Originally, I was looking at proposing a 20 - 30 TB NAS populated with SSDs in the local office, with 10 Gbps NIC. This would provide high speed local access over the LAN to 6 marketing users.
If their PCs accessing a NAS at 1-10Gbps isn't good enough because their primary concern is speed, why would they push for way slower cloud storage, assuming no on-prem cache?
I archive my video in the cloud, but I would not want to work from it without a local cache.
-
@Obsolesce said in Marketing - Video Editing Storage:
@Jimmy9008 said in Marketing - Video Editing Storage:
Originally, I was looking at proposing a 20 - 30 TB NAS populated with SSDs in the local office, with 10 Gbps NIC. This would provide high speed local access over the LAN to 6 marketing users.
If their PCs accessing a NAS at 1-10Gbps isn't good enough because their primary concern is speed, why would they push for way slower cloud storage, assuming no on-prem cache?
1 - 10 Gbps would be more than fine. That is what I proposed. But, the CIO is asking that the storage is Cloud only. Leading to this issue where the editing workstations are in office and the storage is remote.
-
@scottalanmiller said in Marketing - Video Editing Storage:
@IRJ said in Marketing - Video Editing Storage:
What kind of marketing department is this? A movie firm? It seems insane to have video files average 10-20gb. Even high budget commercials are probably only that size.. Is this a bunch of templates or something?
Check your Olympus EM-1 MK2 for video, it'll produce that amount fast (but breaks up the files, but just imagine that it kept them as a single file like newer cameras do.)
If you are shooting with a Fuji XH2 or similar, you'll produce 20GB files every few minutes.
I'm just on a GoPro 11 and all my files are 10GB because that's its file limit. If the GoPro kept the files together, I'd routinely have 20GB - 40GB files. And that's low bit rate compared to serious cameras. Imagine if you shot on Panasonic GH6 or better, especially if you go to 6K or 8K! And I'm using H.265 which is tiny. If you shot on ProRes, which is what most do, the file sizes will explode quickly. 100GB or more would be nothing.
I've not seen the files first hand, am just told these are the sizes by that team. They are telling me the cameras will do 40 minutes to 1 hour recordings of presentations and stuff like that at 4k and 8k, and they are usually 300 GB - 400 GB.
-
@Dashrender said in Marketing - Video Editing Storage:
Sounds like all the bases have been covered.
It's unrealistic to work with files of that size (even 10 GB files are unrealistic to work on from cloud).
As Pete said - show the owners the time it takes to move the data, then show them the cost of a potential VDI solution with hosted storage - your solution of a NAS will quickly be back on the table with backups to cloud.
I have been doing some research and am interested in this company, they seem to have both storage in cloud and editing via browser. Any thoughts on this, or better alternatives?
Does anybody know if Adobe has a similar entirely cloud solution for video editing? When I go to their site I find it a mess and confusing.
-
This looks like another option, although, it does just look like a NAS to me, just through a specific 'media' vendor.
