Incorporating Ransomware Protection into Backup Plan
-
Several years ago Scott gave me some solid advice on arrays & I thought it was worth checking in on what the people here do for incorporating ransomware protection into their backups when online backup services aren’t an option. In the last 3 months we’ve had two customers get hit by ransomware. This is my nightmare scenario & it’s time to spend some resources building a more comprehensive protection plan.
We have two servers. A production server that has 30TB / 30,000,000 files with 1 - 2 TB that rotates on / off weekly basis. This server also has a 30GB production SQL Database.
The second server is our internal company resources, a domain controller / DNS, SVN etc. It's on a VM.
I have a local backup plan - it’s not high tech & runs on a weekly basis. But we can survive any type of hardware failure. I have intentionally avoided daily / hourly backups due to ransomware concerns.
I don’t fully understand or trust incremental / incremental + block level backups. We do use them for local computers on remote services (Cloudberry + Backblaze) which is fine. But a local machine has a lot fewer changes than what I’m dealing with.
My understanding is the processing time of calculating differences is likely to be difficult for this size & this many files - as well as the restore. I like being able to look at / open / copy files.
From my limited research - I’ve come to four broad options that are something like
Large incremental backup array with an airgap only coming online for backups (Possibly manually done on a weekly basis). Phrases like “incremental backup decreases recovery reliability.” terrify me because I’ve been the brunt of “it should have worked but didn’t” in this exact scenario (although this was years ago).
I know that if the network doesn’t have authorization - ransomware shouldn’t be able to touch it - but I like 'can't' more than 'shouldn't' - (Tape Drives or smaller self contained arrays that have even less connection time to the network).
I’m also skeptical about the processing time for 30 mil small files. (Maybe I’m wrong about this?) This would have to include a software solution (Veeam, Cloudberry, HP StoreEasy has free Carbonite software)
$6-8K hardware / software ?Tape drives
I’ve been trying to read on these. Some people have said IT moved away from them because they weren’t reliable, slow to retrieve individual files etc. This seems like it might be a workable solution but I need to know if it’s going to handle 30mil files on a weekly basis - & if not then it may be a dead end solution already.
$8K-12K hardware / software / tapes4 Small(er) weekly backup arrays. Weekly rotating backup that involves physically connecting to the network, starting the backup & unplugging when done. Configuration would probably be 4 DL380 G8s + 48 used/refurbished HGST He8 drives in a small cabinet with a KVM & an actual person weekly backup task (only 1 computer would touch the network per week)
$8K hardware + softwareOther solutions - OneBlox? I know people over at Spiceworks were big fans of OneBlox which seems to have morphed into StorageCraft. I think OneXafe is what I’d be looking at which seems to be a Hardware + Software in a box solution.
OneXafe $15K - $40KMy biggest concern is investing a substantial sum of money & time in a solution I’m sold that tries to do more complex things than I need (realtime protection, instant restore, cloud backup etc), requires a steeper learning curve & then winds up not working out when it’s needed the most.
I don’t mind spending what’s needed for a safe solution. But I bend towards simplicity as only 20% of my time is spent on infrastructure & I’d prefer to keep it that way. Given all this - in my shoes what direction would you look for backups that incorporate ransomware safety & why?
-
@jim9500 30TB is about the max for a single LTO-8 tape. Don't let the bad old days (and probably consumer type cruddy tape drives) put you off of tape. It's more reliable than any other media when you need to transport it, and you will want something off-site to be protected from ransomware.
Lot's more detail others will go into before I can get back here and complete my thoughts I'm sure.
-
The first thing you need to figure out is what you’re actually trying to fight and protect against.
Your back ups are your protection against ransomware.
If you were trying to protect your back ups from being encrypted well that means that you were first using a bad set up that would allow some device to even have access to encrypt your back up target.
The device performing the backups, will go with Veeam as the solution, connect to the back up target with an account that is only used for this there’s no other way to access the back up target except with this account and those credentials that are only used within Veeam.
To date there is nothing that will compromise Veeam itself to get the password to your storage target that could then use that information and encrypt your storage target.
-
First, ransomware is big business run by organized crime. I think about 19 billion dollar per year industry.
Everything can be compromised in different ways. There is just no way to protect your data 100% and to think otherwise is just naive.
We have chosen to go with tape as our last line of defense. Once you take it offline there is no way it can be remotely compromised. We believe that is enough to be able to recover from most attacks and the cost is reasonable.
-
@jim9500 said in Incorporating Ransomware Protection into Backup Plan:
I have a local backup plan - it’s not high tech & runs on a weekly basis. But we can survive any type of hardware failure. I have intentionally avoided daily / hourly backups due to ransomware concerns.
Frequency of backups doesn't affect ransomware risk. It's retention period that does.
-
@jim9500 said in Incorporating Ransomware Protection into Backup Plan:
I don’t fully understand or trust incremental / incremental + block level backups.
Basically they record changes since the last full backup. So if you back up ten files today. Then one file changes. And you take an incremental backup, you only additionally back up the one file that changed, not an additional backup of the ones that didn't change that you already have backed up.
-
@jim9500 said in Incorporating Ransomware Protection into Backup Plan:
Large incremental backup array with an airgap only coming online for backups (Possibly manually done on a weekly basis). Phrases like “incremental backup decreases recovery reliability.” terrify me because I’ve been the brunt of “it should have worked but didn’t” in this exact scenario (although this was years ago).
Incrementals don't have real risks like that. If you've been burnt in the past, you need to identify what actually failed. Incremental backups are trusted by every major company and no one worries about them failing. Incremental failures and almost exclusively caused by people who literally lose their backups and can't find them. If you take a full, and lose it, it'll be gone, too.
-
@jim9500 said in Incorporating Ransomware Protection into Backup Plan:
Large incremental backup array with an airgap only coming online for backups (Possibly manually done on a weekly basis).
Not really an air gap in the situation. If it is online ever, it's not air gapped. It's a nice additional precaution, but don't consider it an air gap.
-
@jim9500 said in Incorporating Ransomware Protection into Backup Plan:
My understanding is the processing time of calculating differences is likely to be difficult for this size & this many files - as well as the restore. I like being able to look at / open / copy files.
Sure, but the time to back them up is there, too. So that's not a consideration. There is no reasonable reason to avoid incrementals / differentials. If you can do what you need using nothing but Fulls, that's perfectly fine. But don't think that there is any technical reason to not use them, they are completely safe and fast.
-
@jim9500 said in Incorporating Ransomware Protection into Backup Plan:
I know that if the network doesn’t have authorization - ransomware shouldn’t be able to touch it - but I like 'can't' more than 'shouldn't' - (Tape Drives or smaller self contained arrays that have even less connection time to the network).
If you can back up to it, ransomware can get to it. The two go together. You can't separate the two.
-
@jim9500 said in Incorporating Ransomware Protection into Backup Plan:
I’m also skeptical about the processing time for 30 mil small files. (Maybe I’m wrong about this?) This would have to include a software solution (Veeam, Cloudberry, HP StoreEasy has free Carbonite software)
Not all backups look at individual files. Many, maybe most, look at blocks not files, and just see a single file system to back up and have no concern for the individual files.
-
@jim9500 said in Incorporating Ransomware Protection into Backup Plan:
Some people have said IT moved away from them because they weren’t reliable, slow to retrieve individual files etc.
Those people aren't in IT and are out of touch with the universe. Tapes are still the "go to" backup medium for companies bigger than the SMB. Only in the SMB where loads of myths tend to take hold does anyone think this. Tape is the most reliable, fastest overall speed. Yes, tapes are slow for individual files, but if you need your backup media to retrieve individual files in most cases, your system is poorly designed.
First, most companies rarely need individual files restored. If this is coming up with any frequency, something is wrong. And essentially any serious backup solution has a way to retrieve nearly any file without going to the final backup media. There is almost always a cache layer between the live systems and the untouchable backup storage that allows for screaming fast file recovery without needing to retrieve the archival media whether tape, disk array, or cloud storage.
-
@jim9500 said in Incorporating Ransomware Protection into Backup Plan:
4 Small(er) weekly backup arrays. Weekly rotating backup that involves physically connecting to the network, starting the backup & unplugging when done. Configuration would probably be 4 DL380 G8s + 48 used/refurbished HGST He8 drives in a small cabinet with a KVM & an actual person weekly backup task (only 1 computer would touch the network per week)
$8K hardware + softwareAnything that involves "weird" solutions like connecting arrays or networks when needed should be ruled out. One of the most dangerous things you can do is try to reinvent the wheel when big business has had trillions of dollars of research and decades of experience and knows how to do backups well. Learn from the industry, don't try to work around it. Nothing in IT should work this way, it's an established industry with deep knowledge. Just make sure you follow good IT practices and not sales people trying to make a quick buck at your expense. But at no point should you feel that you are engineering unheard of solutions that no one else has or needs, guaranteed that means something is being overlooked or misunderstood.
Rule all of this stuff out right away. Tape, Write Once, and similar solutions are what you use. Nothing else.
-
@jim9500 said in Incorporating Ransomware Protection into Backup Plan:
Other solutions - OneBlox? I know people over at Spiceworks were big fans of OneBlox which seems to have morphed into StorageCraft. I think OneXafe is what I’d be looking at which seems to be a Hardware + Software in a box solution.
OneXafe $15K - $40KStorageCraft bought them because they wanted a write once hardware platform to target backups to. Not cheap, but a great product.
-
@jim9500 said in Incorporating Ransomware Protection into Backup Plan:
We have two servers. A production server that has 30TB / 30,000,000 files with 1 - 2 TB that rotates on / off weekly basis. This server also has a 30GB production SQL Database.
At this size, tape is the answer with extremely rare exception. A solid D2D2T stock "by the book" solution is the best for almost everyone, and for you, too. It's affordable and solves every concern that you have. You have disk for rapid file restores, and tape to protect against ransomware.
-
@scottalanmiller said in Incorporating Ransomware Protection into Backup Plan:
A solid D2D2T stock "by the book" solution
This is exactly the method I've used to provide a solid backup solution for a similar situation. A total of ~45 TB of data with multiple SQL database and VMs. The DBs were backed up via built-in methods, but were also backed up "as a whole" as part of the VM backup (but that's besides the point).
This gave quick and efficient local on-prem backup and restores, and also allowed for off-site rotation.
Daily incrementals were done on-prem, whcih was quick using veeam with it's block change tracking.
I did NOT use their synthetic fulls because those took ridiculously long and and on top of that it just seemed like a very volatile process at those sizes, because the daily incrementals could be TB+ sizes. So daily incerementals to on-prem backup repo, weekly fulls to on-prem backup repo, monthly backup repo to tape to off-site. There were 3 or 4 tape sets, so that allowed nearly 6 months of retention of daily backups. Some of the DBs were backed up via built-in methods so because of that we also had hourly DB backups for some DBs for ~6 months (rougly speaking).
And yes, do pay the fee to bring in a tape from off-site to test restore a production system and some data in a test environment. I did this a couple times with success, but you never know.
-
@scottalanmiller said in Incorporating Ransomware Protection into Backup Plan:
D2D2T
Appreciate all of the input. This is the solution I've been leaning towards over the last week. Had an infrastructure hiccup & haven't been able to spend any time on this. But I will utilize my existing backup device for the backup disk & incorporate standard LTO-8 drive library with a rotating weekly offsite storage.
