offline, air-gapped backups / backup rotation (looking for hardware & ideas)
-
@scottalanmiller said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
I would even think an SSD setup would be more stable in this situation since write time and life time would be a lot better. I only mention spindle drives since its a big blob of data.
SSD is faster, and that helps, for sure. But the real issue is the physical connections and the RAID mechanism, not the drives themselves. Physical drives are a perfectly valid media for your use case. It's RAID being used as an archival mechanism rather than as a disaster avoidance mechanism that causes the problems both in software and in hardware.
Maybe I will just have to set up a network repository and simply plug the network cable in to let backup file copy to sync, then disconnect. That would probably be the easiest way to be honest.
I just wanted some mechanism that forced us to always have a full backup of data sitting offline/air-gapped... but F it lol
-
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@travisdh1 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
In its simplest form, I am looking to add offline/rotated backups to our 3-2-1 backup chain. I just want an offline copy as a final failsafe.
My thought is to get a server or NAS appliance with 2 x 15TB+ drives in a RAID1 which would act as a backup repository for ALL backups, and then have a 3rd drive with which to rotate out with one of the RAID1 pairs. Basically we'd pull out 1 drive and insert the free one and let the mirror complete and then swap it out again the next day, back and forth. This way, there would always be an air-gapped drive with a full copy of all our backups.
The only issues are mirror write-time for ~10TB and actually making sure the mirroring is automatic.
My company used to have a BNAS appliance from Highly Reliable which did just this, and it seemed to do an ok job, except mirror times were pretty long. This was 5+ years ago though.
Any ideas? I'm just trying to get the ideas flowing. I'm sure I could probably do a custom server build for something like this if I have to.
Whatever you do, mucking with the RAID is not what you want to be doing. That is something that would guarantee data loss due to someone entering a setting wrong.
I wouldn't call it mucking with RAID. Its just drive rotations.
Each time you add/remove a drive from the array, you chance clicking the wrong drive, the wrong action, etc. I'd call it unnecessary, mucking about where you shouldn't, and very risky as a few of the more friendly terms.
Why do you feel like the removable drives need to be a part of the RAID array?
-
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@scottalanmiller said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
I would even think an SSD setup would be more stable in this situation since write time and life time would be a lot better. I only mention spindle drives since its a big blob of data.
SSD is faster, and that helps, for sure. But the real issue is the physical connections and the RAID mechanism, not the drives themselves. Physical drives are a perfectly valid media for your use case. It's RAID being used as an archival mechanism rather than as a disaster avoidance mechanism that causes the problems both in software and in hardware.
Maybe I will just have to set up a network repository and simply plug the network cable in to let backup file copy to sync, then disconnect. That would probably be the easiest way to be honest.
I just wanted some mechanism that forced us to always have a full backup of data sitting offline/air-gapped... but F it lol
Yeah, it’s called tape. And it’s $8k price tag.
-
One of my clients does what the OP wants.
They bought 5 single drive NAS boxes... the backup software writes to the designated drive each night.
In the morning, they unplug it and take it home...Not great but it is cheap, In comparison
-
@travisdh1 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@travisdh1 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
In its simplest form, I am looking to add offline/rotated backups to our 3-2-1 backup chain. I just want an offline copy as a final failsafe.
My thought is to get a server or NAS appliance with 2 x 15TB+ drives in a RAID1 which would act as a backup repository for ALL backups, and then have a 3rd drive with which to rotate out with one of the RAID1 pairs. Basically we'd pull out 1 drive and insert the free one and let the mirror complete and then swap it out again the next day, back and forth. This way, there would always be an air-gapped drive with a full copy of all our backups.
The only issues are mirror write-time for ~10TB and actually making sure the mirroring is automatic.
My company used to have a BNAS appliance from Highly Reliable which did just this, and it seemed to do an ok job, except mirror times were pretty long. This was 5+ years ago though.
Any ideas? I'm just trying to get the ideas flowing. I'm sure I could probably do a custom server build for something like this if I have to.
Whatever you do, mucking with the RAID is not what you want to be doing. That is something that would guarantee data loss due to someone entering a setting wrong.
I wouldn't call it mucking with RAID. Its just drive rotations.
Each time you add/remove a drive from the array, you chance clicking the wrong drive, the wrong action, etc. I'd call it unnecessary, mucking about where you shouldn't, and very risky as a few of the more friendly terms.
Not if you follow the correct steps and know what you are doing. Its not anything terribly difficult.
Why do you feel like the removable drives need to be a part of the RAID array?
Because the RAID1 mirror would write the changes to the swapped disk.
The idea is to have Drive 1 and Drive X in a RAID1 mirror, where Drive X = both drives 2 and 3 which would be swapped out daily.
-
Nothing is quite as simple as a usb drive that has last week's backups....
/Sarcasm
-
@dashrender said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
One of my clients does what the OP wants.
They bought 5 single drive NAS boxes... the backup software writes to the designated drive each night.
In the morning, they unplug it and take it home...Not great but it is cheap, In comparison
That's a little different, right? Not using the RAID, but abusing the hot swap bays.
-
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
The idea is to have Drive 1 and Drive X in a RAID1 mirror, where Drive X = both drives 2 and 3 which would be swapped out daily.
The idea makes sense, just none of the parts are designed to be used in that fashion.
-
@travisdh1 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
Each time you add/remove a drive from the array, you chance clicking the wrong drive, the wrong action, etc. I'd call it unnecessary, mucking about where you shouldn't, and very risky as a few of the more friendly terms.
That too, it does depend on fencing against human error a bit more than a normal system. There are better, more efficient, more reliable ways to handle this where that isn't the case.
-
To do basically the same thing, what you want is a NAS with local storage (with or without RAID, in this case you are without RAID even though you are using RAID, so no need to have RAID at all) and having a hot swap drive in a mechanism meant to handle this, like a USB style drive, and a script that does a file copy of just the backup, not a block mirror of the drives, to copy the backup to the second drive.
-
@scottalanmiller said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
To do basically the same thing, what you want is a NAS with local storage (with or without RAID, in this case you are without RAID even though you are using RAID, so no need to have RAID at all) and having a hot swap drive in a mechanism meant to handle this, like a USB style drive, and a script that does a file copy of just the backup, not a block mirror of the drives, to copy the backup to the second drive.
Actually, I just remembered that with the Highly Reliable system, they had Windows software RAID 1 which did a good job in this kind of setup. Yes, its not perfect or ideal, but given that I have stated that I already have thorough backups and am only seeking to add offline/air-gapped copies as an added precaution, I don't think its that big an issue.
-
@dashrender said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@scottalanmiller said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
I would even think an SSD setup would be more stable in this situation since write time and life time would be a lot better. I only mention spindle drives since its a big blob of data.
SSD is faster, and that helps, for sure. But the real issue is the physical connections and the RAID mechanism, not the drives themselves. Physical drives are a perfectly valid media for your use case. It's RAID being used as an archival mechanism rather than as a disaster avoidance mechanism that causes the problems both in software and in hardware.
Maybe I will just have to set up a network repository and simply plug the network cable in to let backup file copy to sync, then disconnect. That would probably be the easiest way to be honest.
I just wanted some mechanism that forced us to always have a full backup of data sitting offline/air-gapped... but F it lol
Yeah, it’s called tape. And it’s $8k price tag.
Yeah I'm not doing tape and I think the alternate mechanism I proposed is roughly fine, depending on how its approached.
-
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
Actually, I just remembered that with the Highly Reliable system, they had Windows software RAID 1 which did a good job in this kind of setup.
RAID 1 has to do a block copy of a disk in use and track changes. It works, but isn't an efficient way to do this kind of workload. And most RAID has to mirror the entire drive, not just the portions with data. So in some cases it can be pretty dramatically slower than alternative methods.
ANd it's not like you want a mirror in the end. You just want a file copy. So the overhead of the RAID system doesn't provide benefits.
-
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
Yes, its not perfect or ideal, but given that I have stated that I already have thorough backups and am only seeking to add offline/air-gapped copies as an added precaution, I don't think its that big an issue.
The biggest issue is the hardware. How do you plan to connect and reconnect drives because no business class system that does RAID is meant for this to happen. So you either use business class devices that get abused and aren't expected to remain reliable. Or you use consumer gear to get the hotswap portion but don't have overall good hardware.
It can be done, everyone suggests doing it, and there is a reason that it's considered a horrible idea that should never be done. Trust me, there are simple, better ways to do something similar, rule this out and never think about it again. RAID is close to, but not the actual correct tool. The idea of copying the data to another drive is good, but RAID isn't a file copy and that's the underlying problem... this is triggering a disaster recovery mechanism designed for something totally different.
-
Don't know if still viable but Dell used to have and RDX drive that took hard drive medium instead of tape. Was OEM from either Quantum or Tandberg can't remember who and Google Fu isn't working for me
-
-
@scottalanmiller said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dashrender said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
One of my clients does what the OP wants.
They bought 5 single drive NAS boxes... the backup software writes to the designated drive each night.
In the morning, they unplug it and take it home...Not great but it is cheap, In comparison
That's a little different, right? Not using the RAID, but abusing the hot swap bays.
no, no hotswap anything... these are off the shelf WD self contained NASs.
-
@scottalanmiller said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
Yes, its not perfect or ideal, but given that I have stated that I already have thorough backups and am only seeking to add offline/air-gapped copies as an added precaution, I don't think its that big an issue.
The biggest issue is the hardware. How do you plan to connect and reconnect drives because no business class system that does RAID is meant for this to happen. So you either use business class devices that get abused and aren't expected to remain reliable. Or you use consumer gear to get the hotswap portion but don't have overall good hardware.
It can be done, everyone suggests doing it, and there is a reason that it's considered a horrible idea that should never be done. Trust me, there are simple, better ways to do something similar, rule this out and never think about it again. RAID is close to, but not the actual correct tool. The idea of copying the data to another drive is good, but RAID isn't a file copy and that's the underlying problem... this is triggering a disaster recovery mechanism designed for something totally different.
yeah good points.. I just wanted to entertain the idea by posting here and have you guys sway me... a more attractive idea that I had been mulling around was basically a Veeam copy job to a repository with a scripted on/off network connectivity switch on a schedule. That or I just manually plug and unplug the network cable as I mentioned above. LMAO hey it would technically work.
-
What is the point of all of this? Crypto does not affect backups. That is why they are backups. They are static.
If you are worried about your backup being encrypted, then don't use a common access. Only give the the Veeam credentials with write access to the backup storage location.
Use Veeam to write to B2 or something similar.
-
The possible solutions are of course going to depend on what the initial backup repository is that you're looking to copy off to this air-gapped system. Jared mentions Veeam but I couldn't spot the OP indicating that he's using Veeam, and if yes, is it B&R for hypervisors or the agent individually installed on endpoints or are we only looking to backup a single server? I only raise the point because the veeam windows agent provides a mechanism to automatically mount and unmount the backup target between runs.
