offline, air-gapped backups / backup rotation (looking for hardware & ideas)
-
In its simplest form, I am looking to add offline/rotated backups to our 3-2-1 backup chain. I just want an offline copy as a final failsafe.
My thought is to get a server or NAS appliance with 2 x 15TB+ drives in a RAID1 which would act as a backup repository for ALL backups, and then have a 3rd drive with which to rotate out with one of the RAID1 pairs. Basically we'd pull out 1 drive and insert the free one and let the mirror complete and then swap it out again the next day, back and forth. This way, there would always be an air-gapped drive with a full copy of all our backups.
The only issues are mirror write-time for ~10TB and actually making sure the mirroring is automatic.
My company used to have a BNAS appliance from Highly Reliable which did just this, and it seemed to do an ok job, except mirror times were pretty long. This was 5+ years ago though.
Any ideas? I'm just trying to get the ideas flowing. I'm sure I could probably do a custom server build for something like this if I have to.
-
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
Basically we'd pull out 1 drive and insert the free one and let the mirror complete and then swap it out again the next day, back and forth.
This is one of those "being weird" situations that comes up a lot. Breaking RAID to try to do a disk copy is a very bad idea. The software isn't built for this, the hardware isn't build for this. It's something everyone has thought of, and no one that has tried it is happy with the results.
RAID is never meant to be intentionally broken as part of the production process. If you want a full copy, make a copy using copy tools and hardware meant to be plugged and unplugged regularly. Hot swap bays are designed to have drives replaced a handful of times in a lifetime.
-
If you want a hard drive as your air gapped backup, you need something that can be plugged and unplugged thousands of times before failing. Otherwise, consider tape which is meant specifically for this purpose.
-
@scottalanmiller said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
Basically we'd pull out 1 drive and insert the free one and let the mirror complete and then swap it out again the next day, back and forth.
This is one of those "being weird" situations that comes up a lot. Breaking RAID to try to do a disk copy is a very bad idea. The software isn't built for this, the hardware isn't build for this. It's something everyone has thought of, and no one that has tried it is happy with the results.
RAID is never meant to be intentionally broken as part of the production process. If you want a full copy, make a copy using copy tools and hardware meant to be plugged and unplugged regularly. Hot swap bays are designed to have drives replaced a handful of times in a lifetime.
Yeah that's kind of the general feeling I get. I hear a lot about having "air-gapped" backups now and I've just been trying to think how that would work without using tape.
-
@scottalanmiller said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
If you want a hard drive as your air gapped backup, you need something that can be plugged and unplugged thousands of times before failing. Otherwise, consider tape which is meant specifically for this purpose.
It wouldn't really need to be thousands of times... its not something so crucial. I'd probably order a couple extra drives to have on hand and replace them as needed or every couple of months or once per year. Theoretically the drives would only be removed M-F, so 310 times per year.
Its more of a low-ish priority type thing where as long as we have some form of recently taken offline backups just in case we have a total ransomware incident, then we're good. I already have multiple backups and replications spread out locally and with a cloud connect storage provider.
-
@scottalanmiller said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
If you want a hard drive as your air gapped backup, you need something that can be plugged and unplugged thousands of times before failing. Otherwise, consider tape which is meant specifically for this purpose.
I would even think an SSD setup would be more stable in this situation since write time and life time would be a lot better. I only mention spindle drives since its a big blob of data.
-
@dave247 You'll have a hard time doing a RAID 1 replication of a 16TB drive in less than 24 hours. You have to beat 185 MB/s continuously.
Worm tape sounds like the best option. Then you can just do a full backup and increments. Nobody can overwrite the data or erase it. It's standard LTO tape drives but with worm tapes.
Dell PowerVault TL1000 with LTO8 drive and LTO8 worm media will get the job done. Impossible to ransomware. No need to remove any tapes or do anything until they are full. One tape = 12TB of data or ~30TB of compressed data. Tape drive will compress and encrypt in real-time.
-
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
In its simplest form, I am looking to add offline/rotated backups to our 3-2-1 backup chain. I just want an offline copy as a final failsafe.
My thought is to get a server or NAS appliance with 2 x 15TB+ drives in a RAID1 which would act as a backup repository for ALL backups, and then have a 3rd drive with which to rotate out with one of the RAID1 pairs. Basically we'd pull out 1 drive and insert the free one and let the mirror complete and then swap it out again the next day, back and forth. This way, there would always be an air-gapped drive with a full copy of all our backups.
The only issues are mirror write-time for ~10TB and actually making sure the mirroring is automatic.
My company used to have a BNAS appliance from Highly Reliable which did just this, and it seemed to do an ok job, except mirror times were pretty long. This was 5+ years ago though.
Any ideas? I'm just trying to get the ideas flowing. I'm sure I could probably do a custom server build for something like this if I have to.
Whatever you do, mucking with the RAID is not what you want to be doing. That is something that would guarantee data loss due to someone entering a setting wrong.
-
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
I would even think an SSD setup would be more stable in this situation since write time and life time would be a lot better. I only mention spindle drives since its a big blob of data.
SSD is faster, and that helps, for sure. But the real issue is the physical connections and the RAID mechanism, not the drives themselves. Physical drives are a perfectly valid media for your use case. It's RAID being used as an archival mechanism rather than as a disaster avoidance mechanism that causes the problems both in software and in hardware.
-
@travisdh1 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
In its simplest form, I am looking to add offline/rotated backups to our 3-2-1 backup chain. I just want an offline copy as a final failsafe.
My thought is to get a server or NAS appliance with 2 x 15TB+ drives in a RAID1 which would act as a backup repository for ALL backups, and then have a 3rd drive with which to rotate out with one of the RAID1 pairs. Basically we'd pull out 1 drive and insert the free one and let the mirror complete and then swap it out again the next day, back and forth. This way, there would always be an air-gapped drive with a full copy of all our backups.
The only issues are mirror write-time for ~10TB and actually making sure the mirroring is automatic.
My company used to have a BNAS appliance from Highly Reliable which did just this, and it seemed to do an ok job, except mirror times were pretty long. This was 5+ years ago though.
Any ideas? I'm just trying to get the ideas flowing. I'm sure I could probably do a custom server build for something like this if I have to.
Whatever you do, mucking with the RAID is not what you want to be doing. That is something that would guarantee data loss due to someone entering a setting wrong.
I wouldn't call it mucking with RAID. Its just drive rotations.
-
@scottalanmiller said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
I would even think an SSD setup would be more stable in this situation since write time and life time would be a lot better. I only mention spindle drives since its a big blob of data.
SSD is faster, and that helps, for sure. But the real issue is the physical connections and the RAID mechanism, not the drives themselves. Physical drives are a perfectly valid media for your use case. It's RAID being used as an archival mechanism rather than as a disaster avoidance mechanism that causes the problems both in software and in hardware.
Maybe I will just have to set up a network repository and simply plug the network cable in to let backup file copy to sync, then disconnect. That would probably be the easiest way to be honest.
I just wanted some mechanism that forced us to always have a full backup of data sitting offline/air-gapped... but F it lol
-
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@travisdh1 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
In its simplest form, I am looking to add offline/rotated backups to our 3-2-1 backup chain. I just want an offline copy as a final failsafe.
My thought is to get a server or NAS appliance with 2 x 15TB+ drives in a RAID1 which would act as a backup repository for ALL backups, and then have a 3rd drive with which to rotate out with one of the RAID1 pairs. Basically we'd pull out 1 drive and insert the free one and let the mirror complete and then swap it out again the next day, back and forth. This way, there would always be an air-gapped drive with a full copy of all our backups.
The only issues are mirror write-time for ~10TB and actually making sure the mirroring is automatic.
My company used to have a BNAS appliance from Highly Reliable which did just this, and it seemed to do an ok job, except mirror times were pretty long. This was 5+ years ago though.
Any ideas? I'm just trying to get the ideas flowing. I'm sure I could probably do a custom server build for something like this if I have to.
Whatever you do, mucking with the RAID is not what you want to be doing. That is something that would guarantee data loss due to someone entering a setting wrong.
I wouldn't call it mucking with RAID. Its just drive rotations.
Each time you add/remove a drive from the array, you chance clicking the wrong drive, the wrong action, etc. I'd call it unnecessary, mucking about where you shouldn't, and very risky as a few of the more friendly terms.
Why do you feel like the removable drives need to be a part of the RAID array?
-
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@scottalanmiller said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
I would even think an SSD setup would be more stable in this situation since write time and life time would be a lot better. I only mention spindle drives since its a big blob of data.
SSD is faster, and that helps, for sure. But the real issue is the physical connections and the RAID mechanism, not the drives themselves. Physical drives are a perfectly valid media for your use case. It's RAID being used as an archival mechanism rather than as a disaster avoidance mechanism that causes the problems both in software and in hardware.
Maybe I will just have to set up a network repository and simply plug the network cable in to let backup file copy to sync, then disconnect. That would probably be the easiest way to be honest.
I just wanted some mechanism that forced us to always have a full backup of data sitting offline/air-gapped... but F it lol
Yeah, it’s called tape. And it’s $8k price tag.
-
One of my clients does what the OP wants.
They bought 5 single drive NAS boxes... the backup software writes to the designated drive each night.
In the morning, they unplug it and take it home...Not great but it is cheap, In comparison
-
@travisdh1 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@travisdh1 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
In its simplest form, I am looking to add offline/rotated backups to our 3-2-1 backup chain. I just want an offline copy as a final failsafe.
My thought is to get a server or NAS appliance with 2 x 15TB+ drives in a RAID1 which would act as a backup repository for ALL backups, and then have a 3rd drive with which to rotate out with one of the RAID1 pairs. Basically we'd pull out 1 drive and insert the free one and let the mirror complete and then swap it out again the next day, back and forth. This way, there would always be an air-gapped drive with a full copy of all our backups.
The only issues are mirror write-time for ~10TB and actually making sure the mirroring is automatic.
My company used to have a BNAS appliance from Highly Reliable which did just this, and it seemed to do an ok job, except mirror times were pretty long. This was 5+ years ago though.
Any ideas? I'm just trying to get the ideas flowing. I'm sure I could probably do a custom server build for something like this if I have to.
Whatever you do, mucking with the RAID is not what you want to be doing. That is something that would guarantee data loss due to someone entering a setting wrong.
I wouldn't call it mucking with RAID. Its just drive rotations.
Each time you add/remove a drive from the array, you chance clicking the wrong drive, the wrong action, etc. I'd call it unnecessary, mucking about where you shouldn't, and very risky as a few of the more friendly terms.
Not if you follow the correct steps and know what you are doing. Its not anything terribly difficult.
Why do you feel like the removable drives need to be a part of the RAID array?
Because the RAID1 mirror would write the changes to the swapped disk.
The idea is to have Drive 1 and Drive X in a RAID1 mirror, where Drive X = both drives 2 and 3 which would be swapped out daily.
-
Nothing is quite as simple as a usb drive that has last week's backups....
/Sarcasm
-
@dashrender said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
One of my clients does what the OP wants.
They bought 5 single drive NAS boxes... the backup software writes to the designated drive each night.
In the morning, they unplug it and take it home...Not great but it is cheap, In comparison
That's a little different, right? Not using the RAID, but abusing the hot swap bays.
-
@dave247 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
The idea is to have Drive 1 and Drive X in a RAID1 mirror, where Drive X = both drives 2 and 3 which would be swapped out daily.
The idea makes sense, just none of the parts are designed to be used in that fashion.
-
@travisdh1 said in offline, air-gapped backups / backup rotation (looking for hardware & ideas):
Each time you add/remove a drive from the array, you chance clicking the wrong drive, the wrong action, etc. I'd call it unnecessary, mucking about where you shouldn't, and very risky as a few of the more friendly terms.
That too, it does depend on fencing against human error a bit more than a normal system. There are better, more efficient, more reliable ways to handle this where that isn't the case.
-
To do basically the same thing, what you want is a NAS with local storage (with or without RAID, in this case you are without RAID even though you are using RAID, so no need to have RAID at all) and having a hot swap drive in a mechanism meant to handle this, like a USB style drive, and a script that does a file copy of just the backup, not a block mirror of the drives, to copy the backup to the second drive.
