Backup Systems without on-site external storage
-
To date, my experience with backups has been using Windows Server Backup in the following environment. Synology NAS presenting block storage to Hyper-V host via ISCSI. VMs have a vhdx stored on this block storage, and Windows Server Backup within the VMs store backups to the vhdx. It's time to learn more and how to do stuff right
My lab in colocation is a 1U machine with KVM. I imagine there are probably actual situations where a business might have a single server in colocation and need a method of backing up data stored on the VMs hosted by that single server.
Here is a possible solution:
Have a VM hosting backup software, which stores backup data in that VM. The VM, then has a job to replicate that data to $CloudStorageProvider. This would allow you to have a local copy for quick restorations and data stored off-site. The big weaknesses here are the fact that if the backup VM is hosed, or the KVM host is hosed, then you lose with no way of recovery.
Have any of you deal with backups with this scenario? If so, how did you handle it? One other caveat is that I'm not considering file-level vs. full VM restoration. The scope is more about where you can store said data in a situation where on-site external storage isn't available.
-
@eddiejennings in the rare cases that I want to backup something in my home lab, which is a single server running KVM as well, I use Duplicity with Backblaze B2 as the storage.
The VM gets paused, the backup copies and compresses the config and virtual drive files, and the VM gets restarted. It's possible to do an external snapshot instead of just the pause/start cycle of downtime is an issue as well.
-
@eddiejennings For file level backup, pick your favored backup solution and an online storage provider like B2 or S3 to provide the storage.
-
@travisdh1 said in Backup Systems without on-site external storage:
@eddiejennings in the rare cases that I want to backup something in my home lab, which is a single server running KVM as well, I use Duplicity with Backblaze B2 as the storage.
The VM gets paused, the backup copies and compresses the config and virtual drive files, and the VM gets restarted. It's possible to do an external snapshot instead of just the pause/start cycle of downtime is an issue as well.
Poked around Duplicity's site a bit. So you'd run Duplicity on the KVM host itself, with it storing to Backblaze. Assuming the amount of storage you have isn't an issue, and assuming this was a production server rather than a lab, would you also consider backing up to a directory on the host itself as well? My thinking here is a VM has a problem and needs to be restored, it would be quicker to restore from that on-premises copy. If your KVM host is hosed, then you're done, but you'd rebuild KVM host, configure, and then restore the individual VMs from the backup stored in B2.
-
@eddiejennings said in Backup Systems without on-site external storage:
@travisdh1 said in Backup Systems without on-site external storage:
@eddiejennings in the rare cases that I want to backup something in my home lab, which is a single server running KVM as well, I use Duplicity with Backblaze B2 as the storage.
The VM gets paused, the backup copies and compresses the config and virtual drive files, and the VM gets restarted. It's possible to do an external snapshot instead of just the pause/start cycle of downtime is an issue as well.
Assuming the amount of storage you have isn't an issue, and assuming this was a production server rather than a lab, would you also consider backing up to a directory on the host itself as well?
It all depends on the situation. I don't consider a backup going to the same host to be a backup. It could be convenient if you just need to restore a couple files, that is true, but I wouldn't count it as a part of any backup or DR plan.
Where the backups are stored at doesn't matter, as long as they aren't stored on the same host and/or storage that the originals reside on.
I do like having a system level backup (the VM image), and file level backup where it makes sense. Like a Nextcloud installation. Where it doesn't make sense is on a web server with mostly static content, in which case a system level backup alone would probably be all right (especially considering how small a minimal + web server install is, probably a minute at most to restore.) This is also where I'll mention that state systems like Sale make it longer to restore a system level backup than just rebuilding the system in the first place. So it really does all depend on the situation. There isn't one best way to do things as with so much of our trade craft.
-
@travisdh1 said in [Backup Systems without on-site external storage]
It all depends on the situation. I don't consider a backup going to the same host to be a backup. It could be convenient if you just need to restore a couple files, that is true, but I wouldn't count it as a part of any backup or DR plan.
Where the backups are stored at doesn't matter, as long as they aren't stored on the same host and/or storage that the originals reside on.
I agree, any kind of "backup" that's stored on the host really isn't a backup. Also, you're right in there's never a one-best-way to do things. It looks like in the situation where you have only one server in colo, the only viable options would be services that store the data off-site. I suppose a better question, if this were a production environment, would be "why was this environment designed without a way of having on-premises backups?"
-
@EddieJennings Storing data offsite isnt that bad, and if your ISP is given enough money per month then you can upload and download things to b2 at close to Gb speeds.
-
@eddiejennings
I have similar scenario and my idea to get local backup was to have drives on the host that are passed through to one VM which just works as a file server for those drives - software raid.If the entire machine goes bye bye it would be a matter physically removing the drives in the bays and mounting them on another server. This would be possible because the files are located on a real file system (no LVM) and not inside a VM. Since it's software raid it is 100% portable to another machine with other hardware.
-
@pete-s said in Backup Systems without on-site external storage:
@eddiejennings
I have similar scenario and my idea to get local backup was to have drives on the host that are passed through to one VM which just works as a file server for those drives - software raid.If the entire machine goes bye bye it would be a matter physically removing the drives in the bays and mounting them on another server. This would be possible because the files are located on a real file system (no LVM) and not inside a VM. Since it's software raid it is 100% portable to another machine with other hardware.
So is it a VM our isn't it?
No LVM just complicates managing that storage, why would you think that it's good not to have it?
Yes, you can move those drives to another hardware box, but unless we're talking about a white box or bad NAS, the system almost always outlasts the drives themselves.
With LVM, you could plug in USB storage, mirror the entire thing, and carry the USB drive off with minimal fuss.
-
@travisdh1 said in Backup Systems without on-site external storage:
@pete-s said in Backup Systems without on-site external storage:
@eddiejennings
I have similar scenario and my idea to get local backup was to have drives on the host that are passed through to one VM which just works as a file server for those drives - software raid.If the entire machine goes bye bye it would be a matter physically removing the drives in the bays and mounting them on another server. This would be possible because the files are located on a real file system (no LVM) and not inside a VM. Since it's software raid it is 100% portable to another machine with other hardware.
So is it a VM our isn't it?
No LVM just complicates managing that storage, why would you think that it's good not to have it?
Yes, you can move those drives to another hardware box, but unless we're talking about a white box or bad NAS, the system almost always outlasts the drives themselves.
With LVM, you could plug in USB storage, mirror the entire thing, and carry the USB drive off with minimal fuss.
A backup is a backup and you need to be able to restore it on another machine. Otherwise it's not a backup.
An USB drive hanging of a colo rackserver isn't what I call a backup.
What I'm talking about is backup to disk array that only contain backup files and if the need arises easily can be mounted and restored on another server. If you don't have a hardware problem there will not be any need to remove the drives.
I don't know what purpose LVM would serve on a one partition disk array but if it makes you feel warm and cozy go ahead.
-
@pete-s said in Backup Systems without on-site external storage:
@travisdh1 said in Backup Systems without on-site external storage:
@pete-s said in Backup Systems without on-site external storage:
@eddiejennings
I have similar scenario and my idea to get local backup was to have drives on the host that are passed through to one VM which just works as a file server for those drives - software raid.If the entire machine goes bye bye it would be a matter physically removing the drives in the bays and mounting them on another server. This would be possible because the files are located on a real file system (no LVM) and not inside a VM. Since it's software raid it is 100% portable to another machine with other hardware.
So is it a VM our isn't it?
No LVM just complicates managing that storage, why would you think that it's good not to have it?
Yes, you can move those drives to another hardware box, but unless we're talking about a white box or bad NAS, the system almost always outlasts the drives themselves.
With LVM, you could plug in USB storage, mirror the entire thing, and carry the USB drive off with minimal fuss.
A backup is a backup and you need to be able to restore it on another machine. Otherwise it's not a backup.
An USB drive hanging of a colo rackserver isn't what I call a backup.
Neither is a backup going to drives in the same host. At least with the USB, you'd still have something available if the place burned down.
What I'm talking about is backup to disk array that only contain backup files and if the need arises easily can be mounted and restored on another server. If you don't have a hardware problem there will not be any need to remove the drives.
So long as the system is not writing junk to the array. Which while not common, I have seen happen.
I don't know what purpose LVM would serve on a one partition disk array but if it makes you feel warm and cozy go ahead.
Why would you not want LVM?
It's not that it makes me feel warm and fuzzy, it's the added functionality.
-
@pete-s said in Backup Systems without on-site external storage:
I don't know what purpose LVM would serve on a one partition disk array but if it makes you feel warm and cozy go ahead.
Adds snapshotting.
-
@pete-s said in Backup Systems without on-site external storage:
An USB drive hanging of a colo rackserver isn't what I call a backup.
As a USB connection is a network one (layer 2) and it's an external box, I'd consider it a backup, just a poor one. But technically, a USB connected disk is just a really crappy SAN. So if a SAN is considered good enough to be a back up, then a USB drive qualifies in that way.
To make a USB drive not qualify you'd have to come up with additional rules around backups like that they require internal redundancy (ruling out nearly all backup media like tape), that they need a minimum distance from the machine, etc. USB, as long as it is external, has both the network and physical separation. But is not protected from ransomware, but neither is any SAN.
-
I've recently come across Restic and looks great:
Site:
https://github.com/restic/resticDocumentation:
https://restic.readthedocs.io/en/latest/I haven't tried it yet, but it's on my list to try on my KVM server using Google's cloud as a test.
-
@obsolesce said in Backup Systems without on-site external storage:
I've recently come across Restic and looks great:
Site:
https://github.com/restic/resticDocumentation:
https://restic.readthedocs.io/en/latest/I haven't tried it yet, but it's on my list to try on my KVM server using Google's cloud as a test.
-
Direct backup to both SFTP and loads of cloud providers, very nice.
-
@obsolesce said in Backup Systems without on-site external storage:
I've recently come across Restic and looks great:
Site:
https://github.com/restic/resticDocumentation:
https://restic.readthedocs.io/en/latest/I haven't tried it yet, but it's on my list to try on my KVM server using Google's cloud as a test.
I’ve only tested it with backing up nextcloud user data and it’s been working great so far.
-
@black3dynamite said in Backup Systems without on-site external storage:
@obsolesce said in Backup Systems without on-site external storage:
I've recently come across Restic and looks great:
Site:
https://github.com/restic/resticDocumentation:
https://restic.readthedocs.io/en/latest/I haven't tried it yet, but it's on my list to try on my KVM server using Google's cloud as a test.
I’ve only tested it with backing up nextcloud user data and it’s been working great so far.
What are you backing up to?
-
@scottalanmiller said in Backup Systems without on-site external storage:
@black3dynamite said in Backup Systems without on-site external storage:
@obsolesce said in Backup Systems without on-site external storage:
I've recently come across Restic and looks great:
Site:
https://github.com/restic/resticDocumentation:
https://restic.readthedocs.io/en/latest/I haven't tried it yet, but it's on my list to try on my KVM server using Google's cloud as a test.
I’ve only tested it with backing up nextcloud user data and it’s been working great so far.
What are you backing up to?
I only tried an external hard drive and sftp to a Linux Server. I would like to try Backblaze.
-
@black3dynamite said in Backup Systems without on-site external storage:
@scottalanmiller said in Backup Systems without on-site external storage:
@black3dynamite said in Backup Systems without on-site external storage:
@obsolesce said in Backup Systems without on-site external storage:
I've recently come across Restic and looks great:
Site:
https://github.com/restic/resticDocumentation:
https://restic.readthedocs.io/en/latest/I haven't tried it yet, but it's on my list to try on my KVM server using Google's cloud as a test.
I’ve only tested it with backing up nextcloud user data and it’s been working great so far.
What are you backing up to?
I only tried an external hard drive and sftp to a Linux Server. I would like to try Backblaze.
I would like to check out Wasabi.
