KVM in Production - Build it yourself
-
There appears to be lots of backup solutions for KVM. And these solutions are the best ones. They are simple BASH scripts that do exactly what they are supposed to do, and some PERL scripts.
The best kinds of backups are these, and that you can configure, and forget about, with the exception of testing restores occasionally.
These backup solutions are the least likely to fail.
When you get into vendor-specific and GUI-based backups, that's when thing start to get weird and lose potential reliability.
I prefer the CLI-based backups for production. They are the most reliable.
I've NEVER had a powershell / BASH scripted backup fail. I cannot say the same for any GUI-based backup (Veeam, WSB, etc.). In fact, I use PowerShell / BASH to "clean-up or fix" GUI-based backup failures.
-
@tim_g said in KVM in Production - Build it yourself:
There appears to be lots of backup solutions for KVM. And these solutions are the best ones. They are simple BASH scripts that do exactly what they are supposed to do, and some PERL scripts.
The best kinds of backups are these, and that you can configure, and forget about, with the exception of testing restores occasionally.
These backup solutions are the least likely to fail.
When you get into vendor-specific and GUI-based backups, that's when thing start to get weird and lose potential reliability.
I prefer the CLI-based backups for production. They are the most reliable.
I've NEVER had a powershell / BASH scripted backup fail. I cannot say the same for any GUI-based backup (Veeam, WSB, etc.). In fact, I use PowerShell / BASH to "clean-up or fix" GUI-based backup failures.
I wouldn't mind a text-based UI backups.
-
@black3dynamite said in KVM in Production - Build it yourself:
@tim_g said in KVM in Production - Build it yourself:
There appears to be lots of backup solutions for KVM. And these solutions are the best ones. They are simple BASH scripts that do exactly what they are supposed to do, and some PERL scripts.
The best kinds of backups are these, and that you can configure, and forget about, with the exception of testing restores occasionally.
These backup solutions are the least likely to fail.
When you get into vendor-specific and GUI-based backups, that's when thing start to get weird and lose potential reliability.
I prefer the CLI-based backups for production. They are the most reliable.
I've NEVER had a powershell / BASH scripted backup fail. I cannot say the same for any GUI-based backup (Veeam, WSB, etc.). In fact, I use PowerShell / BASH to "clean-up or fix" GUI-based backup failures.
I wouldn't mind a text-based UI backups.
Yeah, that's great for full backups. But suppose you have a multi-TB sized VM... maybe you want to do a full backup weekly, and incrementals daily. I don't know how you'd accomplish that in the same way.
-
I suppose it, like everything else, comes down to your specific environmental needs.
-
Text, CLI based doesn't bother me, I would like to see actual speed of VM backups using the script, and does it enable incremental backups of the VM?
-
I amazed that cockpit doesn't have some kind of package for managing backups. That would be awesome.
Someone with mad skills could probably create a backup plugin for Cockpit.
http://cockpit-project.org/blog/creating-plugins-for-the-cockpit-user-interface.html -
Seems that Bacula has a KVM plugin: https://www.baculasystems.com/enterprise-backup-solution-with-bacula-systems/virtual-machine-backup-software . Any experience with that?
-
I think what @DustinB3403 feels bad is the mix of config and logic in the same file. If you would be able to separate the working logic from config, then edit a bunch of lines in a txt would not be so terrible .
Conceptually it should mot be so terrible:
1- snap
2- mount snap and backup location
3- run borg backup or similar
4- unmount everythingTesting is the most long part. With stuff like borg you should be confident in both backup and recovery.
Acutally recovery is trickier?! Don't know, should be like:
1- shutdown the machine
2- mount machine and backup location
3- restoreIssue is: how to programmatically tell borg which snap to use to recover...
I guess 2 of days of work... And I'm not super fast at writing code.
What has stopped me from using kvm in production was: what if I have to off load stuff to other people? Will they be
skilled enough for this crap?! So I just moved to hyperv+altaro... -
@matteo-nunziati said in KVM in Production - Build it yourself:
What has stopped me from using kvm in production was: what if I have to off load stuff to other people? Will they be
skilled enough for this crap?! So I just moved to hyperv+altaro...That's sad. I hope some user friendly solution will come out soon. I'm in the KVM boat since 2010, KVM is SO good today. Maybe it will get faster adoption after the AWS move.
-
@stacksofplates said in KVM in Production - Build it yourself:
I have 12 KVM hosts in production. For the very few machines that have stateful data, I use either the backup options built into the software/service that's running to a mounted location, my script, or I use ReaR. The rest of the systems are stored in Git.
Bare KVM is just fast and easy. I second that, but the issue there is no standard way to manage it, especially backups, so this leaves KVM a solution for person that knows much about, and usually the sole IT in that location, cause if there was other IT folks, they just pick ESXi or Hyper-V to ensure continuity.
-
@francesco-provino said in KVM in Production - Build it yourself:
Seems that Bacula has a KVM plugin: https://www.baculasystems.com/enterprise-backup-solution-with-bacula-systems/virtual-machine-backup-software . Any experience with that?
No experience but reading the online man suggests a lot of pita to setup bacula, while the kvm plugin seems quite abysmal. Not much more then the famous perl script or @stacksofplates script
-
Mine isn’t meant to be a full backup replacement. It’s a way to get full disk images once a week or so. I think most people that are using KVM are either using cloud infra or using state machines and agents when needed. I don’t think there is much of a demand for this type of backup solution.
However I firmly believe any org can use automation. It’s not just for large orgs. If I can do it for everything in my house, any size business can.
-
@stacksofplates said in KVM in Production - Build it yourself:
Mine isn’t meant to be a full backup replacement. It’s a way to get full disk images once a week or so. I think most people that are using KVM are either using cloud infra or using state machines and agents when needed. I don’t think there is much of a demand for this type of backup solution.
However I firmly believe any org can use automation. It’s not just for large orgs. If I can do it for everything in my house, any size business can.
So what do you do with your stateless data - like photos? video files, etc?
-
@dashrender said in KVM in Production - Build it yourself:
@stacksofplates said in KVM in Production - Build it yourself:
Mine isn’t meant to be a full backup replacement. It’s a way to get full disk images once a week or so. I think most people that are using KVM are either using cloud infra or using state machines and agents when needed. I don’t think there is much of a demand for this type of backup solution.
However I firmly believe any org can use automation. It’s not just for large orgs. If I can do it for everything in my house, any size business can.
So what do you do with your stateless data - like photos? video files, etc?
For home? Amazon. It’s automatically backed up there. I do have a bunch of music but that’s on one NFS VM that’s running CrashPlan. But like I said it’s one of the few that would require an agent.
-
I am starting to play with KVM after using XenServer for a while. While looking for a good KVM backup solution I found:
https://github.com/jagane/qemu-kvm-livebackup
https://www.linux-kvm.org/images/b/b6/2011-forum-LiveBackup.pdfI will be testing as soon as possible (probably in 1 week). Does anyone has any experience with it?
-
@dave_c said in KVM in Production - Build it yourself:
I am starting to play with KVM after using XenServer for a while. While looking for a good KVM backup solution I found:
https://github.com/jagane/qemu-kvm-livebackup
https://www.linux-kvm.org/images/b/b6/2011-forum-LiveBackup.pdfI will be testing as soon as possible (probably in 1 week). Does anyone has any experience with it?
The last commit was in 2011. Is it still usable?
-
Yes, I know. I am reading documentation and some of the source to determine if it is viable or not. So far it seems like some of the features it implements have been some how included in libvirt. Perhaps it would be a good challenge to bring this up to date.
The most important question is: Does anyone know a complete open source backup system for KVM? Like xen-orchestra for XS
I know about convirture and vprotect; seems like SEP has something but I am looking for open source (you know, because of reasons) -
@dave_c said in KVM in Production - Build it yourself:
Yes, I know. I am reading documentation and some of the source to determine if it is viable or not. So far it seems like some of the features it implements have been some how included in libvirt. Perhaps it would be a good challenge to bring this up to date.
The most important question is: Does anyone know a complete open source backup system for KVM? Like xen-orchestra for XS
I know about convirture and vprotect; seems like SEP has something but I am looking for open source (you know, because of reasons)Backups should be super easy. Take a snapshot -> copy the original disk image file -> remove snapshot, done.
-
@travisdh1 said in KVM in Production - Build it yourself:
@dave_c said in KVM in Production - Build it yourself:
Yes, I know. I am reading documentation and some of the source to determine if it is viable or not. So far it seems like some of the features it implements have been some how included in libvirt. Perhaps it would be a good challenge to bring this up to date.
The most important question is: Does anyone know a complete open source backup system for KVM? Like xen-orchestra for XS
I know about convirture and vprotect; seems like SEP has something but I am looking for open source (you know, because of reasons)Backups should be super easy. Take a snapshot -> copy the original disk image file -> remove snapshot, done.
Backups are totally not that that easy.
Sure they are is you have unlimited storage and unlimited bandwidth on 1 Gbps+ connections.
But in the real world, we need incrementals and/or differentials of those snaphots.
-
But in the real world, we need incrementals and/or differentials of those snaphots.
Exactly. And please, a super easy and fast way to recover, either a VM or a file.
I know that Veeam/Xen Orchestra and others make backups look really easy and I would like something like that for KVM.
So far, https://github.com/dguerri/LibVirtKvm-scripts looks great on paper.
