KVM and Back Ups



  • So in my lab, now that things are operational (Beginner's Guide to KVM Administration) we need to next look at how we're going to be backing up our VM's.

    Using snapshots, while might work in a pinch but they aren't backups.

    And while we could likely copy the files that make up the VM this requires us to shutdown our VM's to get those files.

    • Downtime is yet again a negative.

    So what options do we have? We could use the mainstream tools for our VMs like Veeam or maybe something open source like UrBackup with an agent on every VM.

    • Of course installing an agent onto every VM is a pain in the rear. . . . but maybe.

    So of course our next approach would be something run at the hypervisor, but using the CLI on a regular basis to create backups would be tedious. Maybe crontab or some such approach could work.

    While digging around for an answer to this question, I found an answer on another forum, from several years ago. Anyhow that answer is at least 5 years old. Here is the most current answer, a very in-depth script to backing up your VMs.

    So in-depth that so far I haven't finished reading through the entire process.

    So at this point in the topic, I'm going to simply put a pause here. @scottalanmiller @JaredBusch or anyone else who may be using this script in their labs, let me know what if anything to watch out for.



  • I'm not aware of any agentless KVM backup software.

    I've been using ReaR, but that does the entire server and/or disks of the host (effectively backing up the VMs), but they aren't individually restorable.

    However, ReaR is easy enough to set up automagically in each VM with Salt. I feel ya though, definitely a pain to have to install agents or software in each VM... But then again, Salt would take care of that.

    I'll definitely be watching this thread for ideas or agentless KVM VM backup software.



  • @tim_g I'm trying to figure out the system requirements to get this perl script to run, and then how to run it.

    And maybe an easy way to update it for any systems I'd want to backup.



  • @dustinb3403 said in KVM and Back Ups:

    @tim_g I'm trying to figure out the system requirements to get this perl script to run, and then how to run it.

    And maybe an easy way to update it for any systems I'd want to backup.

    I only took a few 2-3 second glimpses along that script... and I seen a spot that turns the VM off. Does it actually do as you think? You looked at it more thann I did, so perhaps I seen that part out of context.



  • @tim_g said in KVM and Back Ups:

    @dustinb3403 said in KVM and Back Ups:

    @tim_g I'm trying to figure out the system requirements to get this perl script to run, and then how to run it.

    And maybe an easy way to update it for any systems I'd want to backup.

    I only took a few 2-3 second glimpses along that script... and I seen a spot that turns the VM off. Does it actually do as you think? You looked at it more thann I did, so perhaps I seen that part out of context.

    It has several different options, it can snap, turn off the vm and then power it back on etc.



  • @dustinb3403 said in KVM and Back Ups:

    @tim_g said in KVM and Back Ups:

    @dustinb3403 said in KVM and Back Ups:

    @tim_g I'm trying to figure out the system requirements to get this perl script to run, and then how to run it.

    And maybe an easy way to update it for any systems I'd want to backup.

    I only took a few 2-3 second glimpses along that script... and I seen a spot that turns the VM off. Does it actually do as you think? You looked at it more thann I did, so perhaps I seen that part out of context.

    It has several different options, it can snap, turn off the vm and then power it back on etc.

    Ah, good. That must have been the part I seen then. I'll look at it more another time when I have the time...



  • For the requirements, I found out you need to install perl-XML-Simple

    Obviously this needs to be installed to the server.

    So. . .

    ssh [email protected]
    sudo dnf install perl-XML-Simple
    

    I now have some output to play with.

    0_1512874213774_VirtualBox_2017-12-09_21-49-52.png





  • @stacksofplates said in KVM and Back Ups:

    https://mangolassi.it/topic/12537/kvm-snapshot-backup-script

    The name is a little misleading, I had to read into it and pick it apart to get it.

    So it looks like this creates a full backup of the VM. This seems to accomplish what is needed here @DustinB3403

    The function of the script is to create a full backup of the VM and VM's disk, without having to shut down the VM. The way it's done is by creating a snapshot, to get the VM off of it's main virtual disk, allowing you to get a solid replica of it, then the script deletes the snapshot, merging the it back into the main VD.

    Nice @stacksofplates !



  • @dustinb3403 I've been using this script in production for somethin glike 1 year when I was using KVM as hypervisor.
    this is the facto the only good solution for KVM. It also supports gluster volumes!
    you do nopt need to bring the VM down IF you build your VM disk on some snapshottable stuff like LVM or qcow (not sure about the latter).

    I've written a small note for my usage at the time... need a bit to translate it in english. this is how I used it in brief

    wget http://repo.firewall-services.com/misc/virt/virt-backup.pl
    chmod 755 virt-backup.pl
    yum install perl perl-XML-Simple perl-Sys-Virt
    mkdir -p /var/lib/libvirt/backup
    #(maybe you want this to be a share mounted from a remote NAS)
    /$PATH/$TO/virt-backup.pl --action=dump --vm=VM_NAME --compress=pbzip2
    

    note that I installed bbzip from EPEL to speedup compression and make full use of all server cores: my backups were scheduled during not working hours with the server idling for the most.



  • @DustinB3403 almost forgotten... I've this stub on github



  • @dustinb3403

    In my previous experiments which worked but have not used it in production. I would just freeze the VM, then rsync it, then thaw it. If your backing up to LAN down time will be very minimal and with rsync you can backup incrementally of you wish.

    It is recommended to also stop services first then freeze then rsync then thaw then start services again. I liked it and its KISS principle.



  • I found this. I may try it on my laptop later.

    Live backup of QEMU/KVM/libVirt virtual machines via Duplicity
    https://itefix.net/content/live-backup-qemu-kvm-libvirt-virtual-machines-duplicity



  • Here's something as well... KVM live backup software, that's actually current:

    https://www.handybackup.net/kvm-backup.shtml



  • I have found that there are many easy ways to get a backup of your VM.

    What I have not found is something that can look at those snapshots that are made and backup incremental differences.

    I cannot offsite 1+ TB per day in image files.



  • @tim_g said in KVM and Back Ups:

    Here's something as well... KVM live backup software, that's actually current:

    https://www.handybackup.net/kvm-backup.shtml

    Wtf... the download is an EXE. I don't get it...



  • @jaredbusch said in KVM and Back Ups:

    I have found that there are many easy ways to get a backup of your VM.

    What I have not found is something that can look at those snapshots that are made and backup incremental differences.

    I cannot offsite 1+ TB per day in image files.

    Yeah that would be great to find something like that. I'm looking. I remember someone finding something that's paid. But I forget what it was. I forgot to bookmark it.



  • @tim_g said in KVM and Back Ups:

    @jaredbusch said in KVM and Back Ups:

    I have found that there are many easy ways to get a backup of your VM.

    What I have not found is something that can look at those snapshots that are made and backup incremental differences.

    I cannot offsite 1+ TB per day in image files.

    Yeah that would be great to find something like that. I'm looking. I remember someone finding something that's paid. But I forget what it was. I forgot to bookmark it.

    I pay for that functionality now with Hyper-V and Veeam. So I do not mind paying for it for KVM.



  • Found it:

    https://storware.eu/en/storware-vprotect/

    It shows incrementals in the screenshots anyways.



  • You could theoretically just use one of the scripts to do incremental. It’s all AOW/ROW so you just get a new disk each snapshot. Then at the end of the week or whatever block commit everything back into the main image and start over.



  • @stacksofplates said in KVM and Back Ups:

    You could theoretically just use one of the scripts to do incremental. It’s all AOW/ROW so you just get a new disk each snapshot. Then at the end of the week or whatever block commit everything back into the main image and start over.

    Something like that is what I want. Replicating what I get from Veeam now.





  • Okay, so after doing some research, I've found some excellent news regarding KVM and incremental backups. But it also comes with some bad news.

    It looks like the ability and support to do incremental backups is fully integrated into QEMU (I think as of 2.5).

    It uses what I keep seeing referred to as Bitmaps, or more specifically Dirty Bitmaps or Block Dirty Bitmaps.

    The bad news, from the little research I've done so far, is that all this stuff is managed using JSON files, which would be done via the QMP (QEMU Machine Protocol). This basically tells me that the only way to do incremental backups with KVM, is if you are a software engineer and know how to build software that uses the QMP, or have the time to figure it all the hell out. It all seems relatively simple...

    So, this means we're still stick using something that costs money... because I doubt anyone would build something like this open sourced when there's a lot of money to be made... unless Google or someone does it for internal use and decides to make it open source.

    I spent some more time looking at things since writing everything above, and looks like you can get starting issuing QMP commands via telnet to start, and then by CLI after installing socat and rlwrap. Again, no idea what I'm looking here: https://wiki.qemu.org/Documentation/QMP, but perhaps this deserves some more looking into.

    Maybe someone else can clear it up for me so I know what I'm looking at. I wish I had more time and programming skills... I'd definitely be into figuring this stuff out and putting it out there.

    maybe SodiumSuite devs feel like being nice? 😉

    Anyways, here are my references:

    https://kashyapc.fedorapeople.org/QEMU-Docs/_build/html/docs/bitmaps.html

    https://wiki.qemu.org/Features/IncrementalBackup

    https://www.linux-kvm.org/images/e/e6/Kvm-forum2017_backup.pdf

    https://events.static.linuxfound.org/sites/events/files/slides/kvm2015_rh_light_44_vfinal.pdf

    https://wiki.qemu.org/Documentation/QMP



  • @tim_g
    One problem is the dirty bitmapping in the API is push only so anything expecting an interactive API won't work. This means the backup system can't query the API for dirty bitmaps, only create them.

    (At least it was this way the last time I looked at it)



  • This is quite an old thread, but it discusses exactly what I am researching.

    I use KVM (via Fedora) in my home lab, and currently, since I have no production workloads on it, I run file only backups via a couple scripts.

    However, I'd like to explore the possibility of using KVM in a production scenario. Backups are obviously key here. Automated, live, nightly snapshots (entire VM) is what I am looking for.

    What would be the best way to achieve this? Assuming I am running a single KVM server with local storage.

    Is this still a good option: https://storware.eu/products/vprotect/

    Is there something better?

    Can something like ProxMox handle this? https://www.proxmox.com/en/proxmox-ve



  • Proxmox backups are always a full backup.
    https://pve.proxmox.com/wiki/Backup_and_Restore



  • @fuznutz04 said in KVM and Back Ups:

    However, I'd like to explore the possibility of using KVM in a production scenario. Backups are obviously key here. Automated, live, nightly snapshots (entire VM) is what I am looking for.
    What would be the best way to achieve this? Assuming I am running a single KVM server with local storage.

    You come back to the fundamental question of... why are you approaching backups this way?

    Backups have never been something that we think of at the hypervisor level for decades, all of a sudden its the hot new "everyone has to do it" way. But it has all kinds of limitations and should only be a consideration for certain workloads, and only one of the options. Backups are always by workload, never by platform. It's fundamentally a flawed way to think of backups because it hides the scope and complexity of getting reliable backups and causes all kinds of problems.

    Under no circumstances can anyone answer how you should take backups based on knowing your platform - no matter what hypervisor or OS you use, your backups have to be a decision based on your workloads.



  • @black3dynamite said in KVM and Back Ups:

    Proxmox backups are always a full backup.
    https://pve.proxmox.com/wiki/Backup_and_Restore

    Do you use or have any experience using proxmox? Does/can it just run as a VM on the host?



  • @fuznutz04 said in KVM and Back Ups:

    @black3dynamite said in KVM and Back Ups:

    Proxmox backups are always a full backup.
    https://pve.proxmox.com/wiki/Backup_and_Restore

    Do you use or have any experience using proxmox? Does/can it just run as a VM on the host?

    PM doesn't do anything special, you still need to deal with the backups on a workload basis.



  • @scottalanmiller said in KVM and Back Ups:

    @fuznutz04 said in KVM and Back Ups:

    However, I'd like to explore the possibility of using KVM in a production scenario. Backups are obviously key here. Automated, live, nightly snapshots (entire VM) is what I am looking for.
    What would be the best way to achieve this? Assuming I am running a single KVM server with local storage.

    You come back to the fundamental question of... why are you approaching backups this way?

    Backups have never been something that we think of at the hypervisor level for decades, all of a sudden its the hot new "everyone has to do it" way. But it has all kinds of limitations and should only be a consideration for certain workloads, and only one of the options. Backups are always by workload, never by platform. It's fundamentally a flawed way to think of backups because it hides the scope and complexity of getting reliable backups and causes all kinds of problems.

    Under no circumstances can anyone answer how you should take backups based on knowing your platform - no matter what hypervisor or OS you use, your backups have to be a decision based on your workloads.

    So in this case, I’d have a PBX, a Wordpress site and eventually some windows server workloads. All of them are individually backed up via scripts at the OS level.

    What I’m after is an “instant” type restore. For example, I had a developer fubar a server the other day. Completely unrecoverable. It was hosted at vultr, and I used their backup service. I was able to completely restore the server from their snapshot backup. That’s what I am after.