Offsite backups with XenServer, BackBlaze, rclone. Proof of concept, needs tweaking

  • Situation:
    You have XenServer hypervisor and a bunch of important vms.
    Your backup software either doesn't have offsite capabilities built in, or the ones it has built in are expensive.
    You worry about your office burning down along with all your backups during off hours. Especially since there is a 'burn-in room' kept at 150F+ 24x7 for testing the stuff your company makes.

    The basic setup:
    -XenServer pool
    -BackBlaze b2 account
    -Good upload speeds(not required I suppose. Painful without it)
    -A standalone XS install, or some other server you can use as an nfs export target before uploading to b2. I have a XS server with just 1 vm I made for this purpose. If you are brave/crazy, you can do all this on your production XS pool master. I actually have done this with another XS test host using dom0 without issue. But there is a big difference between dom0 on a test server with 2 vms vs production dom0 with 40 or so vms. If you want to do this on your production XS pool, XS 7+ is required as 6.5- has ancient version of python and you cant install b2 easily.

    On my standalone XS host, I have installed Fedora 27 minimal.
    Hostname and nfs share named vm-exports Might change that to avoid confusion.
    On this server I have installed and configured nfs server share, given access to the subnet my production and test XS hosts are on.
    Basic config for server and nfs: This might be totally insecure and bad. Not sure but it does work. Perhaps others can provide more insight here. I have rarely used nfs.

    sudo dnf install glances
    sudo dnf install sysstat
    sudo dnf install NetworkManager-tui
    sudo easy_install pip
    sudo pip install --upgrade b2
    sudo reboot now
    sudo mkdir /var/exports
    sudo dnf install nfs-utils
    sudo dnf install system-config-nfs
    sudo systemctl enable rpcbind
    sudo systemctl enable nfs-server
    cat /etc/exports
    sudo firewall-cmd --add-service=nfs --permanent
    sudo firewall-cmd --add-service=mountd --permanent
    sudo firewall-cmd --add-service=rpc-bind --permanent
    sudo systemctl enable rpc-bind nfs-server
    sudo systemctl start rpc-bind nfs-server
    sudo firewall-cmd --reload
    sudo firewall-cmd –-list-services
    sudo nano /etc/exports
    	/var/vm-exports #contents of /etc/exports
    sudo exportfs -v  #whoops need to restart nfs
    sudo systemctl restart nfs-server
    sudo exportfs -a
    sudo exportfs -v
    showmount -e

    Now you should be able to go and add this nfs share to your XS pool. I used XenCenter to do this.
    Now if you go and
    ls -la /var/vm-exports
    you will see something there, a directory with a name that looks like gibberish.
    Don't worry, this is just the directory your XS made. The gibberish name is in fact the uuid of the SR in XS.
    ls /var/vm-exports
    should return something like this

    Now that you have a place to put your vms for upload, let us make some backups.
    There are lots of scripts out there to do this. In fact this whole write up is mostly me putting together stuff other people wrote and posted online. They did 90%+ of the work, I just put it together to do what I want it to do.
    For this I am just going to do a one off.
    From your XS pool master:
    # xe vm-snapshot uuid=04748678-6637-0d9e-bdf1-0fe0eb33b75c new-name-label=vmbackup
    This simply takes a snapshot of the vm you specify with uuid. You can also use name-label instead of uuid. But thanks to autocomplete uuid is easy.
    This spits out a uuid, which we need for next part.
    xe template-param-set is-a-template=false uuid=83e18306-0a85-2b41-70c9-b543ea496d62 The uuid here is the one that was returned in previous snapshot command.
    You cant autocomplete the uuid here for some reason. Very annoying.
    This part is where scripting will come in handy, and I will do this later when I am sure I wont bleep things up.
    Then you do this
    xe vm-export uuid=83e18306-0a85-2b41-70c9-b543ea496d62 filename=/run/sr-mount/b0b36aca-56aa-68d8-b804-ca040d4246d3/vmbackup.xva
    XS mounts SRs under /run/sr-mount/uuidofSR
    You can see here that b0b36aca-56aa-68d8-b804-ca040d4246d3 is the same as the folder in the /var/vm-exports folder on the NFS server.
    Hopefully you see
    Export succeeded
    Now if you go back to your nfs server and check /var/vm-exports/b0b36aca-56aa-68d8-b804-ca040d4246d3 you will see vmbackup.xva

    Well after all that you have an xva file you can put anywhere.
    login to your nfs server.
    You first need to run
    b2 authorize-account then input your account id and applicationKey
    I have already made a bucket here called vm-exports using the web interface of b2.
    Now you need to do
    b2 upload-file vm-exports /var/vm-exports/b0b36aca-56aa-68d8-b804-ca040d4246d3/vmbackup.xva vmbackup.xva
    This will start uploading.
    When done you have a full vm backup in the cloud, for a few minutes of time and b2's incredibly low cost.

    rclone is similar to rsync, it comes with configuration options for about 30 different cloud storage providers, including b2.
    Ill post more tomorrow about rclone. It is optional, though it makes this upload process easier when dealing with a large # of backup files in your nfs share.
    I go home now.

  • Why not just use XO and export to your staging NAS (primary on-site backup) and then backup directly to b2. Assuming a Synology NAS there is a plugin to sync directly to your bucket.

  • I dont use XO or Synology

  • @momurda said in Offsite backups with XenServer, BackBlaze, rclone. Proof of concept, needs tweaking:

    I dont use XO or Synology

    Can I ask why you don't use XO?

  • What NFS server do you have?