Offsite backups with XenServer, BackBlaze, rclone. Proof of concept, needs tweaking

momurda

Situation:
You have XenServer hypervisor and a bunch of important vms.
Your backup software either doesn't have offsite capabilities built in, or the ones it has built in are expensive.
You worry about your office burning down along with all your backups during off hours. Especially since there is a 'burn-in room' kept at 150F+ 24x7 for testing the stuff your company makes.

The basic setup:
-XenServer pool
-BackBlaze b2 account
-Good upload speeds(not required I suppose. Painful without it)
-A standalone XS install, or some other server you can use as an nfs export target before uploading to b2. I have a XS server with just 1 vm I made for this purpose. If you are brave/crazy, you can do all this on your production XS pool master. I actually have done this with another XS test host using dom0 without issue. But there is a big difference between dom0 on a test server with 2 vms vs production dom0 with 40 or so vms. If you want to do this on your production XS pool, XS 7+ is required as 6.5- has ancient version of python and you cant install b2 easily.

On my standalone XS host, I have installed Fedora 27 minimal.
Hostname and nfs share named vm-exports Might change that to avoid confusion.
On this server I have installed and configured nfs server share, given access to the subnet my production and test XS hosts are on.
Basic config for server and nfs: This might be totally insecure and bad. Not sure but it does work. Perhaps others can provide more insight here. I have rarely used nfs.

sudo dnf install glances
sudo dnf install sysstat
sudo dnf install NetworkManager-tui
sudo easy_install pip
sudo pip install --upgrade b2
sudo reboot now
sudo mkdir /var/exports
sudo dnf install nfs-utils
sudo dnf install system-config-nfs
sudo systemctl enable rpcbind
sudo systemctl enable nfs-server
cat /etc/exports
sudo firewall-cmd --add-service=nfs --permanent
sudo firewall-cmd --add-service=mountd --permanent
sudo firewall-cmd --add-service=rpc-bind --permanent
sudo systemctl enable rpc-bind nfs-server
sudo systemctl start rpc-bind nfs-server
sudo firewall-cmd --reload
sudo firewall-cmd –-list-services
sudo nano /etc/exports
	/var/vm-exports 192.168.70.0/26(rw) #contents of /etc/exports
sudo exportfs -v  #whoops need to restart nfs
sudo systemctl restart nfs-server
sudo exportfs -a
sudo exportfs -v
showmount -e

Now you should be able to go and add this nfs share to your XS pool. I used XenCenter to do this.
Now if you go and
ls -la /var/vm-exports
you will see something there, a directory with a name that looks like gibberish.
Don't worry, this is just the directory your XS made. The gibberish name is in fact the uuid of the SR in XS.
ls /var/vm-exports
should return something like this
b0b36aca-56aa-68d8-b804-ca040d4246d3

Now that you have a place to put your vms for upload, let us make some backups.
There are lots of scripts out there to do this. In fact this whole write up is mostly me putting together stuff other people wrote and posted online. They did 90%+ of the work, I just put it together to do what I want it to do.
For this I am just going to do a one off.
From your XS pool master:
# xe vm-snapshot uuid=04748678-6637-0d9e-bdf1-0fe0eb33b75c new-name-label=vmbackup
This simply takes a snapshot of the vm you specify with uuid. You can also use name-label instead of uuid. But thanks to autocomplete uuid is easy.
This spits out a uuid, which we need for next part.
Then
xe template-param-set is-a-template=false uuid=83e18306-0a85-2b41-70c9-b543ea496d62 The uuid here is the one that was returned in previous snapshot command.
You cant autocomplete the uuid here for some reason. Very annoying.
This part is where scripting will come in handy, and I will do this later when I am sure I wont bleep things up.
Then you do this
xe vm-export uuid=83e18306-0a85-2b41-70c9-b543ea496d62 filename=/run/sr-mount/b0b36aca-56aa-68d8-b804-ca040d4246d3/vmbackup.xva
XS mounts SRs under /run/sr-mount/uuidofSR
You can see here that b0b36aca-56aa-68d8-b804-ca040d4246d3 is the same as the folder in the /var/vm-exports folder on the NFS server.
Hopefully you see
Export succeeded
Now if you go back to your nfs server and check /var/vm-exports/b0b36aca-56aa-68d8-b804-ca040d4246d3 you will see vmbackup.xva

Well after all that you have an xva file you can put anywhere.
login to your nfs server.
You first need to run
b2 authorize-account then input your account id and applicationKey
I have already made a bucket here called vm-exports using the web interface of b2.
Now you need to do
b2 upload-file vm-exports /var/vm-exports/b0b36aca-56aa-68d8-b804-ca040d4246d3/vmbackup.xva vmbackup.xva
This will start uploading.
When done you have a full vm backup in the cloud, for a few minutes of time and b2's incredibly low cost.

rclone is similar to rsync, it comes with configuration options for about 30 different cloud storage providers, including b2.
Ill post more tomorrow about rclone. It is optional, though it makes this upload process easier when dealing with a large # of backup files in your nfs share.
I go home now.

DustinB3403

Why not just use XO and export to your staging NAS (primary on-site backup) and then backup directly to b2. Assuming a Synology NAS there is a plugin to sync directly to your bucket.

momurda

I dont use XO or Synology

DustinB3403

@momurda said in Offsite backups with XenServer, BackBlaze, rclone. Proof of concept, needs tweaking:

I dont use XO or Synology

Can I ask why you don't use XO?

dbeato

What NFS server do you have?