Scale: VM long term archival - Leveraging HC3 VM Export - NAS and Cloud Storage
-
Many customers use the built in HC3 VM export to supplement their regular backup / restore / replication / DR strategies. For those not familiar, HC3 export takes a specified VM, running or shut off, takes a point in time snapshot of that VM, lets you specify a remote SMB file server share (currently must support SMB v1) and creates a fully independent copy of that VM snapshot on that share. The export will create a parent folder with the VM name. an XML file that contains all the configuration information about that VM such as number of vCPU's, RAM, nics, etc. and will create a qcow2 format virtual disk file for each virtual disk in that VM. Obviously there is a VM import function that uses all of that to reverse the process and recreate that VM and it's data from those fully independent export files. (also note that qcow2 is an open format that there are a variety of tools that can convert to and from qcow2 and other virtual disk formats)
While the HC3 UI currently only allows exports to be submitted immediately (and remember they are done from a snapshot so it's fine to export a running VM), the ScaleCare support team can and will set up simple scheduling of VM exports for you using some "under the hood" tools, even giving you some control of which VM's are exported using "tags" you can add and remove in the HC3 UI and storing batch VM exports in a date stamped directory name structure on said SMB file share... hmm, guessing most of you can see where this is going. Well there are a lot of different directions you might go with this depending on your needs.
Could these vm export files be considered an extra level of backup? sure! We have customers using monthly, weekly or even nightly scheduled exports as that.
Could these exports be retained for long periods of time, even many years? Absolutely, and unlike just data backups or archives these are fully bootable VMs with not just the data but the right version of the OS and applications required to access and process that data.
Where might you keep these export files? Well there is all sorts of deep and cheap budget NAS storage available, not to mention roll your own software solutions using commodity hardware if you want to go that route. I've heard of other customers using their "old" / retired production infrastructure (servers and storage) to house this export repository.
Some other things I've personally played around with include storing exports on a Windows server VM with the built in file system de-duplication enabled. Obviously if you are storing lots of versions of the same VM and are able to deduplicate at a sub file level you could see very high deduplication rates.
I've also played around with using cloud storage to achieve high capacity / off-site long term retention and will likely post more about some of these solutions in the future. From "file servers in the sky" to cloud storage gateway solutions available in the market, many that could run as virtual appliances right on your HC3 system. Further, there are all sorts of low level tools to simply copy files from ground to cloud where an admin could script some of that ... for example I've used azcopy . Although the AWS Cloud Storage Gateway only exposes storage as NFS and iSCSI, and is only released as a VMware VMDK or Hyper-V VHD, I have converted and run those virtual appliances on HC3 to provide a local gateway to AWS S3 cloud storage ... hopefully AWS will fully support a native KVM version soon since they are converting their whole EC2 cloud back end to use KVM as the hypervisor.
I'm currently playing with a "preview" feature from Microsoft Azure called Azure File Sync that essentially provides "cloud tiering" to an on prem windows file server. So I do all my HC3 exports to that windows file share which in my case is running as a VM on HC3, those files then get immediately "synced" to an Azure cloud based file share so I get rapid off-site and off-cluster protection but as those files get older and I start to fill my local file share, eventually older files are "stubbed" on my local file server to free that space and the data exists only in azure, yet can be retrieved automatically if it is accessed. So conceptually you could store years worth of HC3 exports on this Azure tiered share with only a small % of the overall storage needed on the ground. This feature is still in preview and there are a number of limitations on share size, etc. that exist today but is one interesting example that may be ready for prime time soon. https://azure.microsoft.com/en-us/blog/announcing-the-public-preview-for-azure-file-sync/
Would be interested in hearing what products and solutions HC3 users are using or are interested in using for deep and cheap, long term archive storage...
