So I did ask around before posting this and I did receive warnings, but I have been testing this and it is very simple and dumb yet works (tested only with Linux VM Distro). sharing it here so you guys can bash me about my method, as well as enlighten me.
So you have your 2 KVM hosts okay + 1 Fedora machine with Virt Manager
You create all the VMs on the KVM and call it primary, let us call it KVM 1
the second KVM 2 will be just the slave incase KVM 1 drops.
So what we do is schedule a script to :
(optional )Stop Software VM activity like DB or WebServer sometime when it is not being utilized at midnight for example, systemctl stop httpd ..etc
Freeze the FileSystem (guest agent needs to be installed.)
virsh domfsfreeze VMNAME
Rsync the VM image to another KVM host (KVM 2) periodically each night.
rsync --progress --inplace -h
rsync --progress --inplace -h -W (safer)
Prepare previously in KVM 2 the VM xml (RAM/CPU)and make it same as the one in KVM1, also copy the MAC address. but keep VM2 shutdown.
You can script Fedora Machine to keep pinging KVM 1 or VM 1 and if it fails to receive thus it is down, so if that happens KVM 2 will virsh start VM2 , which will have the same everything even the same internal IP.
What do you think ? Too cheezy ? Does it show how much I am afraid to play with GLusterFS?
btw the tags are acting weird, whenever I type one it removes itself.
So the issue with rsyncing for anything other than a snapshot is the data will be different.
you rsync VM1 at 5AM to KVM2. At 3PM KVM1 goes down. VM1 will start up on KVM2, but any data written between 5AM and 3PM is lost. I wouldn't recommend rsync for anything other than copying disks as a backup or snapshot.
Gluster will do the real time replication you need for this.