KVM in Production - Build it yourself
-
I have 12 KVM hosts in production. For the very few machines that have stateful data, I use either the backup options built into the software/service that's running to a mounted location, my script, or I use ReaR. The rest of the systems are stored in Git.
ReaR is cli but is really easy to automate.
I ran an oVirt all in one before and it was really slow. It was on my DL380 with 96GB RAM and dual quad core Xeons, so I don't think it was the box. Bare KVM is just fast and easy.
-
@stacksofplates said in KVM in Production - Build it yourself:
I have 12 KVM hosts in production. For the very few machines that have stateful data, I use either the backup options built into the software/service that's running to a mounted location, my script, or I use ReaR. The rest of the systems are stored in Git.
This is not a workable solution for the average SMB right now.
Email is hosted for most place, but the internal infrastructure is not stateful in a normal SMB. Could it get there, or close? Certainly, but that is not reality.
The problem with using KVM in production is the lack of robust backup solutions like Veeam that handle hypervisor level backups with things like forever forward incrementals.
Making a new snapshot every day sucks donkey balls.
I have not looked into @stacksofplates script yet to see what he has done, but you get my point.
-
Oops, I forgot to include the link in my previous post...
KVM host based backup...https://storware.eu/en/storware-vprotect/
-
@stacksofplates said in KVM in Production - Build it yourself:
I have 12 KVM hosts in production. For the very few machines that have stateful data, I use either the backup options built into the software/service that's running to a mounted location, my script, or I use ReaR. The rest of the systems are stored in Git.
ReaR is cli but is really easy to automate.
I ran an oVirt all in one before and it was really slow. It was on my DL380 with 96GB RAM and dual quad core Xeons, so I don't think it was the box. Bare KVM is just fast and easy.
I'm with @JaredBusch on this one, no way this sounds user friendly or easy to use.
-
@fateknollogee said in KVM in Production - Build it yourself:
Oops, I forgot to include the link in my previous post...
KVM host based backup...https://storware.eu/en/storware-vprotect/
How much does it cost?
-
@jaredbusch said in KVM in Production - Build it yourself:
@stacksofplates said in KVM in Production - Build it yourself:
I have 12 KVM hosts in production. For the very few machines that have stateful data, I use either the backup options built into the software/service that's running to a mounted location, my script, or I use ReaR. The rest of the systems are stored in Git.
This is not a workable solution for the average SMB right now.
Email is hosted for most place, but the internal infrastructure is not stateful in a normal SMB. Could it get there, or close? Certainly, but that is not reality.
The problem with using KVM in production is the lack of robust backup solutions like Veeam that handle hypervisor level backups with things like forever forward incrementals.
Making a new snapshot every day sucks donkey balls.
I have not looked into @stacksofplates script yet to see what he has done, but you get my point.
The script does do an external snapshot and copy that. But ReaR does do incrementals which is why I use that more often.
I think if most SMBs looked outside of Windows they would find it's much easier to automate everything. Our DNS, DHCP, and workstation provisioning (kickstarts) is all done through Ansible. I did all of it, so I think it's a fair comparison to an SMB who has one person doing the work. I think those would be the big pieces of what small shops would need, and from there most everything else could be automated.
I agree 100% that most aren't that way now. I think it just takes a different mindset and different way of looking at it. I'm also not trying to get everyone to do this. I was just answering the question that was asked in the OP.
-
@dustinb3403 said in KVM in Production - Build it yourself:
@fateknollogee said in KVM in Production - Build it yourself:
Oops, I forgot to include the link in my previous post...
KVM host based backup...https://storware.eu/en/storware-vprotect/
How much does it cost?
I think it was 600 Euro per host.
I'll have to dig up the price list. -
@fateknollogee said in KVM in Production - Build it yourself:
@stacksofplates said in KVM in Production - Build it yourself:
I have 12 KVM hosts in production. For the very few machines that have stateful data, I use either the backup options built into the software/service that's running to a mounted location, my script, or I use ReaR. The rest of the systems are stored in Git.
ReaR is cli but is really easy to automate.
I ran an oVirt all in one before and it was really slow. It was on my DL380 with 96GB RAM and dual quad core Xeons, so I don't think it was the box. Bare KVM is just fast and easy.
I'm with @JaredBusch on this one, no way this sounds user friendly or easy to use.
There isn't really any need for user friendly anything. It's just easy to use natively. I click one button it builds the server and adds the data it needs. It's really the easiest thing to do. Now I can have Jr. admins build machines exactly the same way every time without any changes. Everything is done through Tower. No more manually logging into boxes to make changes (unless you're trying to learn how something works but that's not in prod).
-
@fateknollogee said in KVM in Production - Build it yourself:
@dustinb3403 said in KVM in Production - Build it yourself:
@fateknollogee said in KVM in Production - Build it yourself:
Oops, I forgot to include the link in my previous post...
KVM host based backup...https://storware.eu/en/storware-vprotect/
How much does it cost?
I think it was 600 Euro per host.
I'll have to dig up the price list.€ 600 per host (2 sockets), w Standard Support includes 1 year maintenance
€ 150 per year Standard Support - 8h response time, 8h/5 working days
€ xxx per year Premium Support - 4h response time, 24h/5 working days
€ 375 per year Titanium Support - 4h response time, 24h/7 working days -
Although I have no issues with KVM, I wouldn't personally use it in a production environment, at least there are more backup solutions for Xenserver and Hyper-V
Even though the backup script gets the job done, I wouldn't exactly call it practical.
-
There appears to be lots of backup solutions for KVM. And these solutions are the best ones. They are simple BASH scripts that do exactly what they are supposed to do, and some PERL scripts.
The best kinds of backups are these, and that you can configure, and forget about, with the exception of testing restores occasionally.
These backup solutions are the least likely to fail.
When you get into vendor-specific and GUI-based backups, that's when thing start to get weird and lose potential reliability.
I prefer the CLI-based backups for production. They are the most reliable.
I've NEVER had a powershell / BASH scripted backup fail. I cannot say the same for any GUI-based backup (Veeam, WSB, etc.). In fact, I use PowerShell / BASH to "clean-up or fix" GUI-based backup failures.
-
@tim_g said in KVM in Production - Build it yourself:
There appears to be lots of backup solutions for KVM. And these solutions are the best ones. They are simple BASH scripts that do exactly what they are supposed to do, and some PERL scripts.
The best kinds of backups are these, and that you can configure, and forget about, with the exception of testing restores occasionally.
These backup solutions are the least likely to fail.
When you get into vendor-specific and GUI-based backups, that's when thing start to get weird and lose potential reliability.
I prefer the CLI-based backups for production. They are the most reliable.
I've NEVER had a powershell / BASH scripted backup fail. I cannot say the same for any GUI-based backup (Veeam, WSB, etc.). In fact, I use PowerShell / BASH to "clean-up or fix" GUI-based backup failures.
I wouldn't mind a text-based UI backups.
-
@black3dynamite said in KVM in Production - Build it yourself:
@tim_g said in KVM in Production - Build it yourself:
There appears to be lots of backup solutions for KVM. And these solutions are the best ones. They are simple BASH scripts that do exactly what they are supposed to do, and some PERL scripts.
The best kinds of backups are these, and that you can configure, and forget about, with the exception of testing restores occasionally.
These backup solutions are the least likely to fail.
When you get into vendor-specific and GUI-based backups, that's when thing start to get weird and lose potential reliability.
I prefer the CLI-based backups for production. They are the most reliable.
I've NEVER had a powershell / BASH scripted backup fail. I cannot say the same for any GUI-based backup (Veeam, WSB, etc.). In fact, I use PowerShell / BASH to "clean-up or fix" GUI-based backup failures.
I wouldn't mind a text-based UI backups.
Yeah, that's great for full backups. But suppose you have a multi-TB sized VM... maybe you want to do a full backup weekly, and incrementals daily. I don't know how you'd accomplish that in the same way.
-
I suppose it, like everything else, comes down to your specific environmental needs.
-
Text, CLI based doesn't bother me, I would like to see actual speed of VM backups using the script, and does it enable incremental backups of the VM?
-
I amazed that cockpit doesn't have some kind of package for managing backups. That would be awesome.
Someone with mad skills could probably create a backup plugin for Cockpit.
http://cockpit-project.org/blog/creating-plugins-for-the-cockpit-user-interface.html -
Seems that Bacula has a KVM plugin: https://www.baculasystems.com/enterprise-backup-solution-with-bacula-systems/virtual-machine-backup-software . Any experience with that?
-
I think what @DustinB3403 feels bad is the mix of config and logic in the same file. If you would be able to separate the working logic from config, then edit a bunch of lines in a txt would not be so terrible .
Conceptually it should mot be so terrible:
1- snap
2- mount snap and backup location
3- run borg backup or similar
4- unmount everythingTesting is the most long part. With stuff like borg you should be confident in both backup and recovery.
Acutally recovery is trickier?! Don't know, should be like:
1- shutdown the machine
2- mount machine and backup location
3- restoreIssue is: how to programmatically tell borg which snap to use to recover...
I guess 2 of days of work... And I'm not super fast at writing code.
What has stopped me from using kvm in production was: what if I have to off load stuff to other people? Will they be
skilled enough for this crap?! So I just moved to hyperv+altaro... -
@matteo-nunziati said in KVM in Production - Build it yourself:
What has stopped me from using kvm in production was: what if I have to off load stuff to other people? Will they be
skilled enough for this crap?! So I just moved to hyperv+altaro...That's sad. I hope some user friendly solution will come out soon. I'm in the KVM boat since 2010, KVM is SO good today. Maybe it will get faster adoption after the AWS move.
-
@stacksofplates said in KVM in Production - Build it yourself:
I have 12 KVM hosts in production. For the very few machines that have stateful data, I use either the backup options built into the software/service that's running to a mounted location, my script, or I use ReaR. The rest of the systems are stored in Git.
Bare KVM is just fast and easy. I second that, but the issue there is no standard way to manage it, especially backups, so this leaves KVM a solution for person that knows much about, and usually the sole IT in that location, cause if there was other IT folks, they just pick ESXi or Hyper-V to ensure continuity.
