Storage and Data Locality
-
So I've been piddling about with an idea in my head and I want holes poked in it.
Essentially what I want to build is a storage setup between my Home (60 Mbit download, 5 Meg Upload) and my hosted server in France (100 Mbit both ways). To start with, this would be used for keeping files synchronized between Home and France, but eventually I'd like to use it to keep my Home VMs replicated to France, and my France VMs replicated to my home.
I'm not looking or "Instant replication" or anything like that. I just want the replication to happen...eventually. I know this could be done with rsync for static files, but I want to do VMs as well. I was just curious as to what other methods could be done.
I thought about Gluster or DRBD, but then data locality becomes an issue, right?
My VMs done with KVM on both systems.
Ideas, comments, tell me I'm crazy... Just looking for input and thoughts or other ideas I can try. This is a lab environment, so I'm up for pretty much anything within reason.
-
If your latency is low enough, @StarWind_Software can do this now. It's a newer feature of theirs. They no longer require that their cluster nodes be at local speeds.
-
-
Current latency is between 200 and 250ms over ZT.
-
@dafyre said in Storage and Data Locality:
Current latency is between 200 and 250ms over ZT.
That's a lot of latency, that'll make things hard. But an async replica is an option.
-
If I can offer any input, please do not use DRBD, it can corrupt your data pretty easily.
-
@scottalanmiller said in Storage and Data Locality:
@dafyre said in Storage and Data Locality:
Current latency is between 200 and 250ms over ZT.
That's a lot of latency, that'll make things hard. But an async replica is an option.
That's the term I was looking for. Async replica -- that's what I'm going for... In the event that my home network burns down, the goal is to be able to spin up anything I need in France.
-
@dbeato said in Storage and Data Locality:
If I can offer any input, please do not use DRBD, it can corrupt your data pretty easily.
Thanks for the forewarning -- that was one of the things I was considering.
-
I was thinking Veeam B&R but don't think you want to go to either Hyper-V or ESXi and dont have piles of cash just laying around, do you?
-
@nerdydad said in Storage and Data Locality:
I was thinking Veeam B&R but don't think you want to go to either Hyper-V or ESXi and dont have piles of cash just laying around, do you?
Yeah, this is a personal lab, so I don't have piles of cash lying around. Since my hosted server is limited in OS choices, I'm sticking with KVM. (ESXi and Hyper-V are not offered).
-
rsync?
-
I think it's just assumed that if your using KVM, that your also competent enough to write your own backup scripts. If you already have a backup script setup, just create a local backup and then rsync those files to the other box.
If you need a backup script still, well, I've been meaning to work on one, you just might be the motivation to get it done.
-
@travisdh1 said in Storage and Data Locality:
I think it's just assumed that if your using KVM, that your also competent enough to write your own backup scripts. If you already have a backup script setup, just create a local backup and then rsync those files to the other box.
This is just a steaming pile of shit.
Anyone can backup any system easy enough. KVM has nothing to do with it.
What you cannot do with a simple script is do differential or incremental backups such as Veeam and Unitrends do.
That is why backups are hard, and why there are not bazillions of backup solutions out there. -
@jaredbusch said in Storage and Data Locality:
@travisdh1 said in Storage and Data Locality:
I think it's just assumed that if your using KVM, that your also competent enough to write your own backup scripts. If you already have a backup script setup, just create a local backup and then rsync those files to the other box.
This is just a steaming pile of shit.
Anyone can backup any system easy enough. KVM has nothing to do with it.
What you cannot do with a simple script is do differential or incremental backups such as Veeam and Unitrends do.
That is why backups are hard, and why there are not bazillions of backup solutions out there.Also, usable automation. It's easy to automate a straight backup copy job; it's very hard to automate a system for restoring a specific file from a specific point in time.
-
I’ve used rsnapshot before and it works really well. Uses hard links to save space. Won’t help with your VM disks but will help with files.
-
ReaR is great as agent based backup for image based recovery and inceementals. For file based, I like FWBackups.
ReaR works great on stateless VMs, really easy and quick to recover.
-
@tim_g said in Storage and Data Locality:
ReaR is great as agent based backup for image based recovery and inceementals. For file based, I like FWBackups.
ReaR works great on stateless VMs, really easy and quick to recover.
Why are you backing up stateless machines?
-
@stacksofplates said in Storage and Data Locality:
@tim_g said in Storage and Data Locality:
ReaR is great as agent based backup for image based recovery and inceementals. For file based, I like FWBackups.
ReaR works great on stateless VMs, really easy and quick to recover.
Why are you backing up stateless machines?
Policy and DR.
-
@tim_g said in Storage and Data Locality:
@stacksofplates said in Storage and Data Locality:
@tim_g said in Storage and Data Locality:
ReaR is great as agent based backup for image based recovery and inceementals. For file based, I like FWBackups.
ReaR works great on stateless VMs, really easy and quick to recover.
Why are you backing up stateless machines?
Policy and DR.
Don't need it for DR. Then the question becomes, why did someone make that policy?
-
@tim_g said in Storage and Data Locality:
@stacksofplates said in Storage and Data Locality:
@tim_g said in Storage and Data Locality:
ReaR is great as agent based backup for image based recovery and inceementals. For file based, I like FWBackups.
ReaR works great on stateless VMs, really easy and quick to recover.
Why are you backing up stateless machines?
Policy and DR.
With stateless machines, wouldn't it be easier to automate the deployment, especially if your data is stored on another virtual disk or storage appliance?
