Offsite Backup Solution Needed
-
@Dashrender said:
The advantage of doing it through the backup software is that the VM host in question is cut out of the loop entirely during the replication process.
No it is not. Veeam does everything on the either the host or the repository in general. It can be told otherwise, but that is not default behavior.
-
@Dashrender said:
@Sparkum said:
@Dashrender said:
@Sparkum said:
Ran out of disk space, what I believe happened is that Veeam was getting ahead of itself creating the backup and the cache just kept growing while the offload was so slow so it just grew and grew then poof.
Unless thats not how it works then nevermind! haha
But ya, definately ran out of space, and I tried a bunch of things, kept failing, deleted the snapshot and poof, worked.
Well that could easily be it. If Veeam kicked off additional backups while the first one was still running, if you have very limited space on the host, that would probably explain it.
You can solve that by limiting Veeam to only allow one backup at a time per VM host.
There was only 1
Was testing so everything was being kicked off manually
Now, in the case where you are running only 1 backup and you still run out of space - that just shows you how much change your doing. How much free space is there on the datastore of the VM host that crashed?
You might have to start by growing that datastore.
He did not run out of space right away. He ran out of space while waiting for data to go out over the wire.
-
@JaredBusch said:
I have never used Veeam to replicate backup sets. Instead I replicate between hypervisors. the incremental limit set in the normal backup job.
What does this mean then, exactly?
You create a local backup with Veeam - which of course creates a snap.... and then you do a replication with Veeam from one hypervisor to another? why are you using Veeam to do that instead of the built in hypervisor tools? But that's really beside the point.
Doing that clearly makes the server run a snap twice (unless it can be run in a single job). and put strain on the VM host while replicating to the remote site.
Perhaps you gain the advantage of one less step on the remote side by doing it this way to have a nearly instant read to go failover setup?
-
@JaredBusch said:
@Dashrender said:
@Sparkum said:
@Dashrender said:
@Sparkum said:
Ran out of disk space, what I believe happened is that Veeam was getting ahead of itself creating the backup and the cache just kept growing while the offload was so slow so it just grew and grew then poof.
Unless thats not how it works then nevermind! haha
But ya, definately ran out of space, and I tried a bunch of things, kept failing, deleted the snapshot and poof, worked.
Well that could easily be it. If Veeam kicked off additional backups while the first one was still running, if you have very limited space on the host, that would probably explain it.
You can solve that by limiting Veeam to only allow one backup at a time per VM host.
There was only 1
Was testing so everything was being kicked off manually
Now, in the case where you are running only 1 backup and you still run out of space - that just shows you how much change your doing. How much free space is there on the datastore of the VM host that crashed?
You might have to start by growing that datastore.
He did not run out of space right away. He ran out of space while waiting for data to go out over the wire.
of course - let's assume he has 10 GB free space, and also assume his server makes 10 GB of changes an hour on average. that means that in less than one hour he will run out of space on that datastore. If the backup can be completed and the snap deleted all in under an hour, the system would be fine, but since he's pushing the data over a tiny pipe.. he ran out of time and space.
-
@Dashrender said:
You create a local backup with Veeam - which of course creates a snap.... and then you do a replication with Veeam from one hypervisor to another? why are you using Veeam to do that instead of the built in hypervisor tools? But that's really beside the point.
Because VMWare.
Doing that clearly makes the server run a snap twice (unless it can be run in a single job). and put strain on the VM host while replicating to the remote site.
- Backups run nightly.
- Not all servers are replicated
- Replication gives you more restore points throughout the day in addition to the failover capability
-
@JaredBusch said:
@Dashrender said:
You create a local backup with Veeam - which of course creates a snap.... and then you do a replication with Veeam from one hypervisor to another? why are you using Veeam to do that instead of the built in hypervisor tools? But that's really beside the point.
Because VMWare.
Doing that clearly makes the server run a snap twice (unless it can be run in a single job). and put strain on the VM host while replicating to the remote site.
- Backups run nightly.
- Not all servers are replicated
- Replication gives you more restore points throughout the day in addition to the failover capability
How do you get number 3?
It creates replication points? That's not how I've ever understood how replication works.
-
@Dashrender said:
@JaredBusch said:
@Dashrender said:
You create a local backup with Veeam - which of course creates a snap.... and then you do a replication with Veeam from one hypervisor to another? why are you using Veeam to do that instead of the built in hypervisor tools? But that's really beside the point.
Because VMWare.
Doing that clearly makes the server run a snap twice (unless it can be run in a single job). and put strain on the VM host while replicating to the remote site.
- Backups run nightly.
- Not all servers are replicated
- Replication gives you more restore points throughout the day in addition to the failover capability
How do you get number 3?
It creates replication points? That's not how I've ever understood how replication works.
Veeam 9 offers multiple restore points on replicas -
https://www.veeam.com/vm-advanced-replication.html
under failover and failback section. -
@wrx7m said:
@Dashrender said:
@JaredBusch said:
@Dashrender said:
You create a local backup with Veeam - which of course creates a snap.... and then you do a replication with Veeam from one hypervisor to another? why are you using Veeam to do that instead of the built in hypervisor tools? But that's really beside the point.
Because VMWare.
Doing that clearly makes the server run a snap twice (unless it can be run in a single job). and put strain on the VM host while replicating to the remote site.
- Backups run nightly.
- Not all servers are replicated
- Replication gives you more restore points throughout the day in addition to the failover capability
How do you get number 3?
It creates replication points? That's not how I've ever understood how replication works.
Veeam 9 offers multiple restore points on replicas -
https://www.veeam.com/vm-advanced-replication.html
under failover and failback section.Even Hyper-V has this built into replication. You can choose to keep XX number of replication points. Honestly this is not much different than people keeping XX snapshots on the local host for immediate rollback needs.
-
@JaredBusch Interesting. I don't know much about Hyper-V yet.
-
@MattSpeller said:
In a similar situation with limited bandwidth like that we just used Symantec to dump to tape. Nice people in a van came by each week to get the tapes.
This could easily be a good situation for tape. Or a better WAN.
-
@Sparkum said:
And ya increasing the line at the store is COMPLETELY an option. Just trying to weigh all my options here.
That carries so many other benefits too, like the restore speed, ability to support them in a good way, etc. That's likely the best bet. Or just tape. Tape really is what people do in these situations.
-
@Sparkum said:
@Dashrender said:
@JaredBusch said:
@Dashrender said:
Replication itself doesn't need to involve snap shots though.
You can have Veeam take a backup of the VM to local NAS. Then you can have Veeam, as a different job, replicate that backup over the WAN. That replication won't touch the VM or make a snapshot.
It is completely impossible to not involve a snapshot.
How do you think the backup was made? With a snapshot. And that information is how the replication job would know what needed replicated.
correct, but it's a two step process.
- create backup - a) create snap b) copy data to backup repository c) delete snap
- replicate data from repository to remote location
As long as step 1 is done completely locally, you shouldn't have a problem with your snaps.
What we still don't know - and really is important before providing any advice of real value, is why the snaps caused the server to crash - if it even was really the snap that cause it.
i.e. did it run out of disk space? out of RAM(though that doesn't make sense) CPU overload, etc, etc, etc....
One possible story behind the crash - the snap was taken - the copy process starts but takes forever, the local VM host runs out of disk space - VM Host crashes.
But this is only one of many possible situations. In this situation local NAS for repository would solve the problem.
Ran out of disk space, what I believe happened is that Veeam was getting ahead of itself creating the backup and the cache just kept growing while the offload was so slow so it just grew and grew then poof.
Unless thats not how it works then nevermind! haha
But ya, definately ran out of space, and I tried a bunch of things, kept failing, deleted the snapshot and poof, worked.
If you want to get away from this issue, you have to change toolsets. Look at someone like StorageCraft instead. No snapshots involved, it is the backup system's agentless API talking to VMware that is your problem here. You need a backtool with an agent.
-
@Sparkum said:
Maybe what happened with the VM filling up and crashing is something stupid that I could fix with a simple phone call to Veeam, but like every company we have certain servers that cant have downtime during business hours, so if something like that happened on one of our critical servers shit will hit the fan so fast.
As mentioned again by others, your "VM" is not filling up. You are running out of space on your datastore during the snapshot process. Can you post your datastore size and free space?
Like JB, I also do replication of certain VMs. I don't count it really as my real backup, but rather like shadow copies in Windows. For example, we have monthlies to tape and a typical tape rotational schedule. Sometimes I need to restore something pretty far back and came to find that since I offload the backup to tape, it can take up to 3 hours just to get the file first off the tape and then get Veeam to restore what I need. So I had an aha moment. I now craft a replica job for my project server each month to a cheap HP Microserver that mirrors my monthly backup tape schedule. I keep 12 replicas corresponding to each month. Now if someone wants something restored from 10 months ago, I can restore it from the replica way faster.
This software is the shiznit.
-
@JaredBusch said:
@wrx7m said:
@Dashrender said:
@JaredBusch said:
@Dashrender said:
You create a local backup with Veeam - which of course creates a snap.... and then you do a replication with Veeam from one hypervisor to another? why are you using Veeam to do that instead of the built in hypervisor tools? But that's really beside the point.
Because VMWare.
Doing that clearly makes the server run a snap twice (unless it can be run in a single job). and put strain on the VM host while replicating to the remote site.
- Backups run nightly.
- Not all servers are replicated
- Replication gives you more restore points throughout the day in addition to the failover capability
How do you get number 3?
It creates replication points? That's not how I've ever understood how replication works.
Veeam 9 offers multiple restore points on replicas -
https://www.veeam.com/vm-advanced-replication.html
under failover and failback section.Even Hyper-V has this built into replication. You can choose to keep XX number of replication points. Honestly this is not much different than people keeping XX snapshots on the local host for immediate rollback needs.
you can't pick and choose roll backs, if you pick two snaps ago, you loose the one from one snap ago.
-
@Dashrender said:
@JaredBusch said:
@wrx7m said:
@Dashrender said:
@JaredBusch said:
@Dashrender said:
You create a local backup with Veeam - which of course creates a snap.... and then you do a replication with Veeam from one hypervisor to another? why are you using Veeam to do that instead of the built in hypervisor tools? But that's really beside the point.
Because VMWare.
Doing that clearly makes the server run a snap twice (unless it can be run in a single job). and put strain on the VM host while replicating to the remote site.
- Backups run nightly.
- Not all servers are replicated
- Replication gives you more restore points throughout the day in addition to the failover capability
How do you get number 3?
It creates replication points? That's not how I've ever understood how replication works.
Veeam 9 offers multiple restore points on replicas -
https://www.veeam.com/vm-advanced-replication.html
under failover and failback section.Even Hyper-V has this built into replication. You can choose to keep XX number of replication points. Honestly this is not much different than people keeping XX snapshots on the local host for immediate rollback needs.
you can't pick and choose roll backs, if you pick two snaps ago, you loose the one from one snap ago.
See above about replication I just posted. I'm using replication like JB in that I'm just using it for convenient backup not true replication for DR failover. JB may be doing it similarly.
-
@DenisKelley said:
@Dashrender said:
@JaredBusch said:
@wrx7m said:
@Dashrender said:
@JaredBusch said:
@Dashrender said:
You create a local backup with Veeam - which of course creates a snap.... and then you do a replication with Veeam from one hypervisor to another? why are you using Veeam to do that instead of the built in hypervisor tools? But that's really beside the point.
Because VMWare.
Doing that clearly makes the server run a snap twice (unless it can be run in a single job). and put strain on the VM host while replicating to the remote site.
- Backups run nightly.
- Not all servers are replicated
- Replication gives you more restore points throughout the day in addition to the failover capability
How do you get number 3?
It creates replication points? That's not how I've ever understood how replication works.
Veeam 9 offers multiple restore points on replicas -
https://www.veeam.com/vm-advanced-replication.html
under failover and failback section.Even Hyper-V has this built into replication. You can choose to keep XX number of replication points. Honestly this is not much different than people keeping XX snapshots on the local host for immediate rollback needs.
you can't pick and choose roll backs, if you pick two snaps ago, you loose the one from one snap ago.
See above about replication I just posted. I'm using replication like JB in that I'm just using it for convenient backup not true replication for DR failover. JB may be doing it similarly.
Does that use 12x time disk space of the VM?
-
@Dashrender said:
@DenisKelley said:
@Dashrender said:
@JaredBusch said:
@wrx7m said:
@Dashrender said:
@JaredBusch said:
@Dashrender said:
You create a local backup with Veeam - which of course creates a snap.... and then you do a replication with Veeam from one hypervisor to another? why are you using Veeam to do that instead of the built in hypervisor tools? But that's really beside the point.
Because VMWare.
Doing that clearly makes the server run a snap twice (unless it can be run in a single job). and put strain on the VM host while replicating to the remote site.
- Backups run nightly.
- Not all servers are replicated
- Replication gives you more restore points throughout the day in addition to the failover capability
How do you get number 3?
It creates replication points? That's not how I've ever understood how replication works.
Veeam 9 offers multiple restore points on replicas -
https://www.veeam.com/vm-advanced-replication.html
under failover and failback section.Even Hyper-V has this built into replication. You can choose to keep XX number of replication points. Honestly this is not much different than people keeping XX snapshots on the local host for immediate rollback needs.
you can't pick and choose roll backs, if you pick two snaps ago, you loose the one from one snap ago.
See above about replication I just posted. I'm using replication like JB in that I'm just using it for convenient backup not true replication for DR failover. JB may be doing it similarly.
Does that use 12x time disk space of the VM?
Not at all. It similar to Forward incremental. One big file with delta snaps.
-
@DenisKelley said:
@Dashrender said:
@DenisKelley said:
@Dashrender said:
@JaredBusch said:
@wrx7m said:
@Dashrender said:
@JaredBusch said:
@Dashrender said:
You create a local backup with Veeam - which of course creates a snap.... and then you do a replication with Veeam from one hypervisor to another? why are you using Veeam to do that instead of the built in hypervisor tools? But that's really beside the point.
Because VMWare.
Doing that clearly makes the server run a snap twice (unless it can be run in a single job). and put strain on the VM host while replicating to the remote site.
- Backups run nightly.
- Not all servers are replicated
- Replication gives you more restore points throughout the day in addition to the failover capability
How do you get number 3?
It creates replication points? That's not how I've ever understood how replication works.
Veeam 9 offers multiple restore points on replicas -
https://www.veeam.com/vm-advanced-replication.html
under failover and failback section.Even Hyper-V has this built into replication. You can choose to keep XX number of replication points. Honestly this is not much different than people keeping XX snapshots on the local host for immediate rollback needs.
you can't pick and choose roll backs, if you pick two snaps ago, you loose the one from one snap ago.
See above about replication I just posted. I'm using replication like JB in that I'm just using it for convenient backup not true replication for DR failover. JB may be doing it similarly.
Does that use 12x time disk space of the VM?
Not at all. It similar to Forward incremental. One big file with delta snaps.
Cool feature in Veeam.
-
@Dashrender said:
@DenisKelley said:
@Dashrender said:
@DenisKelley said:
@Dashrender said:
@JaredBusch said:
@wrx7m said:
@Dashrender said:
@JaredBusch said:
@Dashrender said:
You create a local backup with Veeam - which of course creates a snap.... and then you do a replication with Veeam from one hypervisor to another? why are you using Veeam to do that instead of the built in hypervisor tools? But that's really beside the point.
Because VMWare.
Doing that clearly makes the server run a snap twice (unless it can be run in a single job). and put strain on the VM host while replicating to the remote site.
- Backups run nightly.
- Not all servers are replicated
- Replication gives you more restore points throughout the day in addition to the failover capability
How do you get number 3?
It creates replication points? That's not how I've ever understood how replication works.
Veeam 9 offers multiple restore points on replicas -
https://www.veeam.com/vm-advanced-replication.html
under failover and failback section.Even Hyper-V has this built into replication. You can choose to keep XX number of replication points. Honestly this is not much different than people keeping XX snapshots on the local host for immediate rollback needs.
you can't pick and choose roll backs, if you pick two snaps ago, you loose the one from one snap ago.
See above about replication I just posted. I'm using replication like JB in that I'm just using it for convenient backup not true replication for DR failover. JB may be doing it similarly.
Does that use 12x time disk space of the VM?
Not at all. It similar to Forward incremental. One big file with delta snaps.
Cool feature in Veeam.
Built into every replication system I am aware of.
-
@JaredBusch said:
@Dashrender said:
@DenisKelley said:
@Dashrender said:
@DenisKelley said:
@Dashrender said:
@JaredBusch said:
@wrx7m said:
@Dashrender said:
@JaredBusch said:
@Dashrender said:
You create a local backup with Veeam - which of course creates a snap.... and then you do a replication with Veeam from one hypervisor to another? why are you using Veeam to do that instead of the built in hypervisor tools? But that's really beside the point.
Because VMWare.
Doing that clearly makes the server run a snap twice (unless it can be run in a single job). and put strain on the VM host while replicating to the remote site.
- Backups run nightly.
- Not all servers are replicated
- Replication gives you more restore points throughout the day in addition to the failover capability
How do you get number 3?
It creates replication points? That's not how I've ever understood how replication works.
Veeam 9 offers multiple restore points on replicas -
https://www.veeam.com/vm-advanced-replication.html
under failover and failback section.Even Hyper-V has this built into replication. You can choose to keep XX number of replication points. Honestly this is not much different than people keeping XX snapshots on the local host for immediate rollback needs.
you can't pick and choose roll backs, if you pick two snaps ago, you loose the one from one snap ago.
See above about replication I just posted. I'm using replication like JB in that I'm just using it for convenient backup not true replication for DR failover. JB may be doing it similarly.
Does that use 12x time disk space of the VM?
Not at all. It similar to Forward incremental. One big file with delta snaps.
Cool feature in Veeam.
Built into every replication system I am aware of.
Pretty much. It's very standard.
