Offsite Backup Solution Needed
-
@Dashrender said:
@Sparkum said:
Ran out of disk space, what I believe happened is that Veeam was getting ahead of itself creating the backup and the cache just kept growing while the offload was so slow so it just grew and grew then poof.
Unless thats not how it works then nevermind! haha
But ya, definately ran out of space, and I tried a bunch of things, kept failing, deleted the snapshot and poof, worked.
Well that could easily be it. If Veeam kicked off additional backups while the first one was still running, if you have very limited space on the host, that would probably explain it.
You can solve that by limiting Veeam to only allow one backup at a time per VM host.
There was only 1
Was testing so everything was being kicked off manually
-
@Sparkum said:
@Dashrender said:
@Sparkum said:
Ran out of disk space, what I believe happened is that Veeam was getting ahead of itself creating the backup and the cache just kept growing while the offload was so slow so it just grew and grew then poof.
Unless thats not how it works then nevermind! haha
But ya, definately ran out of space, and I tried a bunch of things, kept failing, deleted the snapshot and poof, worked.
Well that could easily be it. If Veeam kicked off additional backups while the first one was still running, if you have very limited space on the host, that would probably explain it.
You can solve that by limiting Veeam to only allow one backup at a time per VM host.
There was only 1
Was testing so everything was being kicked off manually
I thought you said Veeam was over zealous? what did you mean by that?
-
@Dashrender said:
@JaredBusch said:
@Dashrender said:
Replication itself doesn't need to involve snap shots though.
You can have Veeam take a backup of the VM to local NAS. Then you can have Veeam, as a different job, replicate that backup over the WAN. That replication won't touch the VM or make a snapshot.
It is completely impossible to not involve a snapshot.
How do you think the backup was made? With a snapshot. And that information is how the replication job would know what needed replicated.
correct, but it's a two step process.
- create backup - a) create snap b) copy data to backup repository c) delete snap
- replicate data from repository to remote location
As long as step 1 is done completely locally, you shouldn't have a problem with your snaps.
What we still don't know - and really is important before providing any advice of real value, is why the snaps caused the server to crash - if it even was really the snap that cause it.
i.e. did it run out of disk space? out of RAM(though that doesn't make sense) CPU overload, etc, etc, etc....
One possible story behind the crash - the snap was taken - the copy process starts but takes forever, the local VM host runs out of disk space - VM Host crashes.
But this is only one of many possible situations. In this situation local NAS for repository would solve the problem.
I have never used Veeam to replicate backup sets. Instead I replicate between hypervisors. I have no place doing this with a critical shortage of drive space though as is sounds @Sparkum does.
But my understanding of it is that it uses the data collected fromthe backup job to replicate the changes and then does the merge remotely. Just as it does the local merge when you hit the incremental limit set in the normal backup job.
-
A snap works (as I understand it) by creating a new VMDK file and setting the original VMDK to read only. All changes are made to the new VMDK - this new file will grow and grow as needed until you delete it. Deleting it merges all of the changes in the new file into the old file, and sets the old file to read/write again.
-
This is how Veeam replication works - https://www.veeam.com/university-course/backup-replication-how-it-works.html
This is for v7 but I am pretty sure that it is still the same in v9
Edit: you will have to click through several slides to get to the actual replication section.
-
@JaredBusch said:
But my understanding of it is that it uses the data collected fromthe backup job to replicate the changes and then does the merge remotely. Just as it does the local merge when you hit the incremental limit set in the normal backup job.
except that the merge is done on the backup repository storage, not the VM storage in the local non hypervisor replication scenario.
-
@JaredBusch said:
I have never used Veeam to replicate backup sets. Instead I replicate between hypervisors. I have no place doing this with a critical shortage of drive space though as is sounds @Sparkum does.
Awww - OK now we're seeing the differences. JB is doing replication through the hypervisor,
I'm talking about doing replication through the backup software.The advantage of doing it through the backup software is that the VM host in question is cut out of the loop entirely during the replication process.
-
@Sparkum said:
@Dashrender said:
@Sparkum said:
Ran out of disk space, what I believe happened is that Veeam was getting ahead of itself creating the backup and the cache just kept growing while the offload was so slow so it just grew and grew then poof.
Unless thats not how it works then nevermind! haha
But ya, definately ran out of space, and I tried a bunch of things, kept failing, deleted the snapshot and poof, worked.
Well that could easily be it. If Veeam kicked off additional backups while the first one was still running, if you have very limited space on the host, that would probably explain it.
You can solve that by limiting Veeam to only allow one backup at a time per VM host.
There was only 1
Was testing so everything was being kicked off manually
Now, in the case where you are running only 1 backup and you still run out of space - that just shows you how much change your doing. How much free space is there on the datastore of the VM host that crashed?
You might have to start by growing that datastore.
-
@Dashrender I bet it was during the initial full backup...
-
@Dashrender said:
@JaredBusch said:
I have never used Veeam to replicate backup sets. Instead I replicate between hypervisors. I have no place doing this with a critical shortage of drive space though as is sounds @Sparkum does.
Awww - OK now we're seeing the differences. JB is doing replication through the hypervisor,
I'm talking about doing replication through the backup software.No I was not. I clearly stated I used Veeam to make replicas in the hypervisor. Veeam is the backup software.
-
@Dashrender said:
The advantage of doing it through the backup software is that the VM host in question is cut out of the loop entirely during the replication process.
No it is not. Veeam does everything on the either the host or the repository in general. It can be told otherwise, but that is not default behavior.
-
@Dashrender said:
@Sparkum said:
@Dashrender said:
@Sparkum said:
Ran out of disk space, what I believe happened is that Veeam was getting ahead of itself creating the backup and the cache just kept growing while the offload was so slow so it just grew and grew then poof.
Unless thats not how it works then nevermind! haha
But ya, definately ran out of space, and I tried a bunch of things, kept failing, deleted the snapshot and poof, worked.
Well that could easily be it. If Veeam kicked off additional backups while the first one was still running, if you have very limited space on the host, that would probably explain it.
You can solve that by limiting Veeam to only allow one backup at a time per VM host.
There was only 1
Was testing so everything was being kicked off manually
Now, in the case where you are running only 1 backup and you still run out of space - that just shows you how much change your doing. How much free space is there on the datastore of the VM host that crashed?
You might have to start by growing that datastore.
He did not run out of space right away. He ran out of space while waiting for data to go out over the wire.
-
@JaredBusch said:
I have never used Veeam to replicate backup sets. Instead I replicate between hypervisors. the incremental limit set in the normal backup job.
What does this mean then, exactly?
You create a local backup with Veeam - which of course creates a snap.... and then you do a replication with Veeam from one hypervisor to another? why are you using Veeam to do that instead of the built in hypervisor tools? But that's really beside the point.
Doing that clearly makes the server run a snap twice (unless it can be run in a single job). and put strain on the VM host while replicating to the remote site.
Perhaps you gain the advantage of one less step on the remote side by doing it this way to have a nearly instant read to go failover setup?
-
@JaredBusch said:
@Dashrender said:
@Sparkum said:
@Dashrender said:
@Sparkum said:
Ran out of disk space, what I believe happened is that Veeam was getting ahead of itself creating the backup and the cache just kept growing while the offload was so slow so it just grew and grew then poof.
Unless thats not how it works then nevermind! haha
But ya, definately ran out of space, and I tried a bunch of things, kept failing, deleted the snapshot and poof, worked.
Well that could easily be it. If Veeam kicked off additional backups while the first one was still running, if you have very limited space on the host, that would probably explain it.
You can solve that by limiting Veeam to only allow one backup at a time per VM host.
There was only 1
Was testing so everything was being kicked off manually
Now, in the case where you are running only 1 backup and you still run out of space - that just shows you how much change your doing. How much free space is there on the datastore of the VM host that crashed?
You might have to start by growing that datastore.
He did not run out of space right away. He ran out of space while waiting for data to go out over the wire.
of course - let's assume he has 10 GB free space, and also assume his server makes 10 GB of changes an hour on average. that means that in less than one hour he will run out of space on that datastore. If the backup can be completed and the snap deleted all in under an hour, the system would be fine, but since he's pushing the data over a tiny pipe.. he ran out of time and space.
-
@Dashrender said:
You create a local backup with Veeam - which of course creates a snap.... and then you do a replication with Veeam from one hypervisor to another? why are you using Veeam to do that instead of the built in hypervisor tools? But that's really beside the point.
Because VMWare.
Doing that clearly makes the server run a snap twice (unless it can be run in a single job). and put strain on the VM host while replicating to the remote site.
- Backups run nightly.
- Not all servers are replicated
- Replication gives you more restore points throughout the day in addition to the failover capability
-
@JaredBusch said:
@Dashrender said:
You create a local backup with Veeam - which of course creates a snap.... and then you do a replication with Veeam from one hypervisor to another? why are you using Veeam to do that instead of the built in hypervisor tools? But that's really beside the point.
Because VMWare.
Doing that clearly makes the server run a snap twice (unless it can be run in a single job). and put strain on the VM host while replicating to the remote site.
- Backups run nightly.
- Not all servers are replicated
- Replication gives you more restore points throughout the day in addition to the failover capability
How do you get number 3?
It creates replication points? That's not how I've ever understood how replication works.
-
@Dashrender said:
@JaredBusch said:
@Dashrender said:
You create a local backup with Veeam - which of course creates a snap.... and then you do a replication with Veeam from one hypervisor to another? why are you using Veeam to do that instead of the built in hypervisor tools? But that's really beside the point.
Because VMWare.
Doing that clearly makes the server run a snap twice (unless it can be run in a single job). and put strain on the VM host while replicating to the remote site.
- Backups run nightly.
- Not all servers are replicated
- Replication gives you more restore points throughout the day in addition to the failover capability
How do you get number 3?
It creates replication points? That's not how I've ever understood how replication works.
Veeam 9 offers multiple restore points on replicas -
https://www.veeam.com/vm-advanced-replication.html
under failover and failback section. -
@wrx7m said:
@Dashrender said:
@JaredBusch said:
@Dashrender said:
You create a local backup with Veeam - which of course creates a snap.... and then you do a replication with Veeam from one hypervisor to another? why are you using Veeam to do that instead of the built in hypervisor tools? But that's really beside the point.
Because VMWare.
Doing that clearly makes the server run a snap twice (unless it can be run in a single job). and put strain on the VM host while replicating to the remote site.
- Backups run nightly.
- Not all servers are replicated
- Replication gives you more restore points throughout the day in addition to the failover capability
How do you get number 3?
It creates replication points? That's not how I've ever understood how replication works.
Veeam 9 offers multiple restore points on replicas -
https://www.veeam.com/vm-advanced-replication.html
under failover and failback section.Even Hyper-V has this built into replication. You can choose to keep XX number of replication points. Honestly this is not much different than people keeping XX snapshots on the local host for immediate rollback needs.
-
@JaredBusch Interesting. I don't know much about Hyper-V yet.
-
@MattSpeller said:
In a similar situation with limited bandwidth like that we just used Symantec to dump to tape. Nice people in a van came by each week to get the tapes.
This could easily be a good situation for tape. Or a better WAN.
