Does block level sync exist?
-
@Fredtx said in Does block level sync exist?:
The 3-2-1 strategy that Barracuda offers is a backup is copied to a local backup appliance, which is then deduplicated to save storage space on the appliance. Then, the data from the appliance is compressed and replicated to an offsite storage, which can be another appliance (our setup in my division) or their cloud storage, which I believe is in AWS.
Just for reference, we do this using Duplicati. Rolling it out right now after testing. We've used it for years, but now making it our primary tool. It doesn't do the EXACT same process you are describing, but very similar. But single tool doing the 3-2-1 in an essentially similar way.
We are doing one local copy, one copy to NAS, a snapshot of the NAS, and an offsite to S2 or Wasabi.
-
But we are aware that Duplicati, like Veeam or Baraccuda, isn't application aware (of our applications) and that we have to accommodate for that and take a backup using application aware processes locally, then let Duplicati take a backup of THAT backup.
-
@scottalanmiller said in Does block level sync exist?:
Duplicati
What is Duplicati? It sound like something you install locally on each server that will send backups to NAS/cloud/wherever. Is that correct?
-
@Pete-S said in Does block level sync exist?:
@scottalanmiller said in Does block level sync exist?:
Duplicati
What is Duplicati? It sound like something you install locally on each server that will send backups to NAS/cloud/wherever. Is that correct?
Yup, it's backup software. Does both image and file based. Can send to local, remote, or cloud destinations. It's an agent, so it installs ON Windows, MacOS or Linux.
-
-
@scottalanmiller said in Does block level sync exist?:
@Pete-S said in Does block level sync exist?:
@scottalanmiller said in Does block level sync exist?:
Duplicati
What is Duplicati? It sound like something you install locally on each server that will send backups to NAS/cloud/wherever. Is that correct?
Yup, it's backup software. Does both image and file based. Can send to local, remote, or cloud destinations. It's an agent, so it installs ON Windows, MacOS or Linux.
How do you keep track of all the backups if you have hundreds of duplicati installations running? There are no central backup server or UI right? All installations are independent of each other?
-
@scottalanmiller said in Does block level sync exist?:
You are worrying that the copies of the copies don't change, but not worrying if the original copy is any good.
My main concern is that the copies are good in all locations. In my case with Barracuda, the copies were not good. Actually, even worse, they were not even there.
-
@Fredtx said in Does block level sync exist?:
My main concern is that the copies are good in all locations
No your main concern is that your primary backup is good.
After that, it is just a copy operation. If you are worried about a copy operation, your solution is doing something wrong.
-
@Fredtx said in Does block level sync exist?:
@scottalanmiller said in Does block level sync exist?:
You are worrying that the copies of the copies don't change, but not worrying if the original copy is any good.
My main concern is that the copies are good in all locations. In my case with Barracuda, the copies were not good. Actually, even worse, they were not even there.
Why do you care if you have copies if you don't know what they are copies of? Unless politically you are responsible for the copies but not the backup. Then, that makes sense. If they backup is bad, that's "Betty's fault", but if the sync of the backup is bad, that's your fault. That would make sense.
If that's not the case, you are looking at this all wrong. As a company, the backup is what matters and the sync is completely irrelevant until that is known to be good and once it is good, the sync can be pretty important, but never as important as the thing that is being sync.
It's similar to caring about how good the lock on your safe is but not caring that the back of the safe is missing and people can just walk by and steal everything without opening the door in the first place.
-
@Pete-S said in Does block level sync exist?:
@scottalanmiller said in Does block level sync exist?:
@Pete-S said in Does block level sync exist?:
@scottalanmiller said in Does block level sync exist?:
Duplicati
What is Duplicati? It sound like something you install locally on each server that will send backups to NAS/cloud/wherever. Is that correct?
Yup, it's backup software. Does both image and file based. Can send to local, remote, or cloud destinations. It's an agent, so it installs ON Windows, MacOS or Linux.
How do you keep track of all the backups if you have hundreds of duplicati installations running? There are no central backup server or UI right? All installations are independent of each other?
Ah, excellent question. So there is no central server, that is correct. All installations are independent, also correct. But they are open source and have some APIs and messaging options. That's great when you have one or two. But with hundreds (like we will have once the conversion is done) you need something more.
Thankfully there is a non-open source Duplicati monitoring service that we are using that is cheap and does just that. It does NOT control them, only monitors. But honestly, that's perfect. It is so easy to use MeshCentral to hop on and manage the instances when they need something. The central system just tells us that all is well or not. So while Duplicati ITSELF doesn't do that, third party tools do and that's what we are doing now.
Also talking about building our own open source solution to do that for reasons of integration into other things (namely helpdesk) but that hasn't gone very far since the other solution is cheap and working.
-
@scottalanmiller said in Does block level sync exist?:
@Pete-S said in Does block level sync exist?:
@scottalanmiller said in Does block level sync exist?:
@Pete-S said in Does block level sync exist?:
@scottalanmiller said in Does block level sync exist?:
Duplicati
What is Duplicati? It sound like something you install locally on each server that will send backups to NAS/cloud/wherever. Is that correct?
Yup, it's backup software. Does both image and file based. Can send to local, remote, or cloud destinations. It's an agent, so it installs ON Windows, MacOS or Linux.
How do you keep track of all the backups if you have hundreds of duplicati installations running? There are no central backup server or UI right? All installations are independent of each other?
Ah, excellent question. So there is no central server, that is correct. All installations are independent, also correct. But they are open source and have some APIs and messaging options. That's great when you have one or two. But with hundreds (like we will have once the conversion is done) you need something more.
Thankfully there is a non-open source Duplicati monitoring service that we are using that is cheap and does just that. It does NOT control them, only monitors. But honestly, that's perfect. It is so easy to use MeshCentral to hop on and manage the instances when they need something. The central system just tells us that all is well or not. So while Duplicati ITSELF doesn't do that, third party tools do and that's what we are doing now.
Also talking about building our own open source solution to do that for reasons of integration into other things (namely helpdesk) but that hasn't gone very far since the other solution is cheap and working.
Great, thanks for clarifying. I often find myself looking at a product amazed at how badly they manage to explain what it's for and how it works. But you summed it up nicely.
-
@scottalanmiller Let me clarify. I want to make sure the "good" backups are copied to the offsite storage. So if the building were to catch on fire or something, and the good copies are destroyed. I would want to be able to restore from the offsite storage. In my case, some of the data was missing from the offsite storage that should have been replicated from the local "good" backup. Not sure what happened, and why it was not copied over, but it did not. I figured there would be some kind of sync mechanism that would have caught that ahead of time, which Barracuda said there is no such sync. That is why I reached out to the community.
-
@Fredtx said in Does block level sync exist?:
@scottalanmiller Let me clarify. I want to make sure the "good" backups are copied to the offsite storage. So if the building were to catch on fire or something, and the good copies are destroyed. I would want to be able to restore from the offsite storage. In my case, some of the data was missing from the offsite storage that should have been replicated from the local "good" backup. Not sure what happened, and why it was not copied over, but it did not. I figured there would be some kind of sync mechanism that would have caught that ahead of time, which Barracuda said there is no such sync. That is why I reached out to the community.
We understand. And that's important because clearly your sync failed. It's just that it also exposed the fact that the original backups are not application aware (unless there is no application) so something that you should see as a very, very large issue. If you are responsible for the backups, that is. Otherwise, not your monkeys, not your circus.
-
@scottalanmiller said in Does block level sync exist?:
@Fredtx said in Does block level sync exist?:
@scottalanmiller Let me clarify. I want to make sure the "good" backups are copied to the offsite storage. So if the building were to catch on fire or something, and the good copies are destroyed. I would want to be able to restore from the offsite storage. In my case, some of the data was missing from the offsite storage that should have been replicated from the local "good" backup. Not sure what happened, and why it was not copied over, but it did not. I figured there would be some kind of sync mechanism that would have caught that ahead of time, which Barracuda said there is no such sync. That is why I reached out to the community.
We understand. And that's important because clearly your sync failed. It's just that it also exposed the fact that the original backups are not application aware (unless there is no application) so something that you should see as a very, very large issue. If you are responsible for the backups, that is. Otherwise, not your monkeys, not your circus.
You're making an assumption that there's an app to backup - which wasn't 100% clear until this post. As you mention - he might just be backing up file servers - so no app involved - just files to backup.
-
@Dashrender said in Does block level sync exist?:
@scottalanmiller said in Does block level sync exist?:
@Fredtx said in Does block level sync exist?:
@scottalanmiller Let me clarify. I want to make sure the "good" backups are copied to the offsite storage. So if the building were to catch on fire or something, and the good copies are destroyed. I would want to be able to restore from the offsite storage. In my case, some of the data was missing from the offsite storage that should have been replicated from the local "good" backup. Not sure what happened, and why it was not copied over, but it did not. I figured there would be some kind of sync mechanism that would have caught that ahead of time, which Barracuda said there is no such sync. That is why I reached out to the community.
We understand. And that's important because clearly your sync failed. It's just that it also exposed the fact that the original backups are not application aware (unless there is no application) so something that you should see as a very, very large issue. If you are responsible for the backups, that is. Otherwise, not your monkeys, not your circus.
You're making an assumption that there's an app to backup - which wasn't 100% clear until this post. As you mention - he might just be backing up file servers - so no app involved - just files to backup.
Even a pure file server is normally accessed. "File server" is a form of "database". A very specific form, but surprisingly similar to a document database. It would be super weird, but not actually impossible, to have a file server that holds files but is never accessed. but once you start accessing files, it's an application doing the accessing and we are right back to where we started. File servers tend to have known usage patterns and accepted backup failure modes, but the issue hasn't changed. It just feels that way. No file exists without an application.
