Does block level sync exist?
-
@Fredtx said in Does block level sync exist?:
@scottalanmiller said in Does block level sync exist?:
But, like I said in a meeting recently, all backup software is for the cases where we failed to backup properly. It's all a fallback for failures in data design. In enterprise systems, you don't need backup software. Almost everyone has it as a second option, but you should never need it or rely on it. Or if you use it, it's as an interface to other systems only.
Yea, and most backup vendors will not take the fault if there was some kind of data loss. At least that's what I've seen from the past years of being in IT. They'll say something like "It was corrupted" or give another reason why there was not a copy of the files/folders that were loss.
Right. Well, to their credit, it's really ITs responsibility to understand that only IT can do reliable backups and what we are asking of backup vendors is literally impossible for them to do. So that there is occasional corruption is a guarantee. While they have to be careful not to promise the impossible, it's also our responsibility not to demand it or act like we could get it if we tried hard enough.
Backup software, by definitely, is best effort. Because it has no way to guarantee that the data is healthy when they back it up. The only way to do that is application awareness, which is out of the backup vendors' hands by and large, and even when they have it, it still means "trusting the application".
-
I do backups for financial systems, for example. And we always explain "well, we can quiesce the database and ensure that database is not corrupt, but we can never know if the database has been given quiesced application data because only the developers can tell us that".... and 99% of the time, the devs don't even know themselves and never accounted for needing to make the application safe to back up at all!
When I have my application developer hat on, we make our applications to have their own backup tools, because it's literally the only safe way to know you are getting good backups of a live system. The only. Full stop. If our customers were to buy backup software, it would be so goofy... because it would be extra effort to be less safe.
-
@scottalanmiller said in Does block level sync exist?:
I do backups for financial systems, for example. And we always explain "well, we can quiesce the database and ensure that database is not corrupt, but we can never know if the database has been given quiesced application data because only the developers can tell us that".... and 99% of the time, the devs don't even know themselves and never accounted for needing to make the application safe to back up at all!
I agree. If the application isn't designed for backups in a specific manner then the only safe bet is to shut it down, snapshot the data for backup and power it up again.
The same operations needed to shutdown is a superset of the operations needed to put the database and application data in a safe known state. And most applications are designed to shutdown and startup safely.
It may be clumsy but with VMs the service interruption will usually be short. Maybe 30 seconds or so.
-
@Pete-S said in Does block level sync exist?:
@scottalanmiller said in Does block level sync exist?:
I do backups for financial systems, for example. And we always explain "well, we can quiesce the database and ensure that database is not corrupt, but we can never know if the database has been given quiesced application data because only the developers can tell us that".... and 99% of the time, the devs don't even know themselves and never accounted for needing to make the application safe to back up at all!
I agree. If the application isn't designed for backups in a specific manner then the only safe bet is to shut it down, snapshot the data for backup and power it up again.
The same operations needed to shutdown is a superset of the operations needed to put the database and application data in a safe known state. And most applications are designed to shutdown and startup safely.
It may be clumsy but with VMs the service interruption will usually be short. Maybe 30 seconds or so.
Yeah, it's amazing how many people in IT think that they can buy their way out of this problem. You gotta either be application aware, or stop the backup being of an application (and make it just the storage) by taking the application down. It's super weird because on a desktop we'd likely understand the mechanics but once it is a server, people tend to think it has become a magic black box and they forget the basics that they know from their home computer PC use.
-
Barracuda backup is application aware for things like SQL and AD. Which takes a copy of the files, and puts it on the backup appliance (linux server)
My main thing was the offsite replicated storage did not have some folders/files that were on the backup appliance (linux server), and thus shows it did not work as it's supposed to. My main question was if there was any kind of sync that would be able to report any discrepancies between the Local appliance (linux server) and the remote offsite storage (also linux server).
So with the Backup 3-2-1 rule (3 copies of data). I only had 2 copies of some folders/files. But like Scott said, these products are "best effort" and it's really up to IT to make sure all sensitive data is backed up properly.
-
@Fredtx said in Does block level sync exist?:
Barracuda backup is application aware for things like SQL and AD. Which takes a copy of the files, and puts it on the backup appliance (linux server)
Right, which is just a fancy way of saying it uses VSS. Everything does that, that's not considered application aware, because absolutely everything has that level of awareness - the agent that has the awareness is part of the OS. Barracuda isn't aware of any third party applications, including those that run on top of MS SQL.
-
@Fredtx said in Does block level sync exist?:
Which takes a copy of the files, and puts it on the backup appliance (linux server)
You get copies with or without application awareness. That's the danger and the sales trick of backups. Looks good, but you have to know what is happening under the hood.
-
@Fredtx said in Does block level sync exist?:
But like Scott said, these products are "best effort" and it's really up to IT to make sure all sensitive data is backed up properly.
Well yes, but I mean at the backup level. Baraccuda cannot be fully application aware, ONLY IT can do that. Any good sync can ensure that it synced properly, that does not require IT to oversee it. The issue you are having is that someone bought a bad product that doesn't work and doesn't have support. If you used Rsync, for example, you'd have that functionality. Robocopy does, too.
So yes, it is up to IT to make sure things are backed up. But that's the application awareness part.
-
@Fredtx said in Does block level sync exist?:
My main question was if there was any kind of sync that would be able to report any discrepancies between the Local appliance (linux server) and the remote offsite storage (also linux server).
Of course, any can. It's trivial. Everything does this. Think about products like NextCloud, DropBox, Rsync, RoboCopy, and so forth. They all do this. None do block based, because it's hugely impractical. They all do file based to make this quick and efficient. But you can do a comparison either way, one is just easy (CPU-wise) and one is hard and takes a long time.
You can do this yourself with a simple script. Do a block based transfer from one machine to another. MD5 the resultant file on either location. It's easy to prove that you can do it with nothing more than the provided operating system components.
-
@scottalanmiller said in Does block level sync exist?:
Right, which is just a fancy way of saying it uses VSS. Everything does that, that's not considered application aware, because absolutely everything has that level of awareness - the agent that has the awareness is part of the OS. Barracuda isn't aware of any third party applications, including those that run on top of MS SQL.
Yea, I'm aware it uses VSS. I thought that's what you were referring to when talking about application awareness. But looks like you are referring to something else that I have a lack of knowledge or understanding on.
-
@scottalanmiller said in Does block level sync exist?:
The issue you are having is that someone bought a bad product that doesn't work and doesn't have support. If you used Rsync, for example, you'd have that functionality. Robocopy does, too.
I'm definitely seeing how horrible this product is. At first, I was like hey this is pretty good. Now that I'm getting a more understanding of it, and working with their support I see it sucks. Heck, even today the cloud console that is used to check the status of all the backups was unresponsive. I reached out to support who was aware of the issue, and I asked for a technical answer once they resolve it, and here's what I got from the tech.
I did also reach out to the team for a state, the public statement and I was given the following statement, but I do not believe we have a concrete reason as of yet on why this occurred.
We were experiencing an internal issue which has now been resolved and we are taking steps to prevent issues like this from occurring moving forward.
Barracuda Support
I may look at something else such as Veeam Backup And Replication, but I would need to do further research to see how their backup product works in the back end.
-
@Fredtx said in Does block level sync exist?:
@scottalanmiller said in Does block level sync exist?:
Right, which is just a fancy way of saying it uses VSS. Everything does that, that's not considered application aware, because absolutely everything has that level of awareness - the agent that has the awareness is part of the OS. Barracuda isn't aware of any third party applications, including those that run on top of MS SQL.
Yea, I'm aware it uses VSS. I thought that's what you were referring to when talking about application awareness. But looks like you are referring to something else that I have a lack of knowledge or understanding on.
It's not that complicated. Imagine you are running your own desktop in a VM and you want to take a backup. What about the files you are working on and haven't saved yet? They only exist in RAM and not on any disk. So any backup that backups just your files or blocks on the disk will never be complete.
It's the same on a server. You don't know where the data is that you are trying to backup. Only the application developers knows how it works and where the data is.
VSS is a set of Windows components that communicate with applications so that the OS can tell the application when it need to prepare for a snapshot of the data by writing files to disk etc. But that only works IF the developers actually use the VSS components in their application and that is not always the case. But if they do, the backup should be good concerning that particular application. However there are many things running on a typical server.
-
@Pete-S Good description.
-
@Fredtx said in Does block level sync exist?:
@scottalanmiller said in Does block level sync exist?:
Right, which is just a fancy way of saying it uses VSS. Everything does that, that's not considered application aware, because absolutely everything has that level of awareness - the agent that has the awareness is part of the OS. Barracuda isn't aware of any third party applications, including those that run on top of MS SQL.
Yea, I'm aware it uses VSS. I thought that's what you were referring to when talking about application awareness. But looks like you are referring to something else that I have a lack of knowledge or understanding on.
VSS is application aware, just one only one or two applications that we rarely care very much about. It's not aware of YOUR applications, but it knows about SOME applications. In a world of hundreds of thousands of applications, it probably knows a few dozen and that's about it. Nearly all are MS applications like AD and Exchange. That's about it.
In all these years, I've never once run into any application outside of the stock MS apps that any vendor took the effort to be application aware of natively out of the box for a general purpose backup software. It's just impossible to address everyone (let alone anyone's) needs, so there's no point in pretending.
-
@Fredtx said in Does block level sync exist?:
I may look at something else such as Veeam Backup And Replication, but I would need to do further research to see how their backup product works in the back end.
That won't change the awareness issue. Do they have a sync option that meets your needs?
-
@scottalanmiller said in Does block level sync exist?:
That won't change the awareness issue. Do they have a sync option that meets your needs?
What I'm looking for is adhering to the Backup 3-2-1 best practice. The 3-2-1 strategy that Barracuda offers is a backup is copied to a local backup appliance, which is then deduplicated to save storage space on the appliance. Then, the data from the appliance is compressed and replicated to an offsite storage, which can be another appliance (our setup in my division) or their cloud storage, which I believe is in AWS.
-
@Fredtx said in Does block level sync exist?:
What I'm looking for is adhering to the Backup 3-2-1 best practice.
Well, the FIRST step in 3-2-1 is to get a reliable backup to do the 3-2-1 with. 3-2-1 itself is a really trivial part of the overall picture. That's just talking about where and how many copies you have. But it only matters if the copies you are making are good ones. You are worrying that the copies of the copies don't change, but not worrying if the original copy is any good.
-
@Fredtx said in Does block level sync exist?:
The 3-2-1 strategy that Barracuda offers is a backup is copied to a local backup appliance, which is then deduplicated to save storage space on the appliance. Then, the data from the appliance is compressed and replicated to an offsite storage, which can be another appliance (our setup in my division) or their cloud storage, which I believe is in AWS.
Just for reference, we do this using Duplicati. Rolling it out right now after testing. We've used it for years, but now making it our primary tool. It doesn't do the EXACT same process you are describing, but very similar. But single tool doing the 3-2-1 in an essentially similar way.
We are doing one local copy, one copy to NAS, a snapshot of the NAS, and an offsite to S2 or Wasabi.
-
But we are aware that Duplicati, like Veeam or Baraccuda, isn't application aware (of our applications) and that we have to accommodate for that and take a backup using application aware processes locally, then let Duplicati take a backup of THAT backup.
-
@scottalanmiller said in Does block level sync exist?:
Duplicati
What is Duplicati? It sound like something you install locally on each server that will send backups to NAS/cloud/wherever. Is that correct?
