@Dashrender said in AWS Catastrophic Data Loss:

@PhlipElder said in AWS Catastrophic Data Loss:

@Dashrender said in AWS Catastrophic Data Loss:

@PhlipElder said in AWS Catastrophic Data Loss:

@wrx7m said in AWS Catastrophic Data Loss:

This was one AZ, right? If so, you need to design your environment to span multiple AZs, if not regions. This is beginner AWS design theory.

A few things come to mind:
1: Just how many folks know how to architect a highly available solution in any cloud?
2: At what cost over and above the indicated method does the HA setup incur?
3: It does not matter where the data is, it should be backed up.

Microsoft's central US DC failure, I think it was last year or early this year, cause a substantial amount of data loss as well. Not sure if any HA setup could have saved them from what I recall.

How many people backup their O365 systems? I am willing to bet VERY few!! yet, if MS were to have the same issue, customers would find themselves in a similar situation.

One (invalid) claim I see from time to time when migrating to the cloud - it saves money because backups are part of the solution... which we can see here is definitely not the case.

Veeam was one of the first ones on the block to back up O365. That's messaging that Microsoft has not made clear but I've seen in the grapevine as far as the customer being responsible to do so.

No. My sh#t on their sh#t means no sh#t if something takes a sh#t.

This is a discussion that @scottalanmiller has been asked about before and I could have sworn that for the most part - he was against the need to specifically backup O365.

The prototype of the current model, IMO, was explained by the Exchange Team at the first Microsoft Exchange Conference that I attended back when.

They dogfooded 400 mailboxes using a distributed model with Exchange 2013 non-production bits. No backups. There were a few surprised looks in the room when that was announced.

The G00g demonstrated quite clearly what happens when the distributed model fails some years ago by losing mailboxes (a client of ours was affected by that - lost their entire business continuity).

I wish I could say I could count on one hand the number of times I've dealt with a, "Help! My server crashed and we don't have any backups". But not.

Oh, and Maersk nearly effed themselves if it weren't for an offline DC in Africa. I think it was Ghana thinking their distributed domain would withstand anything. Until they were encrypted with no DC backups. Anywhere. SMH

@IRJ said in AWS Catastrophic Data Loss:

@PhlipElder said in AWS Catastrophic Data Loss:

For another, as mentioned above, how many folks know how to set up any cloud?

That's why the cloud guys make the :money_bag: :money_bag: :money_bag: :money_bag:

Cloud isn't even that difficult. Most could learn the basic concepts in an hour or two.

Like I said, our tech bubble.

@FATeknollogee said in AWS Catastrophic Data Loss:

@PhlipElder said in AWS Catastrophic Data Loss:

@IRJ said in AWS Catastrophic Data Loss:

For IaaS, using a tool like terraform can help you transition from one platform to another as terraform is compatible with many cloud hosts.

I feel like I'm back in the early 2000s when Microsoft released Small Business Server 2000 then Small Business Server 2003 with the business owner DIY message. We got a lot of calls as a result of that messaging over the years.

Then, there was the mess created by the "IT Consultant" that didn't know their butt from a hole in the ground. We cleaned up a lot of those over the years.

At least in the above cases we could work with some sort of box to get their data on a roll.

Today, that possibility is virtually nil.

That is, the business owner being knowledgeable enough to navigate the spaghetti of cloud services setup to get to a point where they are secure and backed up for one. For another, as mentioned above, how many folks know how to set up any cloud?

Then, toss into the mix the message about speed and agility and we have a deadly mix beyond the SBS messaging and failures in that we're talking orders of magnitude more folks losing their businesses as a result of one big FUBAR.

Ever been on the back of a bike holding a case of beer while the "driver" hit 200+ KPH? I have. Once. And lived to never, ever, ever, trust an arse like that again.

The powerful power of marketing
Cloud = :couple_with_heart: :kissing_face_with_smiling_eyes: :kissing_cat_face_with_closed_eyes:

Yes.

The The Great Firewall of Marketing has prevented most of the negative stuff from surfacing.

@wrx7m said in AWS Catastrophic Data Loss:

@IRJ said in AWS Catastrophic Data Loss:

@PhlipElder said in AWS Catastrophic Data Loss:

@IRJ said in AWS Catastrophic Data Loss:

@PhlipElder said in AWS Catastrophic Data Loss:

@wrx7m said in AWS Catastrophic Data Loss:

This was one AZ, right? If so, you need to design your environment to span multiple AZs, if not regions. This is beginner AWS design theory.

A few things come to mind:
1: Just how many folks know how to architect a highly available solution in any cloud?
2: At what cost over and above the indicated method does the HA setup incur?
3: It does not matter where the data is, it should be backed up.

Very valid points, but that is the responsibility of the customer.
Let's look at IaaS (EC2 instances loses EBS volumes after power outage) vs a cloud hosted service like Office 365.

Services are supposed to have HA built into them. Infrastructure has no HA built into it.

As you mentioned, DATA is ALWAYS responsibility of the customer.

One wee problem: Many "cloud" providers provide absolutely no mechanism to get the data out. Or, in some cases if it can be, it's not in usable form.

That is actually one of the biggest things you look at before chosing a vendor. If you do any cloud training, you will hear about data and data all over again. Being able to export it in a valuable way is essential for on prem or cloud though. If you cant export on prem and use the data in a usuable way into another system, you have the same problem.

Exactly. Are you going to be trapped there because you can't get data migrated/transferred out to another service/platform? A lot of people don't think about this. These people should not be making the decisions to go with any vendor without knowing what questions to ask and how to use those answers to make decisions. Sadly, most don't know until it bites them in the ass.

That's because most outside of our little tech bubble don't know what they want or what can bite them in the arse. I'm thinking of a video I saw a while back. SE Asia I think. Folks like me would wear steal toed boots while the locals knew where to look. Snake would jump out along the path to bite someone. Scared the carp out of me the first time I watched that.

@IRJ said in AWS Catastrophic Data Loss:

For IaaS, using a tool like terraform can help you transition from one platform to another as terraform is compatible with many cloud hosts.

I feel like I'm back in the early 2000s when Microsoft released Small Business Server 2000 then Small Business Server 2003 with the business owner DIY message. We got a lot of calls as a result of that messaging over the years.

Then, there was the mess created by the "IT Consultant" that didn't know their butt from a hole in the ground. We cleaned up a lot of those over the years.

At least in the above cases we could work with some sort of box to get their data on a roll.

Today, that possibility is virtually nil.

That is, the business owner being knowledgeable enough to navigate the spaghetti of cloud services setup to get to a point where they are secure and backed up for one. For another, as mentioned above, how many folks know how to set up any cloud?

Then, toss into the mix the message about speed and agility and we have a deadly mix beyond the SBS messaging and failures in that we're talking orders of magnitude more folks losing their businesses as a result of one big FUBAR.

Ever been on the back of a bike holding a case of beer while the "driver" hit 200+ KPH? I have. Once. And lived to never, ever, ever, trust an arse like that again.

@IRJ said in AWS Catastrophic Data Loss:

@PhlipElder said in AWS Catastrophic Data Loss:

@wrx7m said in AWS Catastrophic Data Loss:

This was one AZ, right? If so, you need to design your environment to span multiple AZs, if not regions. This is beginner AWS design theory.

A few things come to mind:
1: Just how many folks know how to architect a highly available solution in any cloud?
2: At what cost over and above the indicated method does the HA setup incur?
3: It does not matter where the data is, it should be backed up.

Very valid points, but that is the responsibility of the customer.
Let's look at IaaS (EC2 instances loses EBS volumes after power outage) vs a cloud hosted service like Office 365.

Services are supposed to have HA built into them. Infrastructure has no HA built into it.

As you mentioned, DATA is ALWAYS responsibility of the customer.

One wee problem: Many "cloud" providers provide absolutely no mechanism to get the data out. Or, in some cases if it can be, it's not in usable form.

@Dashrender said in AWS Catastrophic Data Loss:

@PhlipElder said in AWS Catastrophic Data Loss:

@wrx7m said in AWS Catastrophic Data Loss:

This was one AZ, right? If so, you need to design your environment to span multiple AZs, if not regions. This is beginner AWS design theory.

A few things come to mind:
1: Just how many folks know how to architect a highly available solution in any cloud?
2: At what cost over and above the indicated method does the HA setup incur?
3: It does not matter where the data is, it should be backed up.

Microsoft's central US DC failure, I think it was last year or early this year, cause a substantial amount of data loss as well. Not sure if any HA setup could have saved them from what I recall.

How many people backup their O365 systems? I am willing to bet VERY few!! yet, if MS were to have the same issue, customers would find themselves in a similar situation.

One (invalid) claim I see from time to time when migrating to the cloud - it saves money because backups are part of the solution... which we can see here is definitely not the case.

Veeam was one of the first ones on the block to back up O365. That's messaging that Microsoft has not made clear but I've seen in the grapevine as far as the customer being responsible to do so.

No. My sh#t on their sh#t means no sh#t if something takes a sh#t.

@wrx7m said in AWS Catastrophic Data Loss:

This was one AZ, right? If so, you need to design your environment to span multiple AZs, if not regions. This is beginner AWS design theory.

A few things come to mind:
1: Just how many folks know how to architect a highly available solution in any cloud?
2: At what cost over and above the indicated method does the HA setup incur?
3: It does not matter where the data is, it should be backed up.

Microsoft's central US DC failure, I think it was last year or early this year, cause a substantial amount of data loss as well. Not sure if any HA setup could have saved them from what I recall.

@Pete-S The generators kicked in but eventually failed. Something else up there.

This one is a bad one: Amazon AWS Outage Shows Data in the Cloud is Not Always Safe.

@DustinB3403 said in Backup strategy for customer data?:

@Dashrender HDD's have stagnated, SSDs are their successor and after that who knows.

There's only so many bits that can be stacked up before they start to fall down.

@travisdh1 said in Backup strategy for customer data?:

@PhlipElder said in Backup strategy for customer data?:

@dafyre said in Backup strategy for customer data?:

@travisdh1 said in Backup strategy for customer data?:

@scottalanmiller said in Backup strategy for customer data?:

@Pete-S said in Backup strategy for customer data?:

Bit error is 1 in 10^19 bits (enterprise HDDs are 1 in 10^15). That's actually 10,000 times better than HDDs. And 30 years of archival properties.

yeah, the tech behind LTO8 is freaking fantastic. And unlike HDD where research is stagnating, tape keeps advancing.

I still haven't seen anything that scales like tape. Just keep adding drives and tapes as needed till you're into silly town: https://spectralogic.com/products/tfinity-exascale/

Quick! I need tapes 39,763 and 40,659!

Oh yeah, does that bring back memories. Feeding the machine to get the combination of tapes needed to recover a set of databases or the like. Ugh. SMH

Sounds like a system that was only got half way. BackupExec?

Yes. BUE. Spent three days on the horn with Symantec Support to no avail. It was a FusterCluck to say the least.

We ended up running a side-by-side migration with a new AD and managed to recover all of their data but it took close to a week.

We moved to StorageCraft's ShadowProtect and disk based backups after that.

This particular site was an anomaly. Their building's HVAC was totally messed up with primary feeds not capped above the false ceiling. So, normally cold air draw via the grate in that ceiling and either hot or cold air pumped into the rooms via the vents. Summer, everyone was warm while the temp above the ceiling tiles was +5C and winter everyone was cold while above the tiles it was +40C.
Partners with offices near the server closet would complain about the noise but we couldn't put a portable A/C unit in there because all of the power panels were full in that part of the building.
We had another catastrophic failure at that site with ShadowProtect allowing us to recover everything to a new server while the firm ran on their secondary. Data loss was limited to 24 files for one partner who was very easy going (fortunately). We were extremely happy when they bought a business condo and moved many years ago. Disk and server failures became a thing of the past. Heat is such a killer though Nehalem and later CPUs brought the server failure count down significantly.

EDIT: Important detail in that last failure: Secondary was getting bad bits as was the backups. We found out that image based backups had a weakness in garbage in garbage out there. It took some hoop jumping to get to that 24 file point including the backups and volume shadow copies on both the secondary and the recovered primary. That was the image's strong point: recovering the snapshots along with everything else.

@dafyre said in Backup strategy for customer data?:

@travisdh1 said in Backup strategy for customer data?:

@scottalanmiller said in Backup strategy for customer data?:

@Pete-S said in Backup strategy for customer data?:

Bit error is 1 in 10^19 bits (enterprise HDDs are 1 in 10^15). That's actually 10,000 times better than HDDs. And 30 years of archival properties.

yeah, the tech behind LTO8 is freaking fantastic. And unlike HDD where research is stagnating, tape keeps advancing.

I still haven't seen anything that scales like tape. Just keep adding drives and tapes as needed till you're into silly town: https://spectralogic.com/products/tfinity-exascale/

Quick! I need tapes 39,763 and 40,659!

Oh yeah, does that bring back memories. Feeding the machine to get the combination of tapes needed to recover a set of databases or the like. Ugh. SMH

@Pete-S said in Backup strategy for customer data?:

@PhlipElder said in Backup strategy for customer data?:

How many tapes in the library?
How many briefcases to take off-premises for rotations?
Where is the brain trust to manage the tapes, their backup windows, and whether the correct tape set is in the drives?
If the tape libraries are elsewhere then the above goes away to some degree (distance comes into play).

A 2U high autoloader will have two magazines with 12 tape slots in each. With LTO-8 tapes that means 720TB of data (2.5:1 compression) in one batch without switching any tapes. 24 tapes will fit in one briefcase so not much of a logistical problem. If you go up to a 3U unit it will hold 40 tapes and I think that might fit in one briefcase as well.

Tapes have barcodes that the autoloader will scan so that's how the machine know which tape is the right one.

If you are going to swap several tapes at once, you can get additional magazines that holds the tape and just swap the entire magazine. For daily incremental backups you can swap one tape at a time - if you have less than 30 TB of data change per day.

You can also monitor that tapes have been replaced so you could set up that as a prerequisite for starting the next daily backup. We'll just have to see how long things take and how much data we need to backup on average before putting procedures in place.

I haven't actually used tape since the late 90s so it will be exiting testing this. For off-line storage and archival storage the specs are just so much better than harddrives. Bit error is 1 in 10^19 bits (enterprise HDDs are 1 in 10^15). That's actually 10,000 times better than HDDs. And 30 years of archival properties.

We used to manage HP based tape libraries and their rotation process. It was a bear to manage.

We have one company we are working with that has a grand total of 124 tapes that they need to work with for one rotation.

GFS, that is Grandfather, Father, and Son, is an important factor in any backup regimen. Air-gap is super critical.

Having software the that manages it all for you is all fine and dandy until the software fails. BTDT and what a freaking mess that was when the servers hit a hard-stop.

Ultimately, it does not matter what medium is used as GFS takes care of one HDD or tape dying due to bit rot (BTDT for both HDD and tape).

The critical element in a DR plan is air-gap. No access. Total loss recovery.

@Pete-S said in Backup strategy for customer data?:

In our case I'm thinking about two options.

OPTION 1
We'll put together a backup server with a large-ish disk array (maybe 100TB or so) connected with SAS to a tape autoloader. Backups go from backup clients to the disk array and when done it's all streamed to tape. The tapes are exchanged and put off-line. Each week a full backup of disks are taken off-site as well.

To keep the networks separated as far as possible we can put the backup server on it's own hardware and it's own network and firewall it off from the production servers. So if production servers or VM hosts are breached the backup server is still intact. If somehow it's also compromised we have to restore everything from tape.

OPTION 2
We put a smaller backup array, say 10TB or so, on each physical VM host. Backups are run on each host from the production VMs to the backup VM with the backup array. Remember our VMs are running on local storage so this will not require any network traffic.

When done, we stream the data from each backup VM to a "tape backup"-server that just basically contains the tape drive (with autoloader) and will write the data to tape. Firewall and tape handling will be the same as option 1. Since the disks with the backups are on each host, several backup servers have to be breached to lose all disk backups.

What do you think?

Inside Job puts # 2 to rest. Let's just say there are plenty of stories about entire setups being wiped starting with the backups then hitting go on 0000 for the SANs.

@Pete-S said in Backup strategy for customer data?:

I did some comparisons of the cost involved for disk versus tape and disregarding the difference between the media types.

Tape is much cheaper per TB (about $11/TB) but you need to offset the cost of the tape drive/autoloader.
Disk on the other hand will require a more expensive server with more drive bays and also requires additional disks for partition data.

In our case I found that at 150 TB of native storage it will break even. If you have more data in backup storage than that, then tape is cheaper.

How many tapes in the library?

How many briefcases to take off-premises for rotations?

Where is the brain trust to manage the tapes, their backup windows, and whether the correct tape set is in the drives?

If the tape libraries are elsewhere then the above goes away to some degree (distance comes into play).

@Pete-S said in Backup strategy for customer data?:

@PhlipElder said in Backup strategy for customer data?:

hosting company's day to day network

With day to day network, do you mean the hosting company's own internal IT, for managing their own company?
Or do you mean the hosting company's management network for managing the hosting infrastructure?

DtDN = Sales, HR, Financing, ETC where folks blindly click on things and get hit by drive-by web sites.

Management would be with PAW (Privileged Access Workstation) and segmented away from the DtDN with absolutely no crossover between them.

@Pete-S said in Backup strategy for customer data?:

@PhlipElder said in Backup strategy for customer data?:

We've worked with a variety of hosting solution providers. Most start with a base of one backup done per 24 hours with a fee to restore if required.

Some have a built-in backup feature that we can then set up for the VMs we have our cloud desktop clients running in. It can be set up to run relatively often. They charge a fee for that one.

Start with once per day.

As far as the "how" what is the underlying virtualization platform?

Our hosting solutions are set up to use Veeam at the host level.

StarWind's Virtual Tape Library (VTL) can be used to augment the backup in another DC with Veeam's Cloud Connect being another option to tie in to get the backup data out of the production DC.

As far as expectations go, we're in the process of setting up a BaaS and DRaaS service based on Veeam. Backups and DR will be multi-site with one goal to be a two to four week no-delete option available.

In our investigations of BaaS/DRaaS providers none were able, or wanted, to answer the, "How do you back up our backup data to protect against failures in your system?" question.

As we are are getting into SaaS and not infrastructure, I think our primary concern are being able to restore the customers data in case something bad happens that's our fault or responsibility - for instance software bugs, hackers, ransomware, multiple hardware failures etc.

We are not as concerned with being able to restore the customers data in case they screw up, as we are if we screw up. That said, if we can without to much investment, we might be able to add something here. Have to think about that one. In either case we will provide some way for the customer to export and backup their data.

For now we run on xen (xcp-ng). The goal is to be able to restore the infrastructure with automation, so I don't expect us to really need a lot of host based backups. We have a lot more testing to do on this.

From what I can gather right now, I think we will backup to disk storage on-prem. Then from there we will go to tape. Tape will be moved off site once a week. We will do incremental backups to the cloud or another site so we can restore completely using off-site tape and the incremental backups.

This will allow us to restore from on-prem disk in most cases. If we are hacked or infected we can restore from on-site tape. In case of a fire or something we can restore from off site tape and incremental backups.

There are some keys to providing a customer facing solution:

• Customer facing network(s) are not in any way connected to the hosting company's day to day network (DtDN)
• Privileged Access Workstation structures are in place to keep things separate
• Backups are air-gapped in some way to protect against catastrophic failure or encryption event
• Customer resources are on separate equipment from DtDN

Ultimately, the entire solution set for DtDN, Support, and Customer Facing networks should be segmented completely from each other with significant protections in place to keep them that way.

• iNSYNQ
• 
• 
• Wolters Kluwer/CCH
• Maersk
• PCM
• WiPro
• Hosting company (UK 123 something?) lost everything due to backups being wiped
• Secure mail hosting company lost everything when perp took everything out right through the backups
• ETC

We've worked with a variety of hosting solution providers. Most start with a base of one backup done per 24 hours with a fee to restore if required.

Some have a built-in backup feature that we can then set up for the VMs we have our cloud desktop clients running in. It can be set up to run relatively often. They charge a fee for that one.

Start with once per day.

As far as the "how" what is the underlying virtualization platform?

Our hosting solutions are set up to use Veeam at the host level.

StarWind's Virtual Tape Library (VTL) can be used to augment the backup in another DC with Veeam's Cloud Connect being another option to tie in to get the backup data out of the production DC.

As far as expectations go, we're in the process of setting up a BaaS and DRaaS service based on Veeam. Backups and DR will be multi-site with one goal to be a two to four week no-delete option available.

In our investigations of BaaS/DRaaS providers none were able, or wanted, to answer the, "How do you back up our backup data to protect against failures in your system?" question.

https://www.zdnet.com/article/ransomware-hits-hundreds-of-dentist-offices-in-the-us/
https://krebsonsecurity.com/2019/08/ransomware-bites-dental-data-backup-firm/