Why DR Plans are Important and Need to be Tested
-
Today, Wells Fargo experienced a system wide outage affecting virtually all of their customers. This was the second time in a week they had an outage. Just last Friday customers had issues logging onto their online accounts at Wells Fargo.
Their latest tweet confirms that the fire suppression was activated, and reports from Reddit and other sources indicate that they had to perform a system wide shutdown, and are now attempting to bring the systems back up.
To compound the issue, it appears their fail over failed, according to some inside sources. And to top it off, rumor has it that this came up in their risk management meeting and budgeting was turned down to fix the issue.
The consequences of this event are already impacting the company in countless ways:
-
Their new head of technology, Saul Van Beurden, just 2 weeks into his new job, will not be getting much sleep, and may be looking for a new gig soon.
-
Their stock took an immediate hit, and has lost 2.4% of it's value, or about $5.4 Billion, (yes, billion) according to The Street.
-
Many customers who commented on social media have expressed they will be moving to a new banking solution as soon as they "can get their money!"
-
Their will be a class action lawsuit filed in the next few days if one hasn't been filed already. Many customers are complaining about bills not being paid, incurring late fees, and having their credit scores affected due to late payments. One customer had to reschedule a medical procedure for their child because her card could not be processed.
Of course, the list goes on, and this won't be the end of the damage for Wells Fargo. The only good take away from this disaster is to bring to the forefront the absolute need for a well thought out and tested DR plan.
-
-
@pchiodo hahah as if WF knows how to test their preported DR plans.
-
So on a more serious note, DR planning is a extremely critical conversation. If this was properly evaluated and understood by the people making the decisions for WF (they are publicly traded) means that not only have they sustained a substantial amount of damage from this immediately (dollars spent fixing obvious faults), but also their IT (and public) reputation has that they don't take their environment seriously.
That has so many layers of "hold my beer" that it's difficult to comprehend since the added cost to remedy this glaring issue was nominal compared to the damages they've sustained and will continue to sustain from future pending lawsuits, reimbursements etc.
-
@DustinB3403 No doubt - An ounce of prevention is worth a pound of cure. In this case, that cure might be a couple tons.
-
@DustinB3403 From what else we know about Wells Fargo and their management, I'm surprised something like this took so long to happen. Or at least for people outside the company to find out about it.
-
If they really knew that they didn't have working failover and decided to just ignore it, they are going to be in for a world of hurt.
-
I think its been going on for quite a while too which makes this worse. A friend here has mentioned several times, in about 2 years of time, that she could not get into her account for hours at a time. So I don't know if its the same cause every time but something is up with them and been so for a while.
-
It's easy to point fingers but how do you test failover for an entire location?
Obviously they have location failover but how do you test that it actually works without causing downtime if it doesn't work?
And also there are other events that will have the same effect like floods, extended power outages, terror attacks, mass shooters, employee sabotage etc. How do you test for those?
-
@Pete-S said in Why DR Plans are Important and Need to be Tested:
It's easy to point fingers but how do you test failover for an entire location?
Turn off the site
Obviously they have location failover but how do you test that it actually works without causing downtime if it doesn't work?
Plan for it.
-
@Pete-S said in Why DR Plans are Important and Need to be Tested:
Obviously they have location failover but how do you test that it actually works without causing downtime if it doesn't work?
DR testing requires downtime. On Wall St., the big banks do this every six months. They do it on a weekend and they failover the entire bank. They test both directions (fail to DR, fail back). It's normal and expected and zero excuse for not doing it.
And the DR location should be able to pick up quickly. When doing it on Wall St., downtime was minutes. Not days.
-
@Pete-S said in Why DR Plans are Important and Need to be Tested:
And also there are other events that will have the same effect like floods, extended power outages, terror attacks, mass shooters, employee sabotage etc. How do you test for those?
Same, you test a catastrophic failure. Literally someone yanks the network connection physically for the site and lets all automatic failover mechanisms kick in.
-
I seriously do want to be a fly on the wall to see this process of WF unfold.
-
@Pete-S said in Why DR Plans are Important and Need to be Tested:
It's easy to point fingers but how do you test failover for an entire location?
As someone who had to work in this exact space (competing banks), this isn't a normal thing to lack. There should be fingers pointed. It is their responsibility to have this in place, and to test it regularly, and everyone else is doing it. There's no excuse for not bothering.
-
@scottalanmiller said in Why DR Plans are Important and Need to be Tested:
There's no excuse for not bothering.
But it cost so much money to do.
Not compared to having everything come burning down to realize the sprinklers are just for show!
-
One thing, testing DR Plans and Executing DR plans do not equal the same. DR Testing is critical for the following reasons, First to make sure it works, second find any issues, third come up with recommendations and remediation, fourth amend those plans with the recommendations, fifth implement said remediations, sixth do another DR testing. Then rinse and repeat.
But clearly I have seen that many DR Plans while tested worked but when it comes to the moment of truth, things start happening. You can see with with Microsoft and the MFA issue and their fail-over from EU to US and so forth.
-
@dbeato said in Why DR Plans are Important and Need to be Tested:
One thing, testing DR Plans and Executing DR plans do not equal the same. DR Testing is critical for the following reasons, First to make sure it works, second find any issues, third come up with recommendations and remediation, fourth amend those plans with the recommendations, fifth implement said remediations, sixth do another DR testing. Then rinse and repeat.
But clearly I have seen that many DR Plans while tested worked but when it comes to the moment of truth, things start happening. You can see with with Microsoft and the MFA issue and their fail-over from EU to US and so forth.
How often are they actually testing that do you think, though?
-
@scottalanmiller said in Why DR Plans are Important and Need to be Tested:
@dbeato said in Why DR Plans are Important and Need to be Tested:
One thing, testing DR Plans and Executing DR plans do not equal the same. DR Testing is critical for the following reasons, First to make sure it works, second find any issues, third come up with recommendations and remediation, fourth amend those plans with the recommendations, fifth implement said remediations, sixth do another DR testing. Then rinse and repeat.
But clearly I have seen that many DR Plans while tested worked but when it comes to the moment of truth, things start happening. You can see with with Microsoft and the MFA issue and their fail-over from EU to US and so forth.
How often are they actually testing that do you think, though?
Not sure how often. I know we do it every 6 months for bank customers.
-
@dbeato said in Why DR Plans are Important and Need to be Tested:
@scottalanmiller said in Why DR Plans are Important and Need to be Tested:
@dbeato said in Why DR Plans are Important and Need to be Tested:
One thing, testing DR Plans and Executing DR plans do not equal the same. DR Testing is critical for the following reasons, First to make sure it works, second find any issues, third come up with recommendations and remediation, fourth amend those plans with the recommendations, fifth implement said remediations, sixth do another DR testing. Then rinse and repeat.
But clearly I have seen that many DR Plans while tested worked but when it comes to the moment of truth, things start happening. You can see with with Microsoft and the MFA issue and their fail-over from EU to US and so forth.
How often are they actually testing that do you think, though?
Not sure how often. I know we do it every 6 months for bank customers.
Sure. For banks. But someone like MS? I would not be surprised if they never tested.
-
@scottalanmiller said in Why DR Plans are Important and Need to be Tested:
@dbeato said in Why DR Plans are Important and Need to be Tested:
@scottalanmiller said in Why DR Plans are Important and Need to be Tested:
@dbeato said in Why DR Plans are Important and Need to be Tested:
One thing, testing DR Plans and Executing DR plans do not equal the same. DR Testing is critical for the following reasons, First to make sure it works, second find any issues, third come up with recommendations and remediation, fourth amend those plans with the recommendations, fifth implement said remediations, sixth do another DR testing. Then rinse and repeat.
But clearly I have seen that many DR Plans while tested worked but when it comes to the moment of truth, things start happening. You can see with with Microsoft and the MFA issue and their fail-over from EU to US and so forth.
How often are they actually testing that do you think, though?
Not sure how often. I know we do it every 6 months for bank customers.
Sure. For banks. But someone like MS? I would not be surprised if they never tested.
I see
-
This is a good reason to have more than one credit card from more than one bank.
