Why DR Plans are Important and Need to be Tested
-
So on a more serious note, DR planning is a extremely critical conversation. If this was properly evaluated and understood by the people making the decisions for WF (they are publicly traded) means that not only have they sustained a substantial amount of damage from this immediately (dollars spent fixing obvious faults), but also their IT (and public) reputation has that they don't take their environment seriously.
That has so many layers of "hold my beer" that it's difficult to comprehend since the added cost to remedy this glaring issue was nominal compared to the damages they've sustained and will continue to sustain from future pending lawsuits, reimbursements etc.
-
@DustinB3403 No doubt - An ounce of prevention is worth a pound of cure. In this case, that cure might be a couple tons.
-
@DustinB3403 From what else we know about Wells Fargo and their management, I'm surprised something like this took so long to happen. Or at least for people outside the company to find out about it.
-
If they really knew that they didn't have working failover and decided to just ignore it, they are going to be in for a world of hurt.
-
I think its been going on for quite a while too which makes this worse. A friend here has mentioned several times, in about 2 years of time, that she could not get into her account for hours at a time. So I don't know if its the same cause every time but something is up with them and been so for a while.
-
It's easy to point fingers but how do you test failover for an entire location?
Obviously they have location failover but how do you test that it actually works without causing downtime if it doesn't work?
And also there are other events that will have the same effect like floods, extended power outages, terror attacks, mass shooters, employee sabotage etc. How do you test for those?
-
@Pete-S said in Why DR Plans are Important and Need to be Tested:
It's easy to point fingers but how do you test failover for an entire location?
Turn off the site
Obviously they have location failover but how do you test that it actually works without causing downtime if it doesn't work?
Plan for it.
-
@Pete-S said in Why DR Plans are Important and Need to be Tested:
Obviously they have location failover but how do you test that it actually works without causing downtime if it doesn't work?
DR testing requires downtime. On Wall St., the big banks do this every six months. They do it on a weekend and they failover the entire bank. They test both directions (fail to DR, fail back). It's normal and expected and zero excuse for not doing it.
And the DR location should be able to pick up quickly. When doing it on Wall St., downtime was minutes. Not days.
-
@Pete-S said in Why DR Plans are Important and Need to be Tested:
And also there are other events that will have the same effect like floods, extended power outages, terror attacks, mass shooters, employee sabotage etc. How do you test for those?
Same, you test a catastrophic failure. Literally someone yanks the network connection physically for the site and lets all automatic failover mechanisms kick in.
-
I seriously do want to be a fly on the wall to see this process of WF unfold.
-
@Pete-S said in Why DR Plans are Important and Need to be Tested:
It's easy to point fingers but how do you test failover for an entire location?
As someone who had to work in this exact space (competing banks), this isn't a normal thing to lack. There should be fingers pointed. It is their responsibility to have this in place, and to test it regularly, and everyone else is doing it. There's no excuse for not bothering.
-
@scottalanmiller said in Why DR Plans are Important and Need to be Tested:
There's no excuse for not bothering.
But it cost so much money to do.
Not compared to having everything come burning down to realize the sprinklers are just for show!
-
One thing, testing DR Plans and Executing DR plans do not equal the same. DR Testing is critical for the following reasons, First to make sure it works, second find any issues, third come up with recommendations and remediation, fourth amend those plans with the recommendations, fifth implement said remediations, sixth do another DR testing. Then rinse and repeat.
But clearly I have seen that many DR Plans while tested worked but when it comes to the moment of truth, things start happening. You can see with with Microsoft and the MFA issue and their fail-over from EU to US and so forth.
-
@dbeato said in Why DR Plans are Important and Need to be Tested:
One thing, testing DR Plans and Executing DR plans do not equal the same. DR Testing is critical for the following reasons, First to make sure it works, second find any issues, third come up with recommendations and remediation, fourth amend those plans with the recommendations, fifth implement said remediations, sixth do another DR testing. Then rinse and repeat.
But clearly I have seen that many DR Plans while tested worked but when it comes to the moment of truth, things start happening. You can see with with Microsoft and the MFA issue and their fail-over from EU to US and so forth.
How often are they actually testing that do you think, though?
-
@scottalanmiller said in Why DR Plans are Important and Need to be Tested:
@dbeato said in Why DR Plans are Important and Need to be Tested:
One thing, testing DR Plans and Executing DR plans do not equal the same. DR Testing is critical for the following reasons, First to make sure it works, second find any issues, third come up with recommendations and remediation, fourth amend those plans with the recommendations, fifth implement said remediations, sixth do another DR testing. Then rinse and repeat.
But clearly I have seen that many DR Plans while tested worked but when it comes to the moment of truth, things start happening. You can see with with Microsoft and the MFA issue and their fail-over from EU to US and so forth.
How often are they actually testing that do you think, though?
Not sure how often. I know we do it every 6 months for bank customers.
-
@dbeato said in Why DR Plans are Important and Need to be Tested:
@scottalanmiller said in Why DR Plans are Important and Need to be Tested:
@dbeato said in Why DR Plans are Important and Need to be Tested:
One thing, testing DR Plans and Executing DR plans do not equal the same. DR Testing is critical for the following reasons, First to make sure it works, second find any issues, third come up with recommendations and remediation, fourth amend those plans with the recommendations, fifth implement said remediations, sixth do another DR testing. Then rinse and repeat.
But clearly I have seen that many DR Plans while tested worked but when it comes to the moment of truth, things start happening. You can see with with Microsoft and the MFA issue and their fail-over from EU to US and so forth.
How often are they actually testing that do you think, though?
Not sure how often. I know we do it every 6 months for bank customers.
Sure. For banks. But someone like MS? I would not be surprised if they never tested.
-
@scottalanmiller said in Why DR Plans are Important and Need to be Tested:
@dbeato said in Why DR Plans are Important and Need to be Tested:
@scottalanmiller said in Why DR Plans are Important and Need to be Tested:
@dbeato said in Why DR Plans are Important and Need to be Tested:
One thing, testing DR Plans and Executing DR plans do not equal the same. DR Testing is critical for the following reasons, First to make sure it works, second find any issues, third come up with recommendations and remediation, fourth amend those plans with the recommendations, fifth implement said remediations, sixth do another DR testing. Then rinse and repeat.
But clearly I have seen that many DR Plans while tested worked but when it comes to the moment of truth, things start happening. You can see with with Microsoft and the MFA issue and their fail-over from EU to US and so forth.
How often are they actually testing that do you think, though?
Not sure how often. I know we do it every 6 months for bank customers.
Sure. For banks. But someone like MS? I would not be surprised if they never tested.
I see
-
This is a good reason to have more than one credit card from more than one bank.
-
@wrx7m said in Why DR Plans are Important and Need to be Tested:
This is a good reason to have more than one credit card from more than one bank.
This isn't just a credit issue though. It's people literally unable to access their own money, not money on credit. More so if people have the money in their checking/savings account to pay their bills why should they have to pay their bills on credit?!
The bank had better pay off any interest if their customers had to use a credit card to pay for anything during this time to get things back to Zero.
