Shut the system down pull the failed drive and the hot spare turn it back on. You should be up and running on a degraded RAID array
We are terrified to turn the system off at this point.
I am guessing no backups? Is the system available at all?
System is currently available. In theory there are backups, but no capacity to test them and they are across the country. This isn't a managed customer, so we have no documentation, no visibility, no access. We just have to trust that they say that there are backups. But they confused their UPS with their NAS calling both their "backup" so we have literally no clue what they do and do not have.
I'd burn a couple of billable hours at least doing some investigation into their things using Mesh Commander. That way you have a better idea of where they are on the we're screwed o'meter.
That'd be great. But their CIO doesn't like us doing anything.
Then why are you still helping them?
At this point we are not.
We have made our recommendations and it's up to them.
If they choose to disregard our advise, it's at their own peril and we will watch it burn. That's not of course what we want to happen, but the manager has chosen to be the CIO even though she is not competent to do so.