Burned by Eschewing Best Practices
-
https://mangolassi.it/topic/12378/gitlab-in-massive-data-loss-with-no-working-backups/
Doesn't get much more burned than this one.
-
This is going to go downhill quickly. No response yet.
https://community.spiceworks.com/topic/1962353-storage-spaces-and-iscsis
-
https://community.spiceworks.com/topic/1964263-freenas-how-to-resilver-now
BPs Missed:
- Ran RAID 5
- Use a NAS OS
And then additionally got hit with mistakes caused by the additional complexity of using software RAID instead of hardware RAID. Nothing wrong with software RAID per se but it requires experience and you take risks that you will be able to manage it correctly. That failed here.
-
Corrupted VHDX SQL File Server
Practices Eschewed:
Taking a live copy of the VHDX (not powering off the VM)
Not using a proper backup approach for both file level and block level
Not fact checking what the vendor sold and installed. -
RAID 5 in production.
https://community.spiceworks.com/topic/1965431-need-data-from-raid-5-without-os
Important Note: OP was stuck with it and had nothing to do with the disaster. They are just the ones reporting it.
-
Ouch this one is bad......
First, whoever set the server up:
- RAID 5
- RAID with only three disks
- Really cheap components
- No backups
- No virtualization
- Database on parity
- No installation media
Second, once the RAID failed and they called in the "tech" and the disaster continued:
- Didn't immediately take a backup
- Didn't properly replace the failed disk
- Ran filesystem checks and other intensive disk activity while the RAID was degraded
- Rebooted a degraded system
- Doesn't know the basics of the technologies he's there to work on
- Misusing really basic terms, like calling things VHDs when there is no virtualization (heard a term, repeated it)
- Didn't call someone in to help when he say stuff he didn't know about, just started mucking about.
https://community.spiceworks.com/topic/1966285-dell-perc-h310-and-disk-failures
Adrian let him have it, and he earned it. He took a bad situation and drove it into the ground.
-
Then he tries to defend things by saying he has more experience than Dell has been around (in years) and that lots of SW people think it is okay to run CHKDSK on a degraded array so he's not convinced it is a bad idea. OMG. That kind of moves things into the negligence category.
-
https://community.spiceworks.com/topic/1968019-dell-md3400-das-with-2-servers
This one was close but worked out. It's just a lab and a great learning opportunity. Would have been a big burn in production. But as it is, tons of great learning!
-
@RojoLoco said in What Are You Doing Right Now:
@Dashrender said in What Are You Doing Right Now:
@RojoLoco said in What Are You Doing Right Now:
@Dashrender said in What Are You Doing Right Now:
@RojoLoco said in What Are You Doing Right Now:
Trying to remember (or find) the PSK for my L2TP tunnel so I don't have to reset it a dozen devices... because 1 a-hole decided to factory reset his phone and then whine about "I haven't gotten my work email in over a week, wtf?" Well, you erased the f@cking connection, so of course you can't get work email....
That's pretty extreme security for email... TLS not good enough?
Our email server is not exposed to the outside, and now Crapple doesn't like the PPTP tunnel we already had, so I had to make an L2TP tunnel. It's for all purpose remote access, email RDP, everything. And now 1 guy ruins it for everyone (mostly me) by forcing the whole thing to get reset, and he can't understand why it isn't "pa55word". Yay, I love being forced to manually reconfigure every ipad and iphone here... again.
I take it you didn't record the PSK somewhere you can find? Sadly I've done that before too. it sucks
I've looked everywhere I might have written it down, but I was "smart" and didn't write it down...
I just remembered this thing the boss came up with, gonna try that (typing exercise as a password)
Burned by not keeping clean and concise records, in a known location. Also for being "smart". . .
-
Built an IPOD because he sales guy told him too. Then was told to split his RAID array in two, buy Starwind and use Starwind to combine the two RAID arrays (on the same controller) back again.
-
@scottalanmiller said in Burned by Eschewing Best Practices:
Built an IPOD because he sales guy told him too. Then was told to split his RAID array in two, buy Starwind and use Starwind to combine the two RAID arrays (on the same controller) back again.
None of that makes sense.
-
@scottalanmiller said in Burned by Eschewing Best Practices:
Built an IPOD because he sales guy told him too. Then was told to split his RAID array in two, buy Starwind and use Starwind to combine the two RAID arrays (on the same controller) back again.
I need a minion saying "Whaaa?"
-
And... he wanted to put his backups on it, too.
-
I can't see how that is at all logical. How did it come across to him that any of it made any sense at all?
-
@DustinB3403 said in Burned by Eschewing Best Practices:
I can't see how that is at all logical. How did it come across to him that any of it made any sense at all?
I think "Because Dell said so."
-
@scottalanmiller said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
I can't see how that is at all logical. How did it come across to him that any of it made any sense at all?
I think "Because Dell said so."
Oh well in that case, Dell also told me to cut off my left foot, because left feet stink more than right feet. Better do as they tell me...
-
@DustinB3403 said in Burned by Eschewing Best Practices:
@scottalanmiller said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
I can't see how that is at all logical. How did it come across to him that any of it made any sense at all?
I think "Because Dell said so."
Oh well in that case, Dell also told me to cut off my left foot, because left feet stink more than right feet. Better do as they tell me...
You have more than one left foot and right foot?
-
Backups from another site. That part ended up being okay.
-
@JaredBusch said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@scottalanmiller said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
I can't see how that is at all logical. How did it come across to him that any of it made any sense at all?
I think "Because Dell said so."
Oh well in that case, Dell also told me to cut off my left foot, because left feet stink more than right feet. Better do as they tell me...
You have more than one left foot and right foot?
You'd think with the arguments some of these sales guys make...
-
@scottalanmiller said in Burned by Eschewing Best Practices:
Built an IPOD because he sales guy told him too. Then was told to split his RAID array in two, buy Starwind and use Starwind to combine the two RAID arrays (on the same controller) back again.
