UREs Strike InnoDB on MySQL
-
Had to deal with this one yesterday. Someone (not a client) came to us because their database server had failed. I/O Errors being thrown all over the place. The number of mistakes on the system were pretty crazy: "newly" installed CentOS 6 (why install intentionally old), RAID 0, consumer SATA drives on a Dell server, bare metal install (no virtualization in a one year old install), production deployed on a twelve year old server that was found abandoned with unknown reliability state, no backups, no snapshots, not even local duplicates of the data, no monitoring tools of any sort, no Dell hardware drivers, no visibility of any sort into the PERC controller or drives, 65% of their capacity was used by the database files (so no physical space to use RAID 1, no space to even take a local copy.)
In other issues, several of the InnoDB databases on the MySQL system were in the 140-300GB range. So the chances of their files being hit with a URE were incredibly high.
What we think happened? UREs struck the most critical database which was 139GB. This results in the following errors..
*** glibc detected *** /usr/libexec/mysqld: corrupted double-linked list: 0x000000000013948567 ***And the more general OS error...
end_request: I/O error, dev sda, sector 898111345Every search that I've done says the same thing... unrecoverable and you need to restore from backup.
Given how ridiculously the system was set up, we recommended that as the data wasn't considered to have any value (the entire server it is running on has a street value around $10) that they should just scrap it all and reconsider their choices.
-
Step one is to remove the drives and clone them with dd or recovery tool to a new drive.
You could probably recover 99.9% of the data - if you want. -
@Pete-S said in UREs Strike InnoDB on MySQL:
Step one is to remove the drives and clone them with dd or recovery tool to a new drive.
You could probably recover 99.9% of the data - if you want.As you can guess from all of their previous issues, they don't want to pay for any recovery, they just want it magically fixed for free. They don't own any storage onto which to clone it, either.
-
@scottalanmiller said in UREs Strike InnoDB on MySQL:
@Pete-S said in UREs Strike InnoDB on MySQL:
Step one is to remove the drives and clone them with dd or recovery tool to a new drive.
You could probably recover 99.9% of the data - if you want.As you can guess from all of their previous issues, they don't want to pay for any recovery, they just want it magically fixed for free. They don't own any storage onto which to clone it, either.
Well, that figures. The only things that surprises me here is actually the size of the databases. Unless you store lots of binary data in the db, 100GB or more is usually a lot of data with millions of rows.
-
@Pete-S said in UREs Strike InnoDB on MySQL:
@scottalanmiller said in UREs Strike InnoDB on MySQL:
@Pete-S said in UREs Strike InnoDB on MySQL:
Step one is to remove the drives and clone them with dd or recovery tool to a new drive.
You could probably recover 99.9% of the data - if you want.As you can guess from all of their previous issues, they don't want to pay for any recovery, they just want it magically fixed for free. They don't own any storage onto which to clone it, either.
Well, that figures. The only things that surprises me here is actually the size of the databases. Unless you store lots of binary data in the db, 100GB or more is usually a lot of data with millions of rows.
Yeah, I was pretty shocked, too. No idea what they do.
-
@scottalanmiller said in UREs Strike InnoDB on MySQL:
@Pete-S said in UREs Strike InnoDB on MySQL:
Step one is to remove the drives and clone them with dd or recovery tool to a new drive.
You could probably recover 99.9% of the data - if you want.As you can guess from all of their previous issues, they don't want to pay for any recovery, they just want it magically fixed for free. They don't own any storage onto which to clone it, either.
Then what is the point of any of it? It appears to have zero value to the business.
-
Why did they call if they don’t want to spend money fixing it?
-
@Dashrender said in UREs Strike InnoDB on MySQL:
Why did they call if they don’t want to spend money fixing it?
They went through a partner asking for favours to take a look at it.
-
@Obsolesce said in UREs Strike InnoDB on MySQL:
@scottalanmiller said in UREs Strike InnoDB on MySQL:
@Pete-S said in UREs Strike InnoDB on MySQL:
Step one is to remove the drives and clone them with dd or recovery tool to a new drive.
You could probably recover 99.9% of the data - if you want.As you can guess from all of their previous issues, they don't want to pay for any recovery, they just want it magically fixed for free. They don't own any storage onto which to clone it, either.
Then what is the point of any of it? It appears to have zero value to the business.
I said that to them.
