Disasterous SCCM Imaging Mistake at Emory
-
Read the link from bottom to top, but here are the highlights:
Facts as we know them:
A Windows 7 deployment image was accidently sent to all Windows machines, including laptops, desktops, and even servers. This image started with a repartition / reformat set of tasks.
As soon as the accident was discovered, the SCCM server was powered off – however, by that time, the SCCM server itself had been repartitioned and reformatted.
Restoration of servers began immediately but the process took far longer than expected – we have been using consultants to help validate the health of the SCCM servers and that work only completed last night.
So, we were without our preferred methods for deploying images to desktops/laptops all yesterday and relied on older methods – USB + Ghost, LANDesk (we still had our old LANDesk server) + PXE. These methods required a lot of manual work plus our success was uneven with them.
Today we are pausing – briefly this morning – to see if we can now use our preferred method – SCCM. This will allow us to have a one-touch method for restoring desktops/laptops to a production ready state.
-
Wow, just wow.
-
@Dashrender said:
Wow, just wow.
Yeah, people often forget just how dangerous automation can be. In many cases, I want a human involved in certain decisions rather than making a massive amount of automation. When the blast radius of a change is so large, you need to think about maybe not automating that.
-
Ouch, that is pretty rough. Those guys are having a really rough day.
-
This, folks, is why we test queries before adding the query syntax to create dynamic collections.
-
The same dangers apply to chef, puppet, cfEngine, salt stack, etc.
