Autoupdates Killed My Windows Server 2008 R2


  • Service Provider

    I've had auto updates hose stuff before, but this one takes the cake. I took on a client with a Dell T310 server with Server 2008 R2 Standard on it. It's physical, not virtual. :(

    I'll spare the play by play, but basically 3 updates installed and after that the server was stuck in a loop where it would try to roll back the updates for 3 hours, and then reboot only to repeat the reboot loop. I restored from backup and now if I log in to the desktop, it pretty much locks up the server. If I don't log in to the desktop on the server, all the services run fine.

    Any ideas on what to try? I can get in to safe mode, and I've tried a selective boot where 3rd party services were disabled.

    Going forward I'd like to virtualize the server. Does anyone know if you can do a P2V migration without logging in to the desktop? My other thought was to restore from backup to a virtual machine. Has anyone tried that before with Veeam Endpoint? It would be simple enough to build a new domain controller, but I'm affraid when I get to the last step of dcpromoing the old server out, I won't be able since I can't log in to it. I could hack it out of AD with ADSIedit, but I'd rather avoid that if I could.


  • Service Provider

    Updates that hosed it:
    0_1492194760842_server-error-before-reboot.png

    After those updates tried to install, this is the screen it would stay on for 3 hours before rebooting:
    0_1492194805402_serverStuckAutoUpdate.png



  • Can you get to a Safe mode command prompt and manually uninstall each update? ... or does it freeze on you then too?


  • Service Provider

    @dafyre I tried that. KB890830 and KB4015549 would uninstall. KB4014981 failed to uninstall with this error:
    0_1492195130450_netFrameWorkFailed.png

    It was at that point that I decided to just restore from last night's OS backup.


  • Service Provider

    @dafyre So far I haven't had any problems logging in to safe mode.


  • Service Provider

    The Windows update log had a ton of these errors:

    2017-04-13	14:24:10:022	 428	1614	Agent	WARNING: Failed to evaluate Installed rule, updateId = {6CFCF2A8-52A0-4DDC-AD90-71A7736E83CE}.200, hr = 80080005
    2017-04-13	14:26:10:109	 428	1614	Agent	WARNING: Failed to evaluate Installable rule, updateId = {6CFCF2A8-52A0-4DDC-AD90-71A7736E83CE}.200, hr = 80080005
    2017-04-13	14:30:10:787	 428	1614	Agent	WARNING: Failed to evaluate Installed rule, updateId = {0C91D9FF-FEE6-421C-ACCD-15582919F140}.200, hr = 80080005
    2017-04-13	14:32:10:846	 428	1614	Agent	WARNING: Failed to evaluate Installable rule, updateId = {0C91D9FF-FEE6-421C-ACCD-15582919F140}.200, hr = 80080005
    2017-04-13	14:36:11:259	 428	1614	Agent	WARNING: Failed to evaluate Installed rule, updateId = {89185EE9-622B-4D77-9DF7-FE3F2E2027EE}.200, hr = 80080005
    2017-04-13	14:38:11:326	 428	1614	Agent	WARNING: Failed to evaluate Installable rule, updateId = {89185EE9-622B-4D77-9DF7-FE3F2E2027EE}.200, hr = 80080005
    2017-04-13	14:42:11:656	 428	1614	Agent	WARNING: Failed to evaluate Installed rule, updateId = {728F10D0-EFA2-494E-B116-FFCACBCF094C}.200, hr = 80080005
    2017-04-13	14:44:11:715	 428	1614	Agent	WARNING: Failed to evaluate Installable rule, updateId = {728F10D0-EFA2-494E-B116-FFCACBCF094C}.200, hr = 80080005
    2017-04-13	14:48:13:563	 428	1614	Agent	WARNING: Failed to evaluate Installed rule, updateId = {67BCDD15-BA51-4465-BED2-6202AD4A9D34}.201, hr = 80080005
    2017-04-13	14:50:13:621	 428	1614	Agent	WARNING: Failed to evaluate Installable rule, updateId = {67BCDD15-BA51-4465-BED2-6202AD4A9D34}.201, hr = 80080005
    


  • What happens if you disconnect a network cable and then log in?


  • Service Provider

    @dafyre said in autoupdates killed my server:

    What happens if you disconnect a network cable and then log in?

    I haven't tried that. I read about some people having that issue with Windows 7 machines, but didn't think it would apply to the server since it always has a connection to itself. I can give it a rip next time I'm on site.



  • Only other thing I can think of would be to restore it to a hypervisor (and leave it disconnected) to see if that has any bearing on it... and then, you could restore older backups until you find one that works.


  • Service Provider

    @dafyre said in autoupdates killed my server:

    Only other thing I can think of would be to restore it to a hypervisor (and leave it disconnected) to see if that has any bearing on it... and then, you could restore older backups until you find one that works.

    In hind sight I wish I had brought a copy of the backup with me so I could try to restore to something on my bench. That's a good idea.



  • @dafyre said in autoupdates killed my server:

    Only other thing I can think of would be to restore it to a hypervisor (and leave it disconnected) to see if that has any bearing on it... and then, you could restore older backups until you find one that works.

    But with the old one running you will get conflicts. Power one off, power one on.
    Delete the NIC so it doesnt preserve the MAC after the convert. new IP and rename it, then you can have both running.


  • Service Provider

    @Mike-Davis said in autoupdates killed my server:

    I've had auto updates hose stuff before, but this one takes the cake. I took on a client with a Dell T310 server with Server 2008 R2 Standard on it. It's physical, not virtual. :(

    You can sense the impending disaster right away.



  • @scottalanmiller said in Autoupdates Killed My Windows Server 2008 R2:

    @Mike-Davis said in autoupdates killed my server:

    I've had auto updates hose stuff before, but this one takes the cake. I took on a client with a Dell T310 server with Server 2008 R2 Standard on it. It's physical, not virtual. :(

    You can sense the impending disaster right away.

    That is why I always like to check backups before rebooting something that is unkown or not been rebooted in a while.


  • Service Provider

    @Texkonc said in autoupdates killed my server:

    That is why I always like to check backups before rebooting something that is unkown or not been rebooted in a while.

    That's the odd thing. I do have backups and I was able to restore to a point 6 hours before the automatic updates went on. It isn't stuck in a boot loop, but I can't get to the desktop. It's possible that I haven't logged in to the desktop since the last auto update went on. I could try rolling back even further, but it would be better to do that on a test server.



  • @Texkonc said in Autoupdates Killed My Windows Server 2008 R2:

    @dafyre said in autoupdates killed my server:

    Only other thing I can think of would be to restore it to a hypervisor (and leave it disconnected) to see if that has any bearing on it... and then, you could restore older backups until you find one that works.

    But with the old one running you will get conflicts. Power one off, power one on.
    Delete the NIC so it doesnt preserve the MAC after the convert. new IP and rename it, then you can have both running.

    That's why I said leave it disconnected. So the VM doesn't try to take over... at least not right away! :-D



  • Possibly failed update from before that are just sticking around and that is why the last company did not reboot the server? Maybe it is spiraled worse since you took over, but it was left over from the last guy.



  • @Mike-Davis does terminal server connection works?? what about windows offline updater http://download.wsusoffline.net/



  • Have you checked the CBS.log file in %windir%\logs\cbs
    It will give more verbose errors usually.


  • Service Provider

    Update for all those that suggested ideas. I took @dafyre 's idea to restore it to a hyper visor. I went on site and I'm not sure why, but it took like 11 hours to copy the backups 1TB+ to an external USB drive. I brought that back to my office and started the restore. That took about 7 hours each time I did that. The first time I just restored the c: drive. After messing around with bcdedit I still couldn't get the thing to boot. Veeam said that the M: drive was a system drive to, so I created another VM and restored the C: and M: drive. This time I could boot to the Dell system installer setup, but still couldn't boot the OS.

    Then I decided to restore to another physical Dell server I had on the bench. It booted no problem. Veeam boots you to Directory Services Restore mode and then you have to use msconfig to tell it to do a normal boot and you're good. I did that and it seemed like it was having the same issue where I logged in and it showed the desktop but wouldn't respond to the keyboard. The mouse moved, but wouldn't let me click. I just left it and came back an hour later. At that point it was fine. Not sure what the deal was.

    There were a few variables to take in to consideration. Since the NIC was different, none of the network services came up. I also disabled a few things like CrashPlan because when the NIC does come online, I don't want it to try to backup to the cloud since it's a clone of the real server that is still on production.

    At this point I'm going to try to P2V the thing.



  • @Mike-Davis

    NOT 100% Sure,

    but if you go here:

    C:\Windows\Installer

    Can you try to sorted it by latest modified date, then run the MSI and see if some can be repaired on uninstalled ?



  • We patched over the weekend and had issues with server 2008 R2 on a .net update

    It eventually went through but it took 4-5 hours on that one update


  • Service Provider

    So I changed the disk layout of the VM I was trying to restore to and it booted like a champ. So now I have the offending server running as a VM. It seems as soon as I log in, CPU goes to 100% and it's all due to one svchost that calls Power, PlugPlay, and DcomLaunch. I can kill the svchost but then the server complains and wants to reboot.


  • Service Provider

    I added more CPUs so that the offending process only took up 25% of the total system CPU. Then as I was going through the event viewer I noticed that many of the things that wouldn't start and were timing out had to do with the network. The NIC was still disconnected, so I enabled that. It wasn't recognized in device manager, so I installed the Hyper V integration tools. That fixed the NIC. I still had about 6 devices listed in device manager under "Other devices." Right clicking each one and telling it to update the driver fixed 4 out of 6.



  • @Mike-Davis said in Autoupdates Killed My Windows Server 2008 R2:

    I added more CPUs so that the offending process only took up 25% of the total system CPU. Then as I was going through the event viewer I noticed that many of the things that wouldn't start and were timing out had to do with the network. The NIC was still disconnected, so I enabled that. It wasn't recognized in device manager, so I installed the Hyper V integration tools. That fixed the NIC. I still had about 6 devices listed in device manager under "Other devices." Right clicking each one and telling it to update the driver fixed 4 out of 6.

    Did this improve the usability of the machine? Or was it still dog slow?


  • Service Provider

    @dafyre said in Autoupdates Killed My Windows Server 2008 R2:

    Did this improve the usability of the machine? Or was it still dog slow?

    Yes, adding CPUs was the difference between being able to use it and it being dog slow.


  • Service Provider

    I installed one automatic update, and now the system is blue screening. I knew there might be an issue since that's what started this whole thing, so I created a checkpoint before installing the update. I tried a couple things to clear the STOP: 0x0000006B blue screen that didn't work so I decided to revert to my checkpoint before the update.

    I'm still getting the STOP: 0x0000006B and it keeps booting to the recovery option.

    Building a new DC is looking better every minute. I can still log in to the old physical one I restored to DCpromo it out when I'm done.



  • @Mike-Davis said in Autoupdates Killed My Windows Server 2008 R2:

    I installed one automatic update, and now the system is blue screening. I knew there might be an issue since that's what started this whole thing, so I created a checkpoint before installing the update. I tried a couple things to clear the STOP: 0x0000006B blue screen that didn't work so I decided to revert to my checkpoint before the update.

    I'm still getting the STOP: 0x0000006B and it keeps booting to the recovery option.

    Building a new DC is looking better every minute. I can still log in to the old physical one I restored to DCpromo it out when I'm done.

    Will it work enough to replicate the AD stuff to the new server?


  • Service Provider

    @dafyre said in Autoupdates Killed My Windows Server 2008 R2:

    Will it work enough to replicate the AD stuff to the new server?

    yes, Yesterday I restored one backup to a physical server and restored one back up to a VM. In the end, I had both working, but decided to move ahead with the virtual one. I can now power on the physical one, create a new VM, join the domain, etc and move forward with that. Of course I say this and most of what I have been working on for the last week should have worked...



  • @Mike-Davis said in Autoupdates Killed My Windows Server 2008 R2:

    most of what I have been working on for the last week should have worked...

    When dealing with computers, should work does not always equal does work, lol.



Looks like your connection to MangoLassi was lost, please wait while we try to reconnect.