IIS Service Unavailable aka. Windows Security update kills application pool

Last night all severs running Windows 2003 R2, IIS6 and ColdFusion/ASP.Net were sabotaged! Actually, any windows based web application making use of the IIS6 application pool was affected and guess who killed them… Havoc is consuming the inter-webs while non-technical users are asking the question; “who killed my web site”?

The answer, It was the boys over in Redmond, WA. Quite the debacle ( Fail Whale ) for Microsoft.

After installation of the following updates:

  • Security Update for Windows Server 2003 (KB974318)
  • Windows Malicious Software Removal Tool – December 2009 (KB890830)
  • Security Update for Windows Server 2003 (KB973904)
  • Update for Windows Server 2003 (KB971737)
  • Update for Windows Server 2003 (KB973917)
  • Security Update for Windows Server 2003 (KB974392)
  • Cumulative Security Update for Internet Explorer 8 for Windows Server 2003 (KB976325)

We noticed multiple occurrences  the following warnings in the System event viewer:

A process serving application pool ‘DefaultAppPool’ terminated unexpectedly. The process id was ’3524′. The process exit code was ’0xffffffff’.

These warnings were immediately followed by a lovely error:

Application pool ‘DefaultAppPool’ is being automatically disabled due to a series of failures in the process(es) serving that application pool.

With these messages at hand it became apparent that one of the Windows Security Updates was to blame. Looking into the specifics of each of the updates I had conluded that the suspect was “Update for Windows Server 2003 (KB973917)“.

While I am still working to identify possible solutions for affected servers independently, removing Update 973917 has restored service temporarily for a number of my clients.

Anyone else see the Humor in this?

Take a lesson from this; Windows Automatic Update is kind of like a friend who secretly wants to cause you problems, because its fun….

Updated on December, 9 2009 at 1:59pm

The official response… get this; Microsoft wants us to reinstall Service Pack 2 for Windows Server 2003 on the affected installations to bring all IIS6.0 components up to date. Seems like a Cop out to me. Sure it would fix the issue but this is going to cause a lot of man hours for a number of companies.

16 Comments so far

  1. Howard on December 9th, 2009

    You are awesome. You saved me a day with this post. I uninstalled the updates you mention and my problems went away. I’m working on reinstalling SP2 right now and then I’ll reapply the updates. Hopefully all goes well. :)

  2. Ian on December 9th, 2009

    This morning our intranet server had exactly the same error, I uninstalled kb973917 as specified in your post, rebooted and problems solved, thanks for the advise.

  3. chris on December 10th, 2009

    Same as buddy, this update b0rked 3 of my IIS machines!

  4. Bremaud on December 10th, 2009

    Thanks a lot for this post.You saved me !! T

  5. Xanadu on December 10th, 2009

    Thank you. I hope MS will give elucidations about!

    Bye

    Xanadu

  6. Chris on December 10th, 2009

    Thanks for your timely post. I spent 4+ hours trying to troubleshoot this issue. This morning I decided to investigate any “malicious” patches that may have been installed by Microsoft. This post was at the top of Google. Uninstalled the patch and boom, back in business. Thanks!

  7. Dennis Somerville on December 10th, 2009

    Thank you very much for the post. This saved me a lot of time. Here is a similar response with some additional detail:

    Previous to the installation of the KB973917 update, one or more of the core IIS .dll files were not at the correct file version.  Specifically, the earlier installation of Windows Server 2003 Service Pack 2 on the server did not complete successfully, leaving some of the IIS .dlls at the Service Pack 1 level while bringing others up to the Service Pack 2 level.  The IIS services had been able to run successfully even with that file mismatch in place.  However, the installation of the KB973917 update exposes this pre-existing file mismatch environment to the degree that IIS is now unable to function properly.

    To resolve this problem, reinstall Service Pack 2 for Windows Server 2003 on the web server.  This will bring all IIS 6.0 components up to the correct file versions, and will maintain the installation of the KB973917 update.  Reinstalling the KB973917 update should not be necessary.

  8. Neil on December 10th, 2009

    Thanks mate, Spent a good 4hrs on this. Would have probably taken a few days to spot this otherwise.

  9. Pete on December 11th, 2009

    You beauty, thanks very much!

  10. Packetboy on December 15th, 2009

    Another thank you!! We had some prod report servers go down thanks to this patch you also saved me.. I owe you a beer!!

    Packetboy.

  11. joetATratebeer on December 17th, 2009

    THANKS! I can’t believe this! I uninstalled and then reinstalled SP2 and still had issues with permissions with several processes.

    This was a huge mess.

    A bigger problem is that Redmond didn’t correct the issue and the update was sent down AGAIN several days later!

  12. Josh on December 18th, 2009

    You are the man! Thank you so much! I was this close to re-installing IIS…so nice to simply uninstall that patch so that our CRM Reporting features worked again!

  13. fwiw on December 23rd, 2009

    I’m still having problems related to this update. Microsoft has been totally unresponsive. Uninstalling the security update above allows IIS to run without errors but some other software we run is still crippled and either does not run or terminates related processes.

  14. gavin.mclelland on December 23rd, 2009

    Have you tried reinstalling Service Pack 2? Though inelegant, this should fix the issue for you.

  15. Tom on March 2nd, 2010

    Thanks a bunch for this. I just spent the last two hours trying to figure this out to no avail until I found this. (Would have discovered the problem sooner, but hadn’t worked on this machine since Nov 2009!)

    Thanks again!

  16. Edouard on July 21st, 2010

    Damned…

    You saved my life guys !!!

    Simply uninstall this damned stuff…
    And it wroked fine !!!

    Bless you
    Edouard

Leave a Reply

Get Adobe Flash playerPlugin by wpburn.com wordpress themes