The answer, It was the boys over in Redmond, WA. Quite the debacle ( Fail Whale ) for Microsoft.

After installation of the following updates:

• Security Update for Windows Server 2003 (KB974318)
• Windows Malicious Software Removal Tool – December 2009 (KB890830)
• Security Update for Windows Server 2003 (KB973904)
• Update for Windows Server 2003 (KB971737)
• Update for Windows Server 2003 (KB973917)
• Security Update for Windows Server 2003 (KB974392)
• Cumulative Security Update for Internet Explorer 8 for Windows Server 2003 (KB976325)

We noticed multiple occurrences  the following warnings in the System event viewer:

A process serving application pool ‘DefaultAppPool’ terminated unexpectedly. The process id was ’3524′. The process exit code was ’0xffffffff’.

These warnings were immediately followed by a lovely error:

Application pool ‘DefaultAppPool’ is being automatically disabled due to a series of failures in the process(es) serving that application pool.

With these messages at hand it became apparent that one of the Windows Security Updates was to blame. Looking into the specifics of each of the updates I had conluded that the suspect was “Update for Windows Server 2003 (KB973917)“.

While I am still working to identify possible solutions for affected servers independently, removing Update 973917 has restored service temporarily for a number of my clients.

Anyone else see the Humor in this?

Take a lesson from this; Windows Automatic Update is kind of like a friend who secretly wants to cause you problems, because its fun….

Updated on December, 9 2009 at 1:59pm

The official response… get this; Microsoft wants us to reinstall Service Pack 2 for Windows Server 2003 on the affected installations to bring all IIS6.0 components up to date. Seems like a Cop out to me. Sure it would fix the issue but this is going to cause a lot of man hours for a number of companies.