The answer, It was the boys over in Redmond, WA. Quite the debacle ( Fail Whale ) for Microsoft.

After installation of the following updates:

• Security Update for Windows Server 2003 (KB974318)
• Windows Malicious Software Removal Tool – December 2009 (KB890830)
• Security Update for Windows Server 2003 (KB973904)
• Update for Windows Server 2003 (KB971737)
• Update for Windows Server 2003 (KB973917)
• Security Update for Windows Server 2003 (KB974392)
• Cumulative Security Update for Internet Explorer 8 for Windows Server 2003 (KB976325)

We noticed multiple occurrences  the following warnings in the System event viewer:

A process serving application pool ‘DefaultAppPool’ terminated unexpectedly. The process id was ’3524′. The process exit code was ’0xffffffff’.

These warnings were immediately followed by a lovely error:

Application pool ‘DefaultAppPool’ is being automatically disabled due to a series of failures in the process(es) serving that application pool.

With these messages at hand it became apparent that one of the Windows Security Updates was to blame. Looking into the specifics of each of the updates I had conluded that the suspect was “Update for Windows Server 2003 (KB973917)“.

While I am still working to identify possible solutions for affected servers independently, removing Update 973917 has restored service temporarily for a number of my clients.

Anyone else see the Humor in this?

Take a lesson from this; Windows Automatic Update is kind of like a friend who secretly wants to cause you problems, because its fun….

Updated on December, 9 2009 at 1:59pm

The official response… get this; Microsoft wants us to reinstall Service Pack 2 for Windows Server 2003 on the affected installations to bring all IIS6.0 components up to date. Seems like a Cop out to me. Sure it would fix the issue but this is going to cause a lot of man hours for a number of companies.

Here are the steps to solve the problem:

1. Right click on the target virtual machine in the VMware console and select “Remove from Inventory.” This will remove the virtual machine from the Inventory list in the console without deleting the guest image.
2. In the VMware server console select File -> Open -> Browse to the folder that contains the relevant *.vmx virtual machine configuration file.
3. Select Ok. The problem virtual machine will now be listed VMware Server Console Inventory.
4. You should now be able to power on (start) the guest image.

Note: If your still experiencing the same error message, navigate to the folder that contains the VMware guest images configuration files and verify that the relevant user has full NTFS permission on the folder.