"Microsoft Exchange Service Host" fails to start in Secure Exchange 2007 Environments
On Exchange Server 2007, especially post service pack 1 (I ran into this problem while installing service pack 1 update rollup 2), it appears that some of the assemblies (that is, parts of the various Exchange programs) are "signed with Authenticode". This is basically a public/private key infrastructure (PKI) supported by Microsoft to allow people to verify that the various Exchange programs have not been tampered with.
At this moment, you are either thinking "So what?" or "Sounds like a good idea."
Personally, I'm in the "sounds like a good idea" camp. But I'm not sure that the concept is fully baked.
Consider this: part of any PKI infrastructure is a way to indicated when a particular key is no longer valid. Without going into too many PKI details, a key is normally called a "certificate" and the way you check the validity of a key is by seeing if the key is on a CRL - a certificate revocation list.
The first time a managed and signed assembly loads, Windows Server needs to check and see whether the key (certificate) the assembly is signed with is valid. How does it do that? By checking the CRL - at Microsoft. Specifically, at http://crl.microsoft.com/pki/crl/products/CSPCA.crl.
Even if you are in the "sounds like a good idea" camp, you may be asking "what's wrong with that"?
So I'll tell you - many environments do not allow their mailbox servers to talk to the internet (just to the hubs and to clients on specific networks). Many environments do not allow their hub transport servers to talk to the Internet (just to the mailbox and client access and e-mail gateway servers). Etc. etc. In fact, controlling and minimizing access using IPSec to specific servers has long been considered a best practice.
But it won't work with Exchange Server 2007 any more. When installing rollup 2 on a mailbox server, the "Microsoft Exchange Service Host" will fail to start. It will simply time out. No error messages. No log messages. No nothing.
I suspect that it could be any Exchange service that has this problem. This was just the one I happened to run into.
Debugging this was painful. Eventually, after a couple of hours, you do a "netstat -an" and say WTF???!!! in regards to a SYN-SENT to crl.microsoft.com and then you can start backtracking. This is not really documented directly (that is, that an Exchange server has to be able to access the Internet), but there is some information in KB 944752.
You may also find that if you are using a non-NAT proxy that this doesn't work for you. More on that in my next blog post.
Bottom line: after installing patches on your Exchange server, your Exchange server will have to "check in at home" before those patches can start running and you'll have to enable Internet access for that to happen. After they've "checked in", you can disable Internet access again.
What a management nightmare.
Until next time...
As always, if there are items you would like me to talk about, please drop me a line and let me know!
Update on June 19, 2008; for those folks who want for at least some of their Exchange servers to be fully isolated from the Internet, there is a workaround. See KB 936707. However, an easier (and apparently undocumented) workaround is to put crl.microsoft.com in your local hosts file - and point it to localhost (127.0.0.1)!