Michael's meanderings...

Exchange Server 2007 service pack 1 rollup 2

michael — Fri, 09 May 2008 13:03:00 GMT

Exchange 2007 SP1 RU2 is out! This one is pretty big for clusters and if you use POP3 and IMAP in your organization. I recommend you roll it out quickly.

More about it at KB 948016. Lots of patches in this one.

Other Certificate Limitations with Exchange/OCS/WM

michael — Thu, 08 May 2008 13:02:00 GMT

This is turning out to be SSL certificate week here at TheEssentialExchange.... not planned that way, just the interesting items that are crossing my computer. Yesterday, I wrote ISA 2006 and SAN/UC Certificates describing how ISA 2006 requires careful configuration in order to properly use a SAN/UC certificate.

Today, three more items you need to be aware of:

1) Windows Mobile (any version) does not support wildcard certificates

2) Outlook Anywhere (and RPC/HTTP for Exchange 2003) does not support wildcard certificates

3) UM requires that the machine name of the UM server be the first SAN (or only, for a singly named certificate) in the certificate

Lee Mackey, a fellow Exchange MVP, recently wrote: certs in exchange are like gouging your eye out with a spoon!

I echo his sentiment. That being said, there is no question that the product team understands that certificates are "too hard" right now in both Exchange Server 2007 and in OCS 2007. We'll just have to wait and see what they come up with to deal with this, in the future...

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

ISA 2006 and SAN/UC Certificates

michael — Wed, 07 May 2008 12:49:00 GMT

Much has been written about ISA 2006 "not supporting SAN certificates".

And, to some degree, this is true: the RTM version of ISA 2006 will only recognize the first subject alternative name in a certificate.

So what do you do if the name you need to use is not the first SAN in the certificate? You fake it out. :-)

What you publish using ISA 2006 is the entire certificate, to the outside. What's important to ISA is how you use it, inside. Clear as mud?

This is the three-step work-around:

For the rule that you have an issue with, open the Property sheet.
On the "Web Farm" tab, the "Internal site name" must be the first subject alternative name published on the certificate.
On the "Public Name" tab, add both names - the name you want to use (e.g., mail.contoso.com) as well as the first SAN on the certificate (e.g., autodiscover.contoso.com).

That's all it takes. And, of course, if you add the first SAN first when you create the certificate, none of this is a problem either. And most Certification Authorities will re-issue certificates within a few days for free, if you messed the request up.

Truly, much ado about little...

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Inexpensive UCC/SAN Certificates

michael — Mon, 05 May 2008 12:15:00 GMT

When Exchange Server 2007 was first released (even before then, if truth be told), there was quite an outrage that in order to use a simplified configuration process for Outlook Anywhere, OWA, and AutoDiscovery, the Exchange administrator had to use a special kind of SSL certiifcate that supports multiple Subject Alternative Names (SANs).

Now, most Exchange administrators had never heard of such a thing (I know that I hadn't).

A SAN certificate falls "in the middle" between a wildcard certificate (*.example.com) and a single-server certificate (www.example.com); including a feature that wildcard certificates cannot support - different domains in the alternative names. This allows a SAN certificate to have a list like this: mail.example.com, owa.example.com, autodiscover.example.com, mail.example.local, owa.example.local, and autodiscover.sample.local. Where the first three are EXTERNAL server names, and the last three are INTERNAL server names. Pretty slick, eh?

The problem was the cost. SAN certificates are MUCH more expensive than a single name cert. (Supply and demand, of course...)

Then, Microsoft began using SAN Certs in OCS 2007. The common name for a SAN certificate is gradually changing over to UC Certificate (for Unified Communications Certificate).

Microsoft only has three "certified" partners for providing UC certificates. They are presented in KB 929395 with prices from $325 to $599 per year for a UC certificate. Compare this to RapidSSL at $20 per year for a standard SSL certificate....(all prices in US dollars).

So, as competitive forces usually do, someone else came up with a less expensive solution.

Certificates for Exchange produces inexpensve UC certificates (starting at $30 per year) that work 100% with Exchange Server 2007 and all version of Windows Mobile 5 and up.

I am not associated with Certificates for Exchange in any way. I'm just happy that someone released an inexpensive UC certificate solution and thought you, my readers, would be too.

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Clean Up That Store And Shrink It Too!

michael — Thu, 01 May 2008 01:39:00 GMT

During a database backup you got a dreaded -1018, or -1019 error - indicating that your Exchange store is corrupt. But Exchange keeps running...

Or, you had kept dozens of "old" user mailboxes sitting around and you finally decided to purge them. After that, you find your Exchange database is half empty.

What do you do in these situations?

In the "old days", you would take your Exchange store offline, take an offline backup, and then start up ESEUTIL. The first step is "garbage collection". This removes all the unused space from the Exchange database, and compresses all the indexes - giving you a smaller Exchange database. Then you remount the database and it's all good.

Note: Garbage collection only removes unused space that has been processed by "online maintenance". If you remove 10 GB of mailboxes, and don't allow online maintenance to complete first, when you run garbage collection, that space will not be returned - because as far as the database software is concerned, the space is still in-use. This is similar to the difference between soft-delete and hard-delete in individual mailboxes.

Next, you take an online backup. The act of doing a garbage collection invalidates your earlier backups and transaction logfiles.

If you started this process because of a -1018 or a -1019, and the backup completes successfully, you are golden! You've successfully returned your database to production quality with no data loss. If you again receive an error, you must execute a database repair, during which you will almost certainly lose data. I'm not going to cover that here; if you are not certain what to do, your best bet is to open a case with Microsoft CSS/PSS.

Now, depending on your database size, and the amount of whitespace in that database, you may have had your mailbox database offline for many hours - causing pain and heartache for your users, and potentially bounced e-mail.

Let's talk about a mechanism for avoiding this downtime.

Note: This process will only apply to Exchange 2000 Server and Exchange Server 2003 Enterprise Editions, or to Exchange Server 2007. In Exchange 2000/2003, Standard Edition only supports a single mailbox store. In Exchange Server 2007 Standard Edition that limitation was (finally!) increased to five mailbox stores.

First, you must verify that you have at least twice as much disk storage as the mailbox store in question. If you do not, then an easy way to get significant additional temporary storage is to attach a RAID-1 (mirrored) USB disk (a couple of options are from Lacie's BiggerDisk line and Netgear's StorageCenter line - both of which I have used before). Please don't use a non-RAID solution. Cheap disk tends to not be enterprise quality, and fails quickly. Only trust your mailbox data to a RAID-based disk solution.

Next, create a new storage group and a new mailbox store in the location where you have the extra disk space.

Next, use the move-mailbox wizard to move all of the mailboxes from the old mailbox database to the new mailbox database.

Next, dismount the old mailbox store.

Next, REMOVE THE OLD MAILBOX STORE FILES. Yep, you read that right. But ENSURE and VERIFY you are removing the correct files. If you want to make certain, rename them instead and remove them later.

Next, remount the "old" mailbox store. A new empty database will be created.

Next, use the move-mailbox wizard to move all of the mailboxes from the temporary mailbox database back to their original mailbox database.

Next, dismount and remove the temporary mailbox database and storage group.

Finally, take a full backup.

There is no system downtime. During the entire process your Exchange server can continue to send and receive e-mail. However, there is some impact. Those mailboxes that are currently in use when you try to move-mailbox will not move. Outlook cannot connect to a mailbox in the process of being moved. However, this is a minimal impact compared to hours of system downtime. And, you can do it all in the background.

Add this to your repertoire. Practice it. It's easy to do. I've told you how. :-)

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Exchange ESE, Failed Backups, the Checkpoint Depth, and You

michael — Mon, 28 Apr 2008 18:33:00 GMT

A question came up recently on a mailing list I hang around. A poster wrote: "I'm looking for a concise explanation of when/how the transaction logs are committed. Specifically, is there another way to force logs to be committed other than the backup or stopping the IS? (E2K3)"

As with many things in Exchange, when you get down to the bits and the bytes, things are not always as they seem. My response, which some of you might be interested in, went more-or-less less like the below (I've expanded it a little, and added a few paragraphs on checkpoint exhaustion). Also note that this only applies to streaming backups. Shadow-copy backups are different.

A concise explanation? Hrmmm.

Well, first, doing a backup does NOT force logs to be committed. It means that the backup process waits until a checkpoint occurs, flushes the current transaction log, and during the backup additional checkpoints are not allowed to occur (that is, nothing is allowed to be flushed to the ESE database until after the backup is complete – leading to an increasing checkpoint depth, and in extreme situations, checkpoint exhaustion – discussed a little later). The ESE buffers are not flushed. The "checkpoint depth" is how many checkpointed logfiles must be processed before the ESE database is current with the transaction log files.

So...in the normal case, data is written in a serialized fashion to the in-memory ESE cache and to the log buffers, as updates occur in an Exchange database. Logs are written to disk as logs fill up, or as transaction commits occur. (This may mean that a log can have nothing but a checkpoint record in it – but that is not the normal case.) In the ESE buffers, I/O is accumulated and prioritized by a process known as the “lazy writer”. The lazy writer scans the ESE buffers for dirty (modified) pages and builds an optimized list to flush those to disk. As that list is flushed to disk, the pages are marked as “clean”, and the checkpoint is marked as having advanced (on a transaction by transaction basis, not a log by log basis). Whenever the transaction in a log are fully advanced, then the checkpoint file and the database header are updated.

During a backup, the lazy writer is paused.

The ONLY time you are assured that logs are fully committed is during clean shutdown, or after running soft recovery.

Now, consider the situation when a backup is in process and it gets hung (i.e., waiting on a tape, or there was a fatal write error on the tape and the backup application crashed). What happens? There are three possible answers.

1) The backup application caught the exception (i.e., the error that caused the crash) and cleaned up after itself. In this case, it appears as if the bad backup never happened and everything just goes back to normal. This is the best possible case.

2) The backup application didn't catch the exception, but "notices" when it next runs that there was a backup in process and isn't one now, but the backup state is still "dirty". Then, the backup application cleans up the old "dirty" backup state, and proceeds as normal. This is the middle of the road case.

3) The backup application didn't catch the exception, and when it goes to start the next backup, it gets an error from Exchange - "backup already in process". Then, because of that error, the backup aborts. This is the worst situation. To clear the "backup in process" flag requires a restart of the Microsoft Exchange Information Store service.

In any of these cases, the checkpoint depth is continuing to advance - and the log files are accumulating, but nothing is being committed to the database. This is a problem! Eventually, the Information Store will say "enough!" and dismount the storage group. That's right - it will dismount the entire storage group.

This is called "Checkpoint depth exhaustion". After 1,008 logfiles have been generated, Exchange automatically shuts down the storage group. There is an implementation limit of 1,024 logfiles in a checkpoint, and this shutdown prevents the Exchange database from being corrupted.

After the storage group is dismounted, MSExchangeIS can simply be restarted and the storage group remounted (this will cause all of those transaction logfiles to be replayed - it may take quite awhile!).

This can also happen in very very high update situations (such as running many move mailbox operations all at once time).

Your best bet is to monitor for this. If you are using OpsMgr 2007, this is already one of the issues checked for. If you want to add this check manually, add the performance counter "Database ==> Instances\Log Generation Checkpoint Depth".

For more information on this issue, see KB 905801 and KB 819771.

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Xobni - The Next Revolution in E-Mail

michael — Mon, 21 Apr 2008 00:43:00 GMT

I hate to sound like a marketing goon, but I have to tell you about Xobni.

But I've been using Xobni for four days. I don't know how I ever got by without it. Comparatively speaking, I can now process my e-mail at lightspeed.

I don't know how YOU work with e-mail, but I spend a great deal of time referring back and dealing with the same people and pulling up prior e-mails, etc. etc. In order to do that, I've developed a pretty well-organized but moderately complex filing system based (of course) on Outlook folders.

With Xobni, I still file everything, but Xobni keeps it organized for me. If I click on an e-mail from someone, Xobni finds me all related threads, all related attachments, contact information for that party, and other stuff.

And yes, Xobni is "inbox" spelled backwards.

The website is the obvious place: http://www.xobni.com.

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

A Work In Progress...

michael — Fri, 18 Apr 2008 17:45:00 GMT

Yesterday (Thursday, 2008-04-17) was the last day of the 2008 MVP Summit. Over one-half of the MVPs attending were from countries other than the United States. It was interesting to hear the dozens of languages being spoken around me this week.

Very little of what was covered during the Summit is not covered by NDA. But you can be assured that MVPs from every product that Microsoft produces were there learning about what is coming and providing "feedback" to the Microsoft product teams. Feedback can get quite .... loud .... at times. MVPs tend to be quite passionate about their products of interest and very protective of their user communities. Many features present in current releases of Exchange came at MVP "encouragement".

In fact, one feature added quite recently was only added because MVPs said "it has to be there". And in the future, we see that it will be expanded significantly to meet other needs. Yay MVPs!

The last presentation of the Summit was a keynote given by Steve Ballmer. During his hour on the stage, he discussed many things and took about 20 minutes of questions from the audience. Two of the items he said really struck me. The first: "Windows Vista...is a work in progress". The second was: "We can't go five years in between releasing versions of the operating system".

There are many things that can be read into those two statements. However, it is clear that Microsoft knows that Vista isn't where it should be yet, and it's likely that "Windows 7" won't take as long to get here as Vista did.

Changing directions...Steve took several questions about Groove vs. SharePoint as did Ray Ozzie, who spoke before Steve. There is a definite belief in those MVP communities that one of these collaboration tools has a limited lifetime (there is a lot of overlap). Who knows what it will be?

You can read more about Ballmer's speech in an article from the Seattle Times.

As always I met many people, shook many hands, got to see some old friends, and lifted a few while bonding with the other geeks around me. :-) A good time and very worthwhile.

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Why Would You Want Exchange Server With A SQL Backend?

michael — Wed, 16 Apr 2008 15:54:00 GMT

Consider the format of an Exchange database.

It's flat. Very flat.

Take a mailbox. Basically, it is a table that contains pointers to other tables. What are those other tables? Well, your Inbox, Sent Items, Calendar, Tasks, Notes, etc....

Every folder is a table. Each table contains objects and pointers to other tables. An object may be a message, a task, a calendar entry - basically the low-level objects which Outlook manipulates.

Every view is an index - and an index is just a table of sorted pointers to objects (plus a small bit of meta-data).

So... each mailbox may have thousands of tables. A mailbox database may have millions of tables. Yes, you read that right: millions.

Why is it done that way? Easy answer! To allow the user maximum flexibility. The end-user is not limited in the number of views, not limited by what he can sort by, how many objects you can contain in a folder (although there certainly are performance issues associated large numbers of items), etc. etc. Exchange Server generates these views on the fly whenever Outlook requests them. Real-time. The end-user is not arbitrarily limited in any significant way.

Give a great pat on the back to Exchange!

How does Exchange do this? With ESE. Because ESE - the Exchange database engine - supports these features.

Now, consider Microsoft SQL. A great DB engine. However, it doesn't really generate tables on the fly. Or views on the fly. Or have a hierarchical mechanism for storing table trees. Or clean up after itself. All of these functions have latency in SQL...

So....what would have to happen? A complete redesign. Not just a simple "let's use a different backend". And, it's likely that a lot of features would be lost in any such translation.

Do I want Exchange in SQL? No. Do I think it would work well? Not with any SQL Server released to date.

In the future? Who knows? Maybe. But I don't know.

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Live from Redmond - it's Geeks Argue!

michael — Mon, 14 Apr 2008 15:31:00 GMT

I and 4,000 MVPs from around the world have accumulated in Seattle/Redmond this week. Where do we all go? To the bar, of course!

Last night, a few Exchange MVPs and the odd PowerShell MVP were sitting around discussing stuff I can't talk about here, when another MVP came to join us (who happens to work for a major storage vendor). Hey, we are open minded, no problem there!

Then, out of nowhere, he began to bash our beloved Exchange!

Interestingly enough, the basic complaint was this: Exchange degrades too gracefully. That is, you can configure Exchange in ways that are downright stupid, it will let you, and it will "sorta" run - until you finally hit that "straw that breaks the camel's back" and then it crashes.

Well, that's a valid complaint. Exchange goes to extreme lengths to allow your e-mail to go through - because e-mail is mission critical. This is what it should do, in my opinion.

And in doing this, if you want to put 1.2 million mailboxes on a single LUN of a SAN; well, it'll let you. Doesn't mean that it is the right thing to do! And it doesn't mean that Exchange is broken because it lets you do this!

There is nothing that excuses poor configuration. Whether it is Exchange itself, the storage subsystem, the network, whatever. The entire Exchange ecosystem, in large installations, must be planned and monitored.

Truthfully, in many small to medium installations, Exchange is so forgiving that you can do almost anything to it, and it'll continue to run. Most SMORGS don't have access to the technical expertise to do a proper design/rollout/support/etc. and with the hardware of today, that's ok. And Microsoft puts together packages of software to help this be OK (think Small Business Server and Essential Business Server).

But when an installation gets large than, oh, around 250 mailboxes - you best begin to take planning seriously. Planning, monitoring, and proper operations are a system administrator's ongoing responsibility.

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Why Pay More? Try ImageX

michael — Fri, 11 Apr 2008 13:16:00 GMT

Even though I get pretty hot under-the-collar sometimes when Microsoft removes for-free functionality (Exchange backups on Windows Server 2008, for example), there are some great tools that Microsoft provides for-free that are just as good as their for-pay alternatives - and more tightly integrated with other Windows functionality. However, they are often not well known.

One of those is WDS - Windows Deployment Services. WDS provides easy/simpler ways to, well, Deploy Windows! (D'Oh!)

Part of WDS is ImageX. ImageX provides a free-and-easy way to create images of Windows desktops, including deployment images. One of the great features of ImageX is that you can easily load drivers, patches, service packs, etc - AFTER the image is created, without creating a brand new image. This is a great time saver. It also allows you to create and install "generic" images that have multiple HALs (Hardware Abstraction Layers) and storage subsystems (as long as the drivers are included in the base image!).

Check it out.

Windows Automated Installation Kit

ImageX Technical Reference

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Perfmon tip on scaling

michael — Thu, 10 Apr 2008 14:19:00 GMT

Recently on a mailing list, several folks expressed some confusion about the scaling that Perfmon (Performance Monitor) does in its graphs and how it affects the data.

The answer is: it doesn't. The scaling is used only to make the graph fit in the Perfmon window. The values contained in the Last/Average/Minimum fields are not scaled, they are the actual counter values.

If you want to prove this to yourself in another way, click on "Change Graph Type" (ctrl-G for you keyboard folks like me) and select "Report". You'll see the same size of values reported in that list.

If you've never played with Perfmon before (or the entire "Reliability and Performance Monitor" that is part of Vista and Server 2008), you should learn how to use it. It can make debugging performance issues a lot easier.

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Exchange 2007 Catchall Mailbox

michael — Thu, 10 Apr 2008 14:11:00 GMT

I wrote and supported two versions of a Catchall Mailbox script that worked with Exchange 2000 Server and Exchange Server 2003. Version 2 of it is available on this blog, at this link: Exchange 200x Catchall Script, Version 2.

However, the transport engine in Exchange Server 2007 is completely different. The old script interfaces no longer work.

However, as a sample, an Exchange engineer developed the Catchall for Exchange Server 2007 and posted it on CodePlex, calling it CatchAllAgent Exchange 2007 Transport Protocol Agent. If you need the functionality for Exchange Server 2007, that's where to get it!

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Default Gateways and Exchange Server

michael — Wed, 09 Apr 2008 12:59:00 GMT

Sometimes, you need to step back and take a different look at a problem. When you are "in the weeds", you can't necessarily see the forest for the trees.

Exchange Server is a networking application. Now, before you say "duh!", think about it. If your network is having issues, this means that Exchange Server may have issues too. However, since Exchange Server is always doing something on the network - it may appear that the problem is with Exchange.

I recently saw an issue that was caused by a problem with multiple default gateways.

A very common configuration on an Exchange Server is to have one NIC (Network Interface Card) be the "public" or "untrusted" interface. Through this NIC flows all traffic destined for the Internet. Then, the Exchange Server has another NIC that is the "private" or "trusted" interface that connects to the "internal" network. This NIC is generally for such things as backups, monitoring, etc.

When you configure a NIC with a fixed IP address, you are asked to supply three things: an IP address that uniquely identifies this server, a netmask that defines the size of the network that the IP address is a member of, and a default gateway to be used when an IP address isn't part of the same network as the server.

(Of course, there are many other properties you may configure as well - but these three are the ones that are required to put a computer "on the net".)

Notice the use of the default gateway: it is the destination where information not on the local network is sent. So, for a trivial example, if your local network is "DC-Server" and "Exchange-Server" and you want to send an e-mail to "TheEssentialExchange.com" then that information will go out your default gateway.

As you may have noticed, Windows allows you to enter multiple default gateways on a single NIC. This is for a very rarely used feature of IP called "dead gateway detection". In this case, if one default gateway should happen to go offline, Windows would automatically switch to another gateway. This is not a load-balancing technique or performance enhancing technique.

If you are looking for load-balancing or performance enhancing mechanisms, you should be looking at NICs that support "teaming" and/or switches that support EtherChannel.

Now, consider what happens when, in both the public and the private NIC, you enter a default gateway. Which one is really the default?

Huh. Think about it.

What is Windows to do? Flip a coin?

Well, that is about what happens. Windows becomes a non-deterministic router (this means that you cannot conclusively define which default gateway that Windows will always use). Windows will switch back-and-forth between the default gateways.

What does this mean to you, the Exchange/network administrator? It means that Exchange, on the public interface, will appear to come offline and online at unpredictable intervals. Why only on the public interface? Because there is a specific defined route to access the private network. Not on the public interface though.

So...don't do this. It isn't a good idea on Windows networks, in general, because of the non-deterministic nature of the routing that will occur. And specifically for Exchange, it is known to cause problems.

Here is one reference for you: Multiple default gateways detected.

And even though these KB articles were written for Windows NT 3.5, they still apply today: KB 157025 and KB 159168.

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Business cards....

michael — Tue, 08 Apr 2008 12:28:00 GMT

So, "The Essential Exchange" is my front-end to my consulting business.

I admit that marketing is not my strong suit. So far, I'm supporting myself on writing and people using my services from the communities I help support. But this week I'm meeting with an honest-to-gosh client who doesn't know me.

What do I need? Besides a haircut? :-)

Business cards.

So, me and my s/o, neither of us being graphics people, put our heads together and came up with the below. We did conclude that we need to hire someone to help us develop a logo and other marketing material. Her company has used Logo-Mojo with good success, so I'm thinking that that is who I will use. If you have other suggestions please pass them along!

Until next time...