April 2008 - Posts

During a database backup you got a dreaded -1018, or -1019 error - indicating that your Exchange store is corrupt. But Exchange keeps running...

Or, you had kept dozens of "old" user mailboxes sitting around and you finally decided to purge them. After that, you find your Exchange database is half empty.

What do you do in these situations?

In the "old days", you would take your Exchange store offline, take an offline backup, and then start up ESEUTIL. The first step is "garbage collection". This removes all the unused space from the Exchange database, and compresses all the indexes - giving you a smaller Exchange database. Then you remount the database and it's all good.

Note: Garbage collection only removes unused space that has been processed by "online maintenance". If you remove 10 GB of mailboxes, and don't allow online maintenance to complete first, when you run garbage collection, that space will not be returned - because as far as the database software is concerned, the space is still in-use. This is similar to the difference between soft-delete and hard-delete in individual mailboxes.

Next, you take an online backup. The act of doing a garbage collection invalidates your earlier backups and transaction logfiles.

If you started this process because of a -1018 or a -1019, and the backup completes successfully, you are golden! You've successfully returned your database to production quality with no data loss. If you again receive an error, you must execute a database repair, during which you will almost certainly lose data. I'm not going to cover that here; if you are not certain what to do, your best bet is to open a case with Microsoft CSS/PSS.

Now, depending on your database size, and the amount of whitespace in that database, you may have had your mailbox database offline for many hours - causing pain and heartache for your users, and potentially bounced e-mail.

Let's talk about a mechanism for avoiding this downtime.

Note: This process will only apply to Exchange 2000 Server and Exchange Server 2003 Enterprise Editions, or to Exchange Server 2007. In Exchange 2000/2003, Standard Edition only supports a single mailbox store. In Exchange Server 2007 Standard Edition that limitation was (finally!) increased to five mailbox stores.

First, you must verify that you have at least twice as much disk storage as the mailbox store in question. If you do not, then an easy way to get significant additional temporary storage is to attach a RAID-1 (mirrored) USB disk (a couple of options are from Lacie's BiggerDisk line and Netgear's StorageCenter line - both of which I have used before). Please don't use a non-RAID solution. Cheap disk tends to not be enterprise quality, and fails quickly. Only trust your mailbox data to a RAID-based disk solution.

Next, create a new storage group and a new mailbox store in the location where you have the extra disk space.

Next, use the move-mailbox wizard to move all of the mailboxes from the old mailbox database to the new mailbox database.

Next, dismount the old mailbox store.

Next, REMOVE THE OLD MAILBOX STORE FILES. Yep, you read that right. But ENSURE and VERIFY you are removing the correct files. If you want to make certain, rename them instead and remove them later.

Next, remount the "old" mailbox store. A new empty database will be created.

Next, use the move-mailbox wizard to move all of the mailboxes from the temporary mailbox database back to their original mailbox database.

Next, dismount and remove the temporary mailbox database and storage group.

Finally, take a full backup.

There is no system downtime. During the entire process your Exchange server can continue to send and receive e-mail. However, there is some impact. Those mailboxes that are currently in use when you try to move-mailbox will not move. Outlook cannot connect to a mailbox in the process of being moved. However, this is a minimal impact compared to hours of system downtime. And, you can do it all in the background.

Add this to your repertoire. Practice it. It's easy to do. I've told you how. :-)

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Posted by michael | 2 comment(s)
Filed under:

A question came up recently on a mailing list I hang around. A poster wrote: "I'm looking for a concise explanation of when/how the transaction logs are committed.  Specifically, is there another way to force logs to be committed other than the backup or stopping the IS?  (E2K3)"

As with many things in Exchange, when you get down to the bits and the bytes, things are not always as they seem. My response, which some of you might be interested in, went more-or-less less like the below (I've expanded it a little, and added a few paragraphs on checkpoint exhaustion). Also note that this only applies to streaming backups. Shadow-copy backups are different.

A concise explanation? Hrmmm.

Well, first, doing a backup does NOT force logs to be committed. It means that the backup process waits until a checkpoint occurs, flushes the current transaction log, and during the backup additional checkpoints are not allowed to occur (that is, nothing is allowed to be flushed to the ESE database until after the backup is complete – leading to an increasing checkpoint depth, and in extreme situations, checkpoint exhaustion – discussed a little later). The ESE buffers are not flushed. The "checkpoint depth" is how many checkpointed logfiles must be processed before the ESE database is current with the transaction log files.

So...in the normal case, data is written in a serialized fashion to the in-memory ESE cache and to the log buffers, as updates occur in an Exchange database. Logs are written to disk as logs fill up, or as transaction commits occur. (This may mean that a log can have nothing but a checkpoint record in it – but that is not the normal case.) In the ESE buffers, I/O is accumulated and prioritized by a process known as the “lazy writer”. The lazy writer scans the ESE buffers for dirty (modified) pages and builds an optimized list to flush those to disk. As that list is flushed to disk, the pages are marked as “clean”, and the checkpoint is marked as having advanced (on a transaction by transaction basis, not a log by log basis). Whenever the transaction in a log are fully advanced, then the checkpoint file and the database header are updated.

During a backup, the lazy writer is paused.

The ONLY time you are assured that logs are fully committed is during clean shutdown, or after running soft recovery.

Now, consider the situation when a backup is in process and it gets hung (i.e., waiting on a tape, or there was a fatal write error on the tape and the backup application crashed). What happens? There are three possible answers.

1) The backup application caught the exception (i.e., the error that caused the crash) and cleaned up after itself. In this case, it appears as if the bad backup never happened and everything just goes back to normal. This is the best possible case.

2) The backup application didn't catch the exception, but "notices" when it next runs that there was a backup in process and isn't one now, but the backup state is still "dirty". Then, the backup application cleans up the old "dirty" backup state, and proceeds as normal. This is the middle of the road case.

3) The backup application didn't catch the exception, and when it goes to start the next backup, it gets an error from Exchange - "backup already in process". Then, because of that error, the backup aborts. This is the worst situation. To clear the "backup in process" flag requires a restart of the Microsoft Exchange Information Store service.

In any of these cases, the checkpoint depth is continuing to advance - and the log files are accumulating, but nothing is being committed to the database. This is a problem! Eventually, the Information Store will say "enough!" and dismount the storage group. That's right - it will dismount the entire storage group.

This is called "Checkpoint depth exhaustion". After 1,008 logfiles have been generated,  Exchange automatically shuts down the storage group. There is an implementation limit of 1,024 logfiles in a checkpoint, and this shutdown prevents the Exchange database from being corrupted.

After the storage group is dismounted, MSExchangeIS can simply be restarted and the storage group remounted (this will cause all of those transaction logfiles to be replayed - it may take quite awhile!).

This can also happen in very very high update situations (such as running many move mailbox operations all at once time).

Your best bet is to monitor for this. If you are using OpsMgr 2007, this is already one of the issues checked for. If you want to add this check manually, add the performance counter "Database ==> Instances\Log Generation Checkpoint Depth".

For more information on this issue, see KB 905801 and KB 819771.

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Posted by michael | 2 comment(s)
Filed under: ,

I hate to sound like a marketing goon, but I have to tell you about Xobni.

But I've been using Xobni for four days. I don't know how I ever got by without it. Comparatively speaking, I can now process my e-mail at lightspeed.

I don't know how YOU work with e-mail, but I spend a great deal of time referring back and dealing with the same people and pulling up prior e-mails, etc. etc. In order to do that, I've developed a pretty well-organized but moderately complex filing system based (of course) on Outlook folders.

With Xobni, I still file everything, but Xobni keeps it organized for me. If I click on an e-mail from someone, Xobni finds me all related threads, all related attachments, contact information for that party, and other stuff.

And yes, Xobni is "inbox" spelled backwards.

The website is the obvious place: http://www.xobni.com.

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Posted by michael | with no comments
Filed under:

Yesterday (Thursday, 2008-04-17) was the last day of the 2008 MVP Summit. Over one-half of the MVPs attending were from countries other than the United States. It was interesting to hear the dozens of languages being spoken around me this week.

Very little of what was covered during the Summit is not covered by NDA. But you can be assured that MVPs from every product that Microsoft produces were there learning about what is coming and providing "feedback" to the Microsoft product teams. Feedback can get quite .... loud .... at times. MVPs tend to be quite passionate about their products of interest and very protective of their user communities. Many features present in current releases of Exchange came at MVP "encouragement".

In fact, one feature added quite recently was only added because MVPs said "it has to be there". And in the future, we see that it will be expanded significantly to meet other needs. Yay MVPs!

The last presentation of the Summit was a keynote given by Steve Ballmer. During his hour on the stage, he discussed many things and took about 20 minutes of questions from the audience. Two of the items he said really struck me. The first: "Windows Vista...is a work in progress". The second was: "We can't go five years in between releasing versions of the operating system".

There are many things that can be read into those two statements. However, it is clear that Microsoft knows that Vista isn't where it should be yet, and it's likely that "Windows 7" won't take as long to get here as Vista did.

Changing directions...Steve took several questions about Groove vs. SharePoint as did Ray Ozzie, who spoke before Steve. There is a definite belief in those MVP communities that one of these collaboration tools has a limited lifetime (there is a lot of overlap). Who knows what it will be?

You can read more about Ballmer's speech in an article from the Seattle Times.

As always I met many people, shook many hands, got to see some old friends, and lifted a few while bonding with the other geeks around me. :-) A good time and very worthwhile.

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Posted by michael | with no comments
Filed under: , ,

Consider the format of an Exchange database.

It's flat. Very flat.

Take a mailbox. Basically, it is a table that contains pointers to other tables. What are those other tables? Well, your Inbox, Sent Items, Calendar, Tasks, Notes, etc....

Every folder is a table. Each table contains objects and pointers to other tables. An object may be a message, a task, a calendar entry - basically the low-level objects which Outlook manipulates.

Every view is an index - and an index is just a table of sorted pointers to objects (plus a small bit of meta-data).

So... each mailbox may have thousands of tables. A mailbox database may have millions of tables. Yes, you read that right: millions.

Why is it done that way? Easy answer! To allow the user maximum flexibility. The end-user is not limited in the number of views, not limited by what he can sort by, how many objects you can contain in a folder (although there certainly are performance issues associated large numbers of items), etc. etc. Exchange Server generates these views on the fly whenever Outlook requests them. Real-time. The end-user is not arbitrarily limited in any significant way.

Give a great pat on the back to Exchange!

How does Exchange do this? With ESE. Because ESE - the Exchange database engine - supports these features.

Now, consider Microsoft SQL. A great DB engine. However, it doesn't really generate tables on the fly. Or views on the fly. Or have a hierarchical mechanism for storing table trees. Or clean up after itself. All of these functions have latency in SQL...

So....what would have to happen? A complete redesign. Not just a simple "let's use a different backend". And, it's likely that a lot of features would be lost in any such translation.

Do I want Exchange in SQL? No. Do I think it would work well? Not with any SQL Server released to date.

In the future? Who knows? Maybe. But I don't know.

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Posted by michael | 2 comment(s)
Filed under:

I and 4,000 MVPs from around the world have accumulated in Seattle/Redmond this week. Where do we all go? To the bar, of course!

Last night, a few Exchange MVPs and the odd PowerShell MVP were sitting around discussing stuff I can't talk about here, when another MVP came to join us (who happens to work for a major storage vendor). Hey, we are open minded, no problem there!

Then, out of nowhere, he began to bash our beloved Exchange!

Interestingly enough, the basic complaint was this: Exchange degrades too gracefully. That is, you can configure Exchange in ways that are downright stupid, it will let you, and it will "sorta" run - until you finally hit that "straw that breaks the camel's back" and then it crashes.

Well, that's a valid complaint. Exchange goes to extreme lengths to allow your e-mail to go through - because e-mail is mission critical. This is what it should do, in my opinion.

And in doing this, if you want to put 1.2 million mailboxes on a single LUN of a SAN; well, it'll let you. Doesn't mean that it is the right thing to do! And it doesn't mean that Exchange is broken because it lets you do this!

There is nothing that excuses poor configuration. Whether it is Exchange itself, the storage subsystem, the network, whatever. The entire Exchange ecosystem, in large installations, must be planned and monitored.

Truthfully, in many small to medium installations, Exchange is so forgiving that you can do almost anything to it, and it'll continue to run. Most SMORGS don't have access to the technical expertise to do a proper design/rollout/support/etc. and with the hardware of today, that's ok. And Microsoft puts together packages of software to help this be OK (think Small Business Server and Essential Business Server).

But when an installation gets large than, oh, around 250 mailboxes - you best begin to take planning seriously. Planning, monitoring, and proper operations are a system administrator's ongoing responsibility.

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Posted by michael | with no comments

Even though I get pretty hot under-the-collar sometimes when Microsoft removes for-free functionality (Exchange backups on Windows Server 2008, for example), there are some great tools that Microsoft provides for-free that are just as good as their for-pay alternatives - and more tightly integrated with other Windows functionality. However, they are often not well known.

One of those is WDS - Windows Deployment Services. WDS provides easy/simpler ways to, well, Deploy Windows! (D'Oh!)

Part of WDS is ImageX. ImageX provides a free-and-easy way to create images of Windows desktops, including deployment images. One of the great features of ImageX is that you can easily load drivers, patches, service packs, etc - AFTER the image is created, without creating a brand new image. This is a great time saver. It also allows you to create and install "generic" images that have multiple HALs (Hardware Abstraction Layers) and storage subsystems (as long as the drivers are included in the base image!).

Check it out.

Windows Automated Installation Kit

ImageX Technical Reference

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Posted by michael | with no comments
Filed under: ,

Recently on a mailing list, several folks expressed some confusion about the scaling that Perfmon (Performance Monitor) does in its graphs and how it affects the data.

The answer is: it doesn't. The scaling is used only to make the graph fit in the Perfmon window. The values contained in the Last/Average/Minimum fields are not scaled, they are the actual counter values.

If you want to prove this to yourself in another way, click on "Change Graph Type" (ctrl-G for you keyboard folks like me) and select "Report". You'll see the same size of values reported in that list.

If you've never played with Perfmon before (or the entire "Reliability and Performance Monitor" that is part of Vista and Server 2008), you should learn how to use it. It can make debugging performance issues a lot easier.

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Posted by michael | with no comments

I wrote and supported two versions of a Catchall Mailbox script that worked with Exchange 2000 Server and Exchange Server 2003. Version 2 of it is available on this blog, at this link: Exchange 200x Catchall Script, Version 2.

However, the transport engine in Exchange Server 2007 is completely different. The old script interfaces no longer work.

However, as a sample, an Exchange engineer developed the Catchall for Exchange Server 2007 and posted it on CodePlex, calling it CatchAllAgent Exchange 2007 Transport Protocol Agent. If you need the functionality for Exchange Server 2007, that's where to get it!

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Posted by michael | 3 comment(s)
Filed under:

Sometimes, you need to step back and take a different look at a problem. When you are "in the weeds", you can't necessarily see the forest for the trees.

Exchange Server is a networking application. Now, before you say "duh!", think about it. If your network is having issues, this means that Exchange Server may have issues too.  However, since Exchange Server is always doing something on the network - it may appear that the problem is with Exchange.

I recently saw an issue that was caused by a problem with multiple default gateways.

A very common configuration on an Exchange Server is to have one NIC (Network Interface Card) be the "public" or "untrusted" interface. Through this NIC flows all traffic destined for the Internet. Then, the Exchange Server has another NIC that is the "private" or "trusted" interface that connects to the "internal" network. This NIC is generally for such things as backups, monitoring, etc.

When you configure a NIC with a fixed IP address, you are asked to supply three things: an IP address that uniquely identifies this server, a netmask that defines the size of the network that the IP address is a member of, and a default gateway to be used when an IP address isn't part of the same network as the server.

(Of course, there are many other properties you may configure as well - but these three are the ones that are required to put a computer "on the net".)

Notice the use of the default gateway: it is the destination where information not on the local network is sent. So, for a trivial example, if your local network is "DC-Server" and "Exchange-Server" and you want to send an e-mail to "TheEssentialExchange.com" then that information will go out your default gateway.

As you may have noticed, Windows allows you to enter multiple default gateways on a single NIC. This is for a very rarely used feature of IP called "dead gateway detection". In this case, if one default gateway should happen to go offline, Windows would automatically switch to another gateway. This is not a load-balancing technique or performance enhancing technique.

If you are looking for load-balancing or performance enhancing mechanisms, you should be looking at NICs that support "teaming" and/or switches that support EtherChannel.

Now, consider what happens when, in both the public and the private NIC, you enter a default gateway. Which one is really the default?

Huh. Think about it.

What is Windows to do? Flip a coin?

Well, that is about what happens. Windows becomes a non-deterministic router (this means that you cannot conclusively define which default gateway that Windows will always use). Windows will switch back-and-forth between the default gateways.

What does this mean to you, the Exchange/network administrator? It means that Exchange, on the public interface, will appear to come offline and online at unpredictable intervals. Why only on the public interface? Because there is a specific defined route to access the private network. Not on the public interface though.

So...don't do this. It isn't a good idea on Windows networks, in general, because of the non-deterministic nature of the routing that will occur. And specifically for Exchange, it is known to cause problems.

Here is one reference for you: Multiple default gateways detected.

And even though these KB articles were written for Windows NT 3.5, they still apply today: KB 157025 and KB 159168.

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Posted by michael | 2 comment(s)
Filed under: ,

So, "The Essential Exchange" is my front-end to my consulting business.

I admit that marketing is not my strong suit. So far, I'm supporting myself on writing and people using my services from the communities I help support. But this week I'm meeting with an honest-to-gosh client who doesn't know me.

What do I need? Besides a haircut? :-)

Business cards.

So, me and my s/o, neither of us being graphics people, put our heads together and came up with the below. We did conclude that we need to hire someone to help us develop a logo and other marketing material. Her company has used Logo-Mojo with good success, so I'm thinking that that is who I will use. If you have other suggestions please pass them along!

Until next time...

Posted by michael | 1 comment(s)
Filed under: ,

Next week, in Redmond/Seattle, Washington, those of us who Microsoft calls "Most Valuable Professionals" congregate in an annual event where Microsoft wines us, dines us, asks our opinions, and tells us theirs. Which, shockingly (not), often disagree! :-) If you aren't aware of the MVP program, take a look at the MVP Public Site. Generally, MVPs are folks who are very active in online communities (blogs, newsgroups, forums, mailing lists, etc.) who give accurate and dependable answers.

The event is called the MVP Summit. This is the event where MVPs get the opportunity to meet developers, program managers, support personnel, etc. etc. - pretty much everyone at every level in our various product teams.

For me, I'm an Exchange MVP, with a strong interest in Directory Services (that is, Active Directory) and SharePoint. Many of my clients are in the SBS and EBS space (Small Business Server and Essential Business Server), so I have strong interests there as well. So those products are where I'll be spending my time.

Unfortunately, pretty much all of the content next week is covered under NDA - Non-Disclosure Agreements. But you can bet that we'll hear quite a bit about what's coming next with Exchange, Active Directory, Windows client and server, and many other products.

So, if it seems abnormally quiet next week on the blogs you read - this is probably why!

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

Posted by michael | 2 comment(s)
Filed under:

As I mentioned in yesterday's post, Wrapping up SLDs and Exchange Server 2007, anytime after PrepareAD is run, it is too late to run the Domain Rename (rendom) tool.

As part of PrepareAD, the Exchange Server 2007 setup tool stamps the Active Directory with a number of server names in GUID and fully-qualified domain name (FQDN) formats. This is to enable Exchange Server 2007 to fulfill a much-requested feature: don't require WINS.

Unfortunately, from a Domain Rename perspective, this means that once PrepareAD has occurred, it's too late to go back.

At that time, the ONLY option for a domain rename is to remove ALL Exchange servers. That includes any Exchange 2000 Servers or Exchange Server 2003 servers which may be in the environment. The goal is to be able to remove the Organization container in Active Directory (which removing the last Exchange server in a forest will do). Having an updated schema is not an issue.

Once the Organization container is gone, a domain can be renamed and Exchange re-installed. But that's a very very dangerous option. Doing a full active directory migration to a new forest may be safer.

Consider yourself informed!

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!

In Single Level Domains and Exchange 2007 service pack 1, I commented on the fact that Microsoft was no longer going to support single label domains starting with Exchange Server 2007 service pack 1. Then, in More on Single Label Domains and Exchange 2007 I shared that Microsoft was going to change their position on that a bit.

Finally, last week, Microsoft has released the update that allows Exchange Server 2007 service pack 1 to be installed on single label domains. That guidance is available in the article Update that will allow you to install Exchange 2007 SP1 into a Single-label named domains is now available.

As I noted before:

1) Microsoft will support it - but will not change things that don't work with it
2) SLDs will NOT be supported by the next version of Exchange (code-named E14)
3) A domain rename tool will not be released for Exchange 2007
4) Outlook Anywhere/Autodiscover will not work properly in a SLD

Further, you should be aware that anytime after Exchange Server 2007 PrepareAD has occurred, is too late to run the Domain Rename Tool. More on that topic tomorrow.

Until next time...

As always, if there are items you would like me to talk about, please drop me a line and let me know!


Doing an in-place upgrade from Windows Server 2003 to Windows Server 2008, on a server where Exchange Server 2007 is installed is not supported.

And it WILL break things.

Your first clue that things will go awry is that you can't do an in-place upgrade when PowerShell is installed. So, if you ignore that clue (what? you need to be hit upside the head with a bat?) and remove PowerShell (can you imagine what removing PowerShell does to Exchange?) - the upgrade will proceed.

And guess what? After you are done, Exchange WON'T WORK.

You'll have a major recovery on your hands to get back to Windows Server 2003. In order to move from Windows Server 2003 to Windows Server 2008, you must use the move-mailbox method of migration.

Until next time...

If there are topics that you would like to see covered, please e-mail me or leave me a message in the forums.

Posted by michael | 4 comment(s)
More Posts Next page »