December 2007 - Posts
In this article I discussed a bug with .NET 2.0 that could cause that Exchange Server 2007 to consume too much memory. Now, there is a somewhat related patch that often impacts Exchange and SQL servers. KB 938486 contains a patch against Windows Server 2003 sp1 or sp2 that corrects a situation where a server appears to hang when a process running on the server makes a large memory allocation request. You can either call PSS to request the patch or request the patch online at the link available in the KB article.
Both are recommended for your Exchange Server 2007 servers.
You may remember the issues with Windows Server 2003 sp1 + the scalable network pack (included as part of sp2) and how they negatively impacted Exchange. This issue was documented here and especially affected NICs using the TCP Offload Engine (TOE) capabilities of the NIC (enabled by default in sp2), very specifically using a feature called TCP Chimney Offloading. Broadcomm has finally released new drivers to correct those problems. They are available here. The various vendors using the Broadcomm NICs will likely move the new drivers into their packages, but you can get them more quickly from Broadcomm directly.
Another fine patch to test and install!
I got in today and worked on the blog itself. I finally made the various search buttons work properly, on both the main site page and on the blogs pages. I also rearranged the content on the blogs pages and eliminated some sections that I thought were superfluous.
I also created a couple of forums, just for grins and giggles. I notice that the number of members are growing! That's great.
Thanks for you patronage this year and Merry Christmas! Lots more content and fun coming in 2008...
In the December 13, 2007 EMO (Exchange Messaging Outlook) e-newsletter, I had an article titled "'Tis The Season To Host - Or Not?" regarding hosting your Exchange services.
You can read this article at http://www.slipstick.com/emo/index.htm.
As part of that article, I made the point that switching from in-house Exchange to hosted Exchange was a more difficult choice to make (in my opinion), than the choice of going for hosted Exchange from a non-Exchange e-mail system.
I've gotten quite used to Exchange and its tight integration to Outlook. There are other systems that provide similar services and do so quite well. But I'm partial to Exchange.
There are several reasons why a company running Exchange in-house may decide to outsource their Exchange to another company. Among these:
- Cost - maintaining an Exchange system is not cheap; especially when you factor in the costs of expertise, training, backups, and bandwidth
- Return on Investment - if your company was never able to use all the capabilities of Exchange before moving to hosted services (because of lack of resources, lack of expertise, lack of add-on software) then moving to hosted services may be much cheaper than bringing those other factors in-house (this is somewhat the reverse-side of cost)
- Control - you may find it difficult to make your Exchange environment behave in the way you desire
Hosting may never make sense for businesses on the "upper side" of "small and medium business". But I would suggest that up to around 100 mailboxes, hosting is probably always the better choice. And, if your hosted provider offers dedicated servers, this probably scales well to around 250 mailboxes.
If you are within that size, why wouldn't you want to host? That's a different, but very important question. I would assert that the answer revolves around:
- 1. Loss of control
- 2. Loss of control
- 3. Loss of control
- 4. Minor loss of features
In a hosted environment, you can only do what the hosting company allows you to do. When moving from an in-house solution to a hosted solution, you may find the new restrictions onerous and not acceptable.
When evaluating hosting companies, and you are moving from an in-house solution, there are some difficult questions you should ask. Below, I list quite a number of them that you may miss, even after you've done end-user due diligence.
1. Does the hosting environment allow multiple hosting clients to have contacts with the same e-mail address? (This question can be restated as: how does the hosting software deal with SMTP address collisions?)
2. Does the hosting environment allow you to share SMTP address space, either as a master or as a slave environment, with a hosted SMTP domain? (This question can be restated as: can you do a step-wise migration, or do you have to migrate all mailboxes at once?)
3. Does the hosting environment support Deleted Item Retention? For how long? Does their deployment environment set the DumpsterAlwaysOn registry key for Outlook? (This question can be restated as: what happens when someone deletes something they didn't mean to!)
4. Does the hosting environment support Deleted Mailbox Retention? For how long? (Restatement: can I easily restore the mailbox if my company administrator deletes a mailbox by mistake?)
5. Does the hosting company do backups? How often and how long do they retain them? Can they do single mailbox recovery? (Restatement: if the hosting company has a "disaster" can they recover my mailboxes? Also, if the timeframe for Deleted Mailbox Retention has expired, can I recover the company president's mailbox from last month?)
6. Does the hosting environment support journaling? What are the data-retention options for the journal mailbox? Can I have an external interface to a journal solution?
7. Does the hosting environment support catchall mailboxes? (This is simple a feature that some companies use. Others don't.)
8. Does the hosting environment have a decent anti-spam solution? (More than the Outlook Junk Mail Filter!) Does the anti-spam solution support individual mailbox quarantines? If there is a false-positive, how can you get your file/message delivered?
9. Does the hosting environment allow you to truly white-label their services? (Restatement: can you have a custom OWA URL? Can you have a custom RPC/HTTP URL? When you connect to an SMTP virtual server, does it say YOUR domain name?)
10. Does the hosting environment allow you to have custom OWA themes? Does it support OWA segmentation
11. Does the hosting environment support SPF and/or Sender-ID incoming? Does it require it outgoing? Can you decide or are you limited to their default?
12. Does the hosting environment support SSL for OWA? TLS for SMTP? Form-based authentication for OWA? Two-factor authentication for OWA and for Outlook?
13. Does the hosting environment allow you to specify on a per-user basis who gets EAS (ActiveSync)? Blackberry services? Goodlink services?
14. Does the hosting environment allow you to create custom address lists?
15. Does the hosting environment allow you to force an Offline Address Book (OAB) update?
16. How is disk space aggregated? Is each mailbox billed separately? Is the company/domain aggregated together? Can different mailboxes have different default allocations? Can you manage the limits? Can you get disk space reports? Can you create/manage a "Mailbox Manager" policy for your domain?
17. What are the hard limits on mailboxes sizes?
18. Does the hosting environment run a gateway anti-virus solution? An information store anti-virus solution? A file-based anti-virus solution? If there is a false-positive, how can you get your file/message delivered?
19. Does the hosting environment support "Send As" permissions and "Send On Behalf Of" permissions? Can you manage this yourself?
20. Does the hosting environment support LDAP access to your address books?
21. Do you have access to SMTP log files? Do you have access to message tracking log files?
22. What is the maximum incoming message size? The maximum outgoing message size? Can you adjust it?
23. What is the maximum number of message recipients? Can you adjust it?
24. Does the hosting environment support public folders? How many? How big? Can you mail-enable public folders?
25. Does the hosting environment support an interface to SharePoint services?
26. Does the hosting environment allow for external SMTP relays by IP address? What about by authorized users?
27. Does the hosting environment allow for POP-3 or IMAP users to access Exchange mailboxes?
28. Does the hosting company offer a network Service Level Agreement (SLA)? Does the hosting company offer an Exchange SLA? Does the SLA have any teeth?
This list is far from being all-inclusive. However, it gives you a flavor for topics that you may have missed when you did your first evaluation.
Also, make sure you know how a hosting company may meet your needs. All Exchange hosting companies should have mailbox control panels in this day and age, but many of the items in the list above may not be manageable via the control panel. If you are told that you have to open a support ticket to get things done - find out the guaranteed turn-around and the escalation policy for issues.
Good luck and happy hosting!
A new blog on technet appeared a few days ago, from the Hot Fix and KB content team.
The intent is to publish weekly all of the KB articles and hotfixes that were released. If it happens, and is updated regularly, it will be a great resource!
The blog is to include hotfixes and KBs for Windows Server, Windows Client, Exchange Server, IIS, SQL Server, etc.
Take a gander: http://blogs.technet.com/hot/default.aspx
Do you know what a flat domain is? Do you know what a single-level domain is? You may not even be aware that Active Directory supported single-level domains.
If you do - then you've probably run into problems with it in the past. Now there are more problems.
A flat or single-level domain is one that only has a single piece to the Active Directory domain name - often people make it similar to their NetBIOS name.
For example, my Active Directory domain is "essential.local". A single-level domain would be just "essential".
Well, Exchange 2007 sp1 (and later) will no longer support single-level domains.
This is documented in the README for the service pack (but who reads them?).
DON'T INSTALL EXCHANGE 2007 IF YOU HAVE A SINGLE-LEVEL DOMAIN.
Before you install Exchange 2007 you have the option of renaming your domain (Exchange 2003 supports rendom). After you've installed Exchange 2007 you can't rename your domain. And you can't install service pack 1 (or any later service packs).
I'm guessing this is a budding project of some of those crazy Exchange kids at Microsoft, but you have to check out http://www.lolexchange.com/ !
Not much content, at least not yet, but if you like cats - you'll get a laugh or three!
Wow. I've never quite seen such a service pack release. Six separate files, for nineteen products - all released at once.
It looks like the only Office desktop product that didn't get a service pack was Communicator. Two Office Servers didn't get a service pack, PerformancePoint Server and Portfolio Server - if I were to guess, the 1/2 dozen folks that have installed those packages are big enough to get patches whenever they want.
The office.microsoft.com website was updated early this morning (AM Eastern Time) with the download information and links; but the files weren't available for download until about 11:15 am. Probably just due to replication going on throughout the Microsoft web-i-verse.
Interestingly, the service packs don't offer an uninstall feature. I guess that means you have to remove the base product to remove the service pack - so test test test!
(That being said, I installed all of these in my test environments this afternoon - I've got a customer who requires some of these updates, so I'll be rolling them out in the "real world" tomorrow!)
Quick gut feel: Outlook and Word are noticeably faster. Quite a few bugs are fixed in Project Professional and Project Server. The others -I'm not really a power user, so I can't have a reasonable comment.
Without further ado, here they are, all together:
Office Suite Service Pack (Access, Excel, Groove, Infopath, OneNote, Outlook, PowerPoint, Publisher, and Word)
Office Project Service Pack (Project Standard and Project Professional):
Office Servers Service Pack (Forms Server, Groove Server, Project Server, and SharePoint Server):
Windows SharePoint Services 3.0 Service Pack (WSS 3.0 only):
Office Visio Service Pack (Visio Standard and Visio Professional):
and last but certainly not least,
Office SharePoint Designer Service Pack (SharePoint Designer only):
Microsoft is practically flooding us with service packs right now (similarly to the flooding that is going on in Washington state and Oregon this week!).
As I earlier noted, Exchange 2007 service pack 1 is now released, and available here. For information about what is new in Exchange 2007 service pack 1, check out the posting here.
It has also been announced this week here that Office 2007 service pack 1 will be released next week (December 11). Not only does this service pack include updates to the familiar desktop suite of products, but it will also include important patches for the Office server products, including SharePoint Server 2007 (MOSS) and Project Server 2007 (MOPS).
We have also seen these recent releases:
.NET Framework 2.0 service pack 1, available here
(that is the link to the x86 patch, but the x64 and IA64 patches have links available there too)
.NET Framework 3.0 service pack 1, available here
.NET Framework 3.5, available here
(that link is for the bootstrapper, if you want the full installation,it is available here
The service packs for 2.0 and 3.0 are prerequisites to installing the 3.5 framework.
And while there are always new hot fixes coming out, I had one cross my desk (laptop?) today that has caught me more than once! I'll be pushing this out very soon. KB 944704 (When you try to shut down or to restart a Windows Server 2003-based computer that has USB devices connected, the computer stops responding and displays a black screen).
The only workaround to the above is to power-cycle the box! Not very practical if you are a few thousand miles away...
By the way, I was recently informed that Dell suggests a workaround of disabling USB booting as a workaround. I've not tested it, but it comes from a generally reliable source. :-) No clue as to whether a similar workaround may be successful for an HP/Compaq server.
In this article, I discussed one of the type of anti-virus available for Exchange Servers: information store antivirus. These two posts are follow-ons to an article I wrote for EMO. That article is discussed here.
Today, I'm going to talk about file-level antivirus on Exchange Servers.
If you have a multi-leveled antivirus solution at your Internet gateway, on all desktops in your organization, and you strongly control the introduction of external data into your workplace, some experts will say that file-level virus scanners on your servers are not required. I would respectfully disagree given that I've seen servers become compromised in those environments.
The purpose behind file-level antivirus programs is basically to scan everything but the information store itself (where the store includes the transactions logs, the EDB file and the STM file). File-level scanners have two modes of operation:
Most virus programs provide a mechanism for performing a virus scan on a specified schedule or whenever a particular event occurs. This event may be a manual request, on a reboot, when a new set of virus definitions is being loaded or practically any other occurrence supported by that antivirus program. On-demand scanning has the particular attribute of occurring after the fact. Any infestation that an on-demand scan finds has already had a chance to spread.
Many modern antivirus programs interface with the Windows operating system in such a way that when a file is created, opened, written to, closed, or modified, the file is scanned for virus infestations. While modern computers can typically handle this load, this process can be extremely processor and memory intensive. On file servers, it is common to see that on-access scanning is consuming over one-third of the computer processor resource.
Both on-demand and on-access scanning have a significant negative: while they are scanning a file, they lock the file so that no other program can access that file. There are multiple reasons for this, but this is primarily so that no other program can become infected by transferring data that is present in the file.
If the file the antivirus program is scanning (and thus has locked from any other access) is an Exchange database or a temporary file that Exchange is expecting to have access to, real problems can occur including data corruption, program failure, and even worse - information-store corruption. Just imagine how horrible it would be if an antivirus program scanned one of your Exchange information stores and found a virus (or got a false positive) before the Exchange information-store antivirus program found it - and therefore quarantined (or deleted!) your entire information store. While this is quite unlikely to happen with a mounted information store, it is extremely likely to occur with transaction log files. This situation can be even worse - imagine your Exchange Server undergoing a database recovery and it failing - because many of your transaction logs are sitting in your antivirus package's quarantine!
Scanning the Exchange related directories is a recipe for disaster.
In order to prevent this, you should exclude a number of Exchange directories from the file-level scanning process. These include (for Exchange Serrver 2003):
- All Exchange databases and log files on all volumes (typically the \Program Files\Exchsrvr\MDBDATA folder).
- All Exchange MTA files (typically the \Program Files\Exchsrvr\MTADATA folder).
- Message tracking log files (typically the \Program Files\Exchsrvr\<servername>.log folder).
- All virtual SMTP server folders (typically the \Program Files\Exchsrvr\Mailroot folder).
- The folder used for the storage of temporary files (typically the \Program Files\Exchsrvr\MDBDATA folder).
- The folder used by the Site Replication Service (typically the \Program Files\Exchsrvr\SrsData folder).
- The folder containing the .chk file (typically the \Program Files\Exchsrvr\MDBDATA folder).
- Any ExIFS folder (typically the M: drive, but not present by default in Exchange Server 2003).
- The %SystemRoot%\System32\InetSrv folder.
- Any other folder specified by your antivirus product.
Many organizations simply exclude all Exchsrvr folders and subdirectories, plus the Inetsrv folder mentioned next to last above. Most antivirus applications also have a folder where they create/process temporary files for Exchange that also need to be excluded. You will need to refer to the documentation for your specific antivirus application for more information.
Also, specifically in the case of the MDBDATA folder, you notice that it is in the list multiple times. This is due to the fact that several types of files may be stored there and that they may be moved from the default locations-and if they are moved, those non-default locations will also need exclusion from the scanning.
Be aware that these folders should be excluded from both on-demand and on-access file scanning. If you do not, data corruption can result.
Microsoft's general recommendations on antivirus software and Exchange Server 2003 can be found in Microsoft KB 823166 (Overview of Exchange Server 2003 and antivirus software). The situation is a bit more complicated for Exchange Server 2007. The exclusions for Exchange Server 2007 are documented in this TechNet article (File-Level Antivirus Scanning on Exchange 2007).
While these restrictions may seem somewhat onerous, they are not unique to Exchange Server. There are also specific recommendations concerning antivirus and other types of servers:
- Domain Controllers, KB 822158 (Virus scanning recommendations for computers that are running Windows Server 2003, Windows 2000, or Windows XP)
- SQL Server, KB 309422 (Guidelines for choosing antivirus software to run on the computers that are running SQL Server)
- Clustered Servers, KB 250355 (Antivirus Software May Cause Problems with Cluster Services)
This is a common issue across the Microsoft product line.
In my recent EMO article, I referred to the fact that there are two different kinds of anti-virus that may execute on an Exchange Server: file-level scanning and information store scanning. In this article, I write more about information store scanning.
While written specifically with Exchange Server 2003 in mind, the information applies to Exchange Server 2007 as well.
Information-store antivirus programs break down into three categories:
MAPI ScannersMessaging Application Programming Interface (MAPI) scanners were the first generation of information-store scanners available for Exchange. MAPI scanners had a number of significant disadvantages that led to their replacement, including: speed (they were slow – they had to log into each individual mailbox in order to scan messages), speed (they were so slow that it was possible for a user to read an infected message before it was scanned), speed (they don’t understand single-instance storage so one message could be scanned many times), and finally that they couldn’t scan outgoing messages, only incoming messages. While there are still MAPI scanners being sold, none of the major brands of Exchange Antivirus are MAPI scanners.ESE-Based Scanners
The Extensible Storage Engine (ESE) is the database engine that Exchange uses for its information stores. ESE-Based scanners directly access the ESE databases and interpose their message scanning functionality between Exchange and the database engine by installing a piece of software technology known as either a shim or thunking layer between Exchange and the database engine. ESE-Based scanners are not officially supported by Microsoft, even though they market such a solution. ESE-Based scanners can claim speeds that no other scanner type can match.VSAPI Scanners Virus Scanning Application Programming Interface (VSAPI) scanning was first possible in Exchange 5.5 service pack 3. VSAPI is currently up to version 2.5, which was introduced in the original release of Exchange 2003 Server. The current version of VSAPI corrects pretty much every problem that was experienced with MAPI scanners. VSAPI understands single instance store, it will not place a message into a user’s mailbox until the message has been scanned, it can scan incoming as well as outgoing messages, plus other features not previously discussed. All major Exchange Antivirus solutions are now using VSAPI. A few have not upgraded to VSAPI 2.5, so you should check for this compliance prior to making a decision as to your antivirus solution. VSAPI is supported in Exchange 2007, but de-emphasized. Microsoft recommends running a transport scanner on the hub server.
Information-store antivirus solutions also often package other add-ons to Exchange (such as disclaimers, anti-spam, attachment filtering, etc.), but those are beyond the scope of this article.A common antivirus-related question is if you have an email gateway that feeds your Exchange server which already does virus scanning, should there also be a information-store antivirus solution on the Exchange server? The answer is a qualified yes.A multi-layered antivirus defense is your best protection against viruses. However, best practices dictate that each layer should be from a different vendor. This is due to an unfortunate fact that all vendors do not prevent or scan against all potential harmful viruses, and that each vendor will release updates at different times for different threats. Having one vendor’s solution scan the same data multiple times is of little value. It is possible that the added cost of having a multi-layered defense from multiple vendors may make it infeasible for some companies.However, and let me say this very strongly, having an email server that does not have either a gateway-based antivirus or an information store based antivirus in this day and age is nothing but pure negligence. Some studies have concluded that your servers will very likely be infected within a week.