January 2010 - Posts

Note: This article is written for "modern" versions of the Windows operating system - that is, Windows Server 2008, Windows Server 2008 R2, Windows Vista, and Windows 7. For older versions of the Windows operating system, the concepts still apply, but some of the command line parameters for w32tm have changed.

Windows, especially in an Active Directory environment, requires "good" time. For this discussion, having "good" time means that all members of a domain are capable of synchronizing their clock to a domain controller. Domain controllers synchronize their clocks with the domain controller which holds the PDCe (Primary Domain Controller emulator) role in their Active Directory domain. PDCe's in child domains synchronize their clocks with the PDCe of the root domain of the Active Directory forest.

When Windows does not have good time, log file entries have incorrect timestamps, event logs have incorrect timestamps, database transaction logs have incorrect timestamps, etc. etc. When the time on a computer becomes too far off from that of a domain controller (more than five minutes above or below), the computer is no longer capable of acquiring Kerberos tickets - this means that a computer and/or a user will not be able to log in to the Active Directory domain, nor will they be able to access any resources on the Active Directory network.

This can happen to user workstations and to servers. Obviously, a server may effect more users than a single workstation; but that doesn't mean you should pay any less attention to your user workstations.

The "Windows Time" service is responsible for keeping a computer's clock synchronized. This service can be controlled and configured on each computer by a command line tool named w32tm. Modifying any parameters for the Windows Time service requires local administrator permissions (and if UAC is enabled on the computer, it also requires an elevated command prompt or PowerShell session).

Determining whether a computer can synchronize its clock is easy to test (this is irrespective of whether the correct time source is configured - just that the computer can synchronize to the configured time source). Open an elevated command prompt or PowerShell session, and then enter:

                w32tm /resync

Does it work? If so, then this computer can synchronize its clock with its configured time source. If the clock on the computer is off ("skewed" is the typical term used for this situation), then further analysis is required. If the time on the clock is off by an even number of hours, you should probably be looking at the timezone configured for the user or computer, not at the time synchronization sources.

If there are other computers whose time is skewed, then enter the same command on the other computers. The command should work there too.

If the resync commands work, but the computers are getting the wrong time, you need to begin analyzing the configuration for the Windows Time service. From your shell or PowerShell session, enter:

                w32tm /query /source

This allows you to determine where the particular computer is getting its time. There are a number of possible responses. These include:

Local CMOS Clock

In this case, the computer is using the hardware clock on the computer as its time source. If you are using VMware, this means that the virtual machine is synchronizing to the VMware host.

Free Running System Clock

In this case, the computer is not using any external source, but depending on the time tick generated by the System Idle Process running on the computer. This value will generate a skewed time more quickly than any other.

a hostname of a domain controller in the Active Directory forest

In this case, the computer is using a domain controller as either an NTP server or as the time source via Active Directory. To determine which, see "/query /configuration", discussed later.

a hostname of a computer running a NTP server

In this case, the computer is using a non-Active Directory server running an NTP server as its time source.

VM IC Time Synchronization Provider

In this case, the computer is using Hyper-V virtualization services as its time source.

Best practices from Microsoft recommend that you never use virtualization services (regardless of your hypervisor provider) as a time source for domain-joined computer; instead, you should depend on typical Active Directory synchronization methods.

VMware recommends that, for domain-joined computers, you install an NTP server on the VMware host and you have the computers synchronize to that NTP server.

*** Edit on June 25, 2010 - A VMware employee contacted me at the end of May suggesting that the above line was not accurate. Indeed, VMware updated their documentation (the linked VMware KB 1318 article below) in March of 2010. Now, for most intents and purposes, the VMware recommendations match the Microsoft recommendations.

In my mind, you are better off starting with the Microsoft recommendations and then go from there.

Here are references to the above comments and best practices:

Virtual Domain Controllers and Time Synchronisation
Considerations when hosting Active Directory domain controller in virtual hosting environments
Deployment Considerations for Virtualized Domain Controllers
VMware KB: Timekeeping best practices for Windows

Now, if the initial resync command doesn’t work – that particular failure reason is what you need to figure out. The first thing I always check is the firewall configuration. By default, time synchronization requires that that a computer be capable of sending a UDP request to port 123 on the NTP server (and receiving the response). NTP servers also listen on port 123 for TCP requests. The Windows Advanced Firewall in the modern Windows operating systems will automatically have an entry opened for time synchronization on UDP port 123 to your domain controllers. However, if you are configuring your PDC emulator server, you also need to ensure that the external firewall also allows that request. If you have non-domain-joined computers, then you may need to globally allow port 123 requests in your firewall.

The command below will tell you the time source for a particular computer:

                w32tm /query /configuration

You are initially most interested in the value of the Type variable which is displayed. There are a number of possible responses. These include:

NTP - the external time source is the NTP server(s) specified by the NtpServer variable
NT5DS - the external time source is the domain hierarchy (that is, time synchronization originates from a domain controller)
NoSync - there is no external time source
AllSync - the computer should use both the domain hierarchy and the manually specified NTP server(s) as external time sources

There may be multiple external NTP servers listed in the NtpServer variable.

To properly set up a time source synchronization hierarchy for your domain, you need to begin by locating the domain controller which holds the PDC emulator FSMO role (obviously, if you have a single domain controller, such as is normally the case in SBS 2008, this process can be shortcut). To determine the holders of the FSMO roles, at that earlier-opened command prompt or PowerShell session, enter:

                netdom query fsmo

Next, on the domain controller which is revealed to hold the PDC emulator role, you should do something like this:

                w32tm /config /manualpeerlist:pool.ntp.org /syncfromflags:manual
                w32tm /config /update
                net stop w32time
                net start w32time
                w32tm /resync /rediscover

This ensures that this particular domain controller will attempt to synchronize with an external source providing known good time. pool.ntp.org is a common source. Windows computers come configured by default to use time.windows.com, which sometimes works and sometimes doesn't.

For all other domain-joined computers, the appropriate configuration is:

                w32tm /config /syncfromflags:domhier
                w32tm /config /update
                net stop w32time
                net start w32time
                W32tm /resync /rediscover

That really should take care of it. /syncfromflags:domhier is the default for domain-joined workstations and should be for all DCs except for the one in the root domain holding the PDCe role.

When a computer is properly synchronizing from an external source (after the Windows Time service restarts or becomes capable of synchronizing after some interval where it can't synchronize), the following entry is made to the System Event Log:

Log Name:      System
Source:        Microsoft-Windows-Time-Service
Date:          1/24/2010 1:01:27 AM
Event ID:      35
Task Category: None
Level:         Information
Keywords:     
User:          LOCAL SERVICE
Computer:      W2008R2-DC
Description:
The time service is now synchronizing the system time with the time source pool.ntp.org (ntp.m|0x0|0.0.0.0:123->69.26.112.120:123).

If the time source is a DC, the DC will be named and its IP address listed, just as if it were an external source.

Until next time...

If there are things you would like to see written about, please let me know.

I recently had a very confusing issue arise at one of my Exchange 2007 clients and I decided to share it with you. At this particular company, an Active Directory site is reserved for Exchange, and there are two domain controllers (both global catalog servers) in that AD site. The front-end is two CAS (CAS1 and CAS2) load-balanced by an ISA enterprise array with a CCR backend.

The week before, we had replaced all of the domain controllers in the forest with Windows Server 2008 R2 domain controllers, and bumped both the domain functional level and the forest functional level to Server 2008 R2 (we are going to enable the AD Recycle Bin). The new DCs replaced the old DCs and kept the original IP addresses.

That's the setup.

An onsite technician was applying patches late one night (good for him!). Unfortunately, he patched and rebooted both of the Exchange AD site DCs at the same time (bad for him!). As you may already know - that makes Exchange very unhappy. System Center Operations Manager is also running in the environment and it immediately started to generate alerts about the missing domain controllers.

Sidebar: In Exchange 2003 and above, Exchange executes an Active Directory Topology discovery every 15 minutes. The specifics vary between versions of Exchange, but suffice it to say that, within 15 minutes, Exchange will find another DC/GC set (if they exist). In that case, your best bet is just to wait out that 15 minutes.

The technician reacted to the alerts from OpsMgr by rebooting the Client Access Servers. They both found out-of-site DCs and began working.

Then, the fun began. When the in-site DCs came back online (just a few minutes later), CAS1 reassociated with the in-site DCs and reset its secure channel to one of the in-site DCs. CAS2 did not.

The symptom of this is that all users connected to CAS1 through ISA were fine. However, the users that ISA connected to CAS2 were redirected through the same URL that they had already used - and CAS-to-CAS proxying did not work, so they couldn't access any Exchange services - OWA, Exchange, anything. Quick workaround: remove CAS2 from the webfarm and RPC publishing in ISA so everything was routed through CAS1. However, redundancy is now lost.

Why this problem happened - I don't know. The NetLogon service is responsible for maintaining the AD site a computer identifies itself with and maintaining the secure channel to a proper DC. However, for CAS2, NetLogon refused to reassociate to an in-site DC.

NetLogon bases site affinity on DNS. Both servers, CAS1 and CAS2, were configured identically for DNS. NetLogon uses a Windows API call named DsGetSiteName. In Windows Server 2008 and Windows Server 2008 R2 (and in Windows 7), you can use the nltest.exe utility to check this value. To wit:

PS C:\> nltest.exe /dsgetsite
Default-First-Site-Name
The command completed successfully
PS C:\>

Sidebar: nltest is available for Windows Server 2003 and Windows Server 2003 R2 as well, you just have to download and install the Windows Support Tools.

NetLogon does its check-and-reset once an hour, and upon startup. Once you know that, it should be easy to just restart the NetLogon service, right? Well, that didn't make any difference.

So, we have the capability of forcing a particular secure channel for a server, and this also will set its AD site. To wit:

PS C:\> netdom.exe reset cas2 /server:DsEx2
The secure channel from CAS2 to DOMAIN was reset.
The command completed successfully.
PS C:\>

Note: nltest.exe has this functionality too, but netdom.exe has been around longer and I was more familiar with its parameters. See the SC_RESET parameter to nltest.

The AD site is updated, the secure channel is updated, and everything looks great. I declare success, put the server back into ISA, and move on.

An hour later, the client calls and says it is broken again.

Well, he's right. The AD site has flipped back again and CAS2 is thus not operating properly. Obviously this has happened because NetLogon did its cycle.

C-r-a-p.

OK. Now its time to buckle down. AD sites are based on DNS. We know that. So, I ran dcdiag on all the servers. replmon on all the servers. Everything is clean.

But then visually examine the DC locator records in DNS - and... I find an extra one.

During the process of standing up all the new DCs, and configuring the new DCs with old permanent IP addresses, the OLD DCs ended up with the temporary IP addresses of the new DCs. Then, the old DCs were demoted.

All of the DCs but one cleaned up after themselves. The extra locator record was one of those DCs, and shockingly, now has stale DNS.

The fix? Remove the stale DC locator record. Reset the secure channel again, just to ensure it gets to the right place.

And Voila! It's fixed.

If you've ever been to one of my installation seminars or read many of my articles, I talk about the importance of DNS in both Active Directory and Exchange Server. Here, yet again, is another example of that. Sometimes, you just have to take a look in the right place to find the problem.

Until next time...

If there are things you would like to see written about, please let me know.

As I've noted in several previous blog entries (such as here and here), installing Exchange on a domain controller and/or a global catalog server is not a best practice. However, if you are running SBS (Small Business Server) or EBS (Essential Business Server) or if you only have a single server in your environment - you may not/don't have much choice.

Given that you or your company may have no choice in the decision, it still may come as a disappointment (disgust?) that it takes so long to reboot your Exchange server.

This typically happens because of two primary reasons:

  • When Exchange is installed on a DC/GC, that Exchange server will refer to no other DC/GCs in the Active Directory, and
  • When a shutdown or reboot request is received, it isn't possible for Exchange to terminate prior to Active Directory shutting down.

Now, you may think "poor poor Exchange, do what _I_ want anyway!" Well, in Exchange's defense, that may be harder than you think. Consider a common scenario that may occur:

  • A VSS backup is running against your server and it's just entered the Freeze stage against all writers
  • Exchange is running
  • RPC/HTTP is up
  • OWA is up
  • SQL is running\
  • ...a shutdown request comes in

What is the right order to shut things down in that ensure everything gets shut down before AD starts shutting down?

The answer is - can't be done!

Exchange and Active Directory have no mechanism for terminating the right things in the right order. So, it is up to a human brain to help them out.

I suggest you create a directory on your combination Exchange / Active Directory server named c:\scripts. Within that directory, create a file named shutdown.cmd. In that file, place the following commands:

echo %DATE% %TIME% Shutting Down Services >>c:\scripts\shutdown.txt
net stop msexchangeadtopology /y
echo %DATE% %TIME% Shut Down MSExchangeADTopology >>c:\scripts\shutdown.txt
net stop msftesql-exchange /y
echo %DATE% %TIME% Shut Down MSFteSQL-Exchange >>c:\scripts\shutdown.txt
net stop msexchangeis /y
echo %DATE% %TIME% Shut Down MSExchangeIS >>c:\scripts\shutdown.txt
net stop msexchangesa /y
echo %DATE% %TIME% Shut down MSExchangeSA >>c:\scripts\shutdown.txt
net stop iisadmin /y
echo %DATE% %TIME% Shut down IISAdmin >>c:\scripts\shutdown.txt
echo %DATE% %TIME% Shut down services script complete >>c:\scripts\shutdown.txt

Note that the echo statements are completely optional. They are simply present to allow you to record the sequence of events that does occur during a shutdown.

Once you have created this file, open Administrative Tools -> Group Policy Management.

Expand the domains node, then expand the node for your domain, and then expand the Group Policy Objects node.

Under the GPO node, right click on the Default Domain Controllers Policy and select Edit...

Expand Computer Configuration -> Policies -> Windows Settings and then click on Scripts.

In the right pane, double click on Shutdown, then click on Add in the dialog that opens. Browse to the shutdown.cmd that you created earlier and click OK.

Now, click OK until you are back to the group policy main window and close it and then close the Group Policy Management window.

If you have a single DC, you are done. Otherwise, wait for 15-20 minutes to allow your modified group policy to replicate to other DCs in your Active Directory.

Now, each time your DC that has Exchange Server installed on it reboots (or shuts down), it will execute the above script. This will reduce the required reboot time 50% - 75%.

Enjoy!

Until next time...

If there are things you would like to see written about, please let me know.

For the third time, I'll be speaking at the upcoming The Experts Conference, sponsored by Quest.

I'll be discussing using Exchange Web Services (EWS) from PowerShell -- and for some reason, my abstract isn't yet posted on the conference website! I need to get that corrected.

TEC'2010 is being held in Los Angeles, CA this year, from April 25 - 28. The Exchange track will have it's strongest expert content ever, with MVPs and Microsoft personnel from all over the United States and Europe. Of course, as always, TEC'2010 will have a huge Directory Services and Identity Management track and is introducing a SharePoint track.

See the official TEC'2010 website for more information.

I hope you'll come join me and many other people at TEC'2010. It's a great time with lots of talk-time with some of the best folks in the business.

Until next time...

If there are things you would like to see written about, please let me know

SBS 2008 has IIS logging enabled by default. For most websites on an SBS server, this probably isn't an issue.

However, the WSUS Administration website can generate very high traffic. On my client's servers, I've seen 5 GB generated in just a couple of months. One person reported as much as 7.5 GB generated within a month.

Unless you need this logging for some debugging purpose, you can easily disable the logging. Sure, there are command line ways to do it, but in this case, using the GUI is pretty easy.

Open the IIS Manager and expand both the server and the Sites nodes in the Connections pane. See the figure below.

Next, click on the WSUS Administration website, then locate the IIS feature named Logging in the main pane. Double-click on it (or single click and select "Open Feature" from the Action pane).

Finally, click Disable, red-circled in the figure below. That's all it takes!

If you should ever need to re-enable logging, you can return to this same window. Once disabled, the "Disable" action changes to "Enable".

Disabling WSUS Logging

 

Until next time...

If there are things you would like to see written about, please let me know

Posted by michael | 2 comment(s)
Filed under: , ,