Cleaning Up Metadata and SRV Records

In the last video, I showed how to clean up the metadata for a Domain Controller that’s no longer able to participate in your AD forest.  In this video, which follows from the previous post, I want to show one additional bit of cleanup that will be necessary.  In the video below, I briefly describe the need to manually cleanup SRV records as part of the overall metadata cleanup effort.  After the video, I provide a deeper overview of SRV records and how to read one:

SRV Records and DNS

Most of you probably know how must Active Directory depends on DNS.  In fact, AD can’t function without it because DNS is how Active Directory services are located within an environment.

To allow Active Directory services to be located, DNS uses a particular type of record known as an SRV record.  SRV records are not specific to Active Directory or even Microsoft.  These records are used by servers to advertise the types of services they have available for clients to use.  For example, Active Directory provides LDAP, Kerberos and Global Catalog services (among others) and advertises this through DNS SRV records.

These records, located in the forward lookup zone for your Active Directory domain, come from a file called NETLOGON.DNS.  On a domain controller, this file is populated during the DCPROMO process.  NETLOGON.DNS is located at the following location on your domain controller:

%systemroot%\Windows\System32\Config\netlogon.dns

If DNS is configured to accept dynamic updates, these records will also be added to DNS.  However, if DNS doesn’t accept dynamic update it will be necessary to send this file to your DNS team so they can enter the files directly.

Understanding SRV Records

When you first look at an SRV record, it can be a bit confusing.  To help with that, I’ve provided the following walk-through of a few SRV records that are used by DCs to advertise services.

First, let’s look at the list of SRV records available for W2K8-DC1.W2K8Forest.com (note that some of these are wrapped onto multiple lines here, but they should be viewed as one line):

_ldap._tcp.W2K8Forest.com. 600 IN SRV 0 100 389 W2K8-DC1.W2K8Forest.com.
_ldap._tcp.pdc._msdcs.W2K8Forest.com. 600 IN SRV 0 100 389 W2K8-DC1.W2K8Forest.com.
_ldap._tcp.467fe6a7-28e3-475f-9739-b11684cc1bf3.domains._msdcs.W2K8Forest.com. 600 IN SRV 0 100 389 W2K8-DC1.W2K8Forest.com.
61657dc6-742e-4fae-98cb-60ddb9406c41._msdcs.W2K8Forest.com. 600 IN CNAME W2K8-DC1.W2K8Forest.com.
_ldap._tcp.dc._msdcs.W2K8Forest.com. 600 IN SRV 0 100 389 W2K8-DC1.W2K8Forest.com.
_ldap._tcp.gc._msdcs.W2K8Forest.com. 600 IN SRV 0 100 3268 W2K8-DC1.W2K8Forest.com.
_gc._tcp.W2K8Forest.com. 600 IN SRV 0 100 3268 W2K8-DC1.W2K8Forest.com.
_ldap._tcp.DomainDnsZones.W2K8Forest.com. 600 IN SRV 0 100 389 W2K8-DC1.W2K8Forest.com.
_ldap._tcp.ForestDnsZones.W2K8Forest.com. 600 IN SRV 0 100 389 W2K8-DC1.W2K8Forest.com.
_ldap._tcp.Dallas._sites.W2K8Forest.com. 600 IN SRV 0 100 389 W2K8-DC1.W2K8Forest.com.
_ldap._tcp.Dallas._sites.gc._msdcs.W2K8Forest.com. 600 IN SRV 0 100 3268 W2K8-DC1.W2K8Forest.com.
_ldap._tcp.Dallas._sites.dc._msdcs.W2K8Forest.com. 600 IN SRV 0 100 389 W2K8-DC1.W2K8Forest.com.
_gc._tcp.Dallas._sites.W2K8Forest.com. 600 IN SRV 0 100 3268 W2K8-DC1.W2K8Forest.com.
_ldap._tcp.Dallas._sites.DomainDnsZones.W2K8Forest.com. 600 IN SRV 0 100 389 W2K8-DC1.W2K8Forest.com.
_ldap._tcp.Dallas._sites.ForestDnsZones.W2K8Forest.com. 600 IN SRV 0 100 389 W2K8-DC1.W2K8Forest.com.
_kerberos._tcp.dc._msdcs.W2K8Forest.com. 600 IN SRV 0 100 88 W2K8-DC1.W2K8Forest.com.
_kerberos._tcp.Dallas._sites.dc._msdcs.W2K8Forest.com. 600 IN SRV 0 100 88 W2K8-DC1.W2K8Forest.com.
_kerberos._tcp.W2K8Forest.com. 600 IN SRV 0 100 88 W2K8-DC1.W2K8Forest.com.
_kerberos._tcp.Dallas._sites.W2K8Forest.com. 600 IN SRV 0 100 88 W2K8-DC1.W2K8Forest.com.
_kerberos._udp.W2K8Forest.com. 600 IN SRV 0 100 88 W2K8-DC1.W2K8Forest.com.
_kpasswd._tcp.W2K8Forest.com. 600 IN SRV 0 100 464 W2K8-DC1.W2K8Forest.com.
_kpasswd._udp.W2K8Forest.com. 600 IN SRV 0 100 464 W2K8-DC1.W2K8Forest.com.
W2K8Forest.com. 600 IN A 192.168.1.40
gc._msdcs.W2K8Forest.com. 600 IN A 192.168.1.40
DomainDnsZones.W2K8Forest.com. 600 IN A 192.168.1.40
ForestDnsZones.W2K8Forest.com. 600 IN A 192.168.1.40

Now let’s select a few records from this list and analyze them.

_ldap._tcp.W2K8Forest.com 600 IN SRV 0 100 389 W2K8-DC1.W2K8Forest.com

This is a generic LDAP SRV record.  The information it’s conveying is broken out as follows:

• SERVICE (_ldap): this lets us know that W2K8-DC1 is providing the LDAP service
• PROTOCOL (_tcp): this lets us know that W2K8-DC1 is providing this service through the TCP protocol
• NAME (W2K8Forest.com): this is the domain for which this record is relevant
• TTL (600): this is the default time to live (TTL) of the record (in seconds)
• CLASS (IN): this represents the standard Internet Class for DNS
• RESOURCE RECORD (SRV): this tells us that the record is intended to help clients locate services on the network (in this case, the LDAP service)
• PRIORITY (0): this is the DNS priority for the server.  Clients will try to connect to servers that have a lower number before they attempt to use those with a higher number (in this case, a lower number equals a higher priority).
• WEIGHT (100): This allows for load-balancing of the servers providing the listed service.  If you have newer and faster servers that can handle more client connections, you might want to set a higher weight (a server with a record that has a higher weight has a greater likelihood of being chosen over a lower weight).
• PORT (389): This represents the port that will be used by the server.  So if a client makes an LDAP request to W2K8-DC1, it will need to do so through port 389.
• TARGET (W2K8-DC1.W2K8Forest.com): This is the fully qualified domain name of the server providing the listed service

61657dc6-742e-4fae-98cb-60ddb9406c41._msdcs.W2K8Forest.com. 600 IN CNAME W2K8-DC1.W2K8Forest.com.

Though this isn’t strictly an SRV record (it’s a CNAME record), it’s important to mention since it’s found in the netlogon.dns file.  What this record does is allow the DCs to talk with each other.  The relevant components are as follows:

• DSAGUID (61657DC6-742E-4FAE-98CB-60DDB9406C41): this is the globally unique identifier of the DSA (directory system agent), which is the DC.
• RESOURCE RECORD (CNAME): identifies this record as a canonical name record, providing translation (in this case) between the server GUID and the known computer name.
• TARGET (W2K8-DC1.W2K8Forest.com): The fully qualified domain name of the DC.

There are a few special terms in the netlogon.dns file that deserve a bit more attention.  Those are explained below:

• _gc: This indicates that the server is providing the Global Catalog service
• _pdc: This indicates that the server owns the Primary Domain Controller Emulator role within the domain.  The PDC Emulator is one of the Single Master Operator roles that was explained in an earlier post.
• _msdcs: This is used exclusively for Active Directory Domain Controllers.  It is necessary because services like Kerberos and LDAP aren’t specific to Microsoft, thus clients need a way to identify the Microsoft implementation of these services.

Site-Specific Records

Briefly, I want to make sure and mention an element that appears in some of the SRV records.  Take a look at the record below:

_ldap._tcp.Dallas._sites.W2K8Forest.com. 600 IN SRV 0 100 389 W2K8-DC1.W2K8Forest.com.

Notice that it contains a reference to “Dallas._sites”.  The purpose of this added element is so that clients can specify which Active Directory site they’re in (a site is defined here) and can leverage the resources that are closest to them.  Without these records, DNS might direct a client to use a resource at the other end of a slow network link, but with site-specific records clients can be directed to resource that will provide the most optimal experience.

NOTE: This highlights the importance of always keeping each of your IP Subnets updated and properly linked to sites in Active Directory.

DomainDnsZones and ForestDnsZones

These represent the Application partitions used when DNS is integrated and replicated via Active Directory.  DomainDnsZones are used for information replicated within a domain and ForestDnsZones are used for replicating information between domain controllers within the forest.

The records above are:

DomainDnsZones.W2K8Forest.com. 600 IN A 192.168.1.40
ForestDnsZones.W2K8Forest.com. 600 IN A 192.168.1.40

These are made available for DCs that are seeking other domain controllers that host these partitions (which DCs host an Application partition is specified when the partition is created).  In this case, these records indicate that these partitions can be contacted at the IP address 192.168.1.40 (the IP Address of W2K8-DC1).

Summary

SRV Records can be a very detailed topic and it’s important to spend time getting to know them.  Below I’ve provided several links that provide additional reading on this important topic.

As I said in the previous post, sometimes bad things happen that allow us to share information with each other.  Hopefully this has been helpful.

Articles that will be useful for further reading on this topic:

• SRV Resource Records
• How to verify that SRV DNS records have been created for a domain controller
• Configuring DNS for the Forest Root Domain

Cleaning Up Active Directory Metadata

I’m not 100% satisfied with the video associated with this post, but I’m including it since I think it’s important to AD Administrators to be familiar with this process.  Yesterday, while trying to do something in my lab, I realized I hadn’t brought one of my DCs online in a very long time, and that the tombstone lifetime for the forest had expired.

Deleted Objects, Tombstones and Garbage Collection

As I explain in the video, when an object is deleted in AD it isn’t actually removed from the directory.  Instead, it’s marked as an object to be deleted, it’s hidden and it’s put in a special deleted items container.  The purpose of this is to allow the change to the object (the fact that it’s been removed from AD) to replicate to all other DCs so they can remove their copy of it from their instance of NTDS.DIT.  This “hidden but not-yet-deleted” object is called a tombstoned object.

Additionally, because Active Directory doesn’t allow objects to remain in a “hidden but not-yet-deleted” state, it assigns a lifetime to that tombstoned object measured in days.  The default number of days that the deleted object will be retained is:

• Windows 2000 (all versions): 60 days
• 2003 RTM: 60 days
• Windows 2003 SP1: 180 days
• Windows 2008/Windows 2008 R2: 180 days

You can see (and, if necessary, change) this value by taking the following steps:

1. Connect to a domain controller using ADSI Edit
2. Connect to the Configuration Partition
3. Navigate to the following path:

CN=Directory Service,CN=Windows NT,CN=Services,CN=Configuration,CN=<domain name>

Once at this object, right-click CN=Directory Service and select “properties”.  In the list of attributes, scroll down to “tombstoneLifetime” and you’ll see how many days the deleted objects will be retained in your directory before they’re deleted.

Once a deleted object has been in a deleted state for longer than its tombstone lifetime, it will be permanently removed from the directory.  This happens when a process known as Garbage Collection runs.  This process looks for deleted objects that have exceeded the tombstone lifetime.  If one is found, this object is permanently deleted.

By default, the Garbage Collection process runs once ever 12 hours (known as the Garbage Collection Interval), though the frequency of this process can be changed by going to the same object listed above (CN=Directory Service) and modifying the “garbageCollPeriod”.

By default, a longer tombstone lifetime and a longer Garbage Collection Interval will result in a larger NTDS.DIT file, but it might also allow a longer time to recover accidentally deleted objects. 

WARNING!!!

You’ll have to do a bit of experimenting to see the effect of changing these values, though the strong recommendation is to leave them at their defaults unless you have a valid business need AND you’ve fully tested and understand what these changes will mean to your environment.

Exceeding the Tombstone Lifetime

You can probably already tell why this is relevant to Active Directory, but just in case I’ll explain.  Let’s assume that you have two DCs (DC1 and DC2).  In DC1 you create a user named Steve.  Steve’s user object is replicated to DC2.  Then DC2 goes offline due to a problem and during that outage, Steve leaves the company and you delete his user object.  Time goes by and the replacement for DC2 is delayed and it’s quite awhile before you are able to bring it back online (let’s say that it’s 182 days later).  If DC2 were allowed to begin replicating with DC1 again, we’d have a problem.  Why?

Remember that DC2 received a copy of Steve’s user object, but it never received word that Steve had left the company.  Further, since the tombstone lifetime has expired for Steve’s user object, it’s been completely removed from DC1 and thus there is nothing to replicate to DC2.  So we end up with an inconsistent directory.  DC1 has no record of Steve while DC2 still considers Steve to be a valid user object.

To correct for this problem, Microsoft has designed Active Directory so that a DC can no longer participate in replication if it’s been offline for longer than the tombstone lifetime.  This prevents the sort of scenario I’ve just described from happening.

But it also means that if you need to remove that DC from the Active Directory forest, you must use a manual process.

Removing the Failed DC

All of that explanation is background to get us to the video I’ve created for this post.  What I’ve done in the video is start at the point that DC2 has to be removed from Active Directory since it’s exceeded the tombstone lifetime and can no longer replicate with DC1.  Below, I walk you through the steps of removing DC2 from the Active Directory forest using a process known as Metadata Cleanup.

Though I took one step while the video was not running (and thus wasn’t able to demonstrate it), I fully explain it so that you can understand what will happen when you need to do Metadata Cleanup on your own.

Sometimes when things happen, it opens the door to explain a principle that AD Admins should know about.  Hopefully this post was one of those instances.  Until next time…

For further information on this process, the following articles can provide assistance:

• The Active Directory database garbage collection process
• Clean Up Server Metadata