Posts

Posted on

Windows Server 8 Beta and dumpReferences

Yesterday when I was researching the new Windows Server 8 Beta bits I did a discovery of a new operational attribute [1] – dumpReferences (that I haven’t seen documented anywhere)

So what does “dumpReferences” really do? It works similar to the “dumpDatabase” operational attribute, but instead it finds all none-linked (simple) references to a given objects [2] DNT (Distinguee Name Tag)

[1] List of operational attributes in previous Windows Server releases: http://msdn.microsoft.com/en-us/library/cc223297(v=prot.10).aspx.

[2] If you want to know how DNTs work have a look at: http://blogs.chrisse.se/2012/02/15/how-the-active-directory-data-store-really-works-inside-ntds-dit-part-2

So how do you play with this?

Table 1: dumpReferences

LDIF Sample 

dn:

changetype: modify

add: dumpReferences

dumpReferences:<DN to dump references of>

Let’s try an example
We create two users with the following DNs:

  • CN=Lena Andersson,CN=Users,DC=wtf,DC=nttest,DC=chrisse,DC=com
  • CN=Elina Andersson,CN=Users,DC=wtf,DC=nttest,DC=chrisse,DC=com

Now set the “Se-Also” attribute on “CN= Lena Andersson,CN=Users,DC=wtf,DC=nttest,DC=chrisse,DC=com” to a value of “CN=Elina Andersson,CN=Users,DC=wtf,DC=nttest,DC=chrisse,DC=com”

Let’s dump the references to in the DIT to “CN=Elina Andersson,CN=Users,DC=wtf,DC=nttest,DC=chrisse,DC=com” now – I choose to do this with LDP.exe

Now go into the path where you store [3] the NTDS.dit database on the DC you executed the above operation against and you should find a “NTDS.ref” file open it in your preferred text editor and you can know see the DNTs of all object’s refereeing to “Elina” and by which attribute.

DNT: 4044 is representing the object of “Elina” itself as all objects references them self
DNT: 4047 is representing the object of “Lena” that is referencing “Elina” in the “see-Also” attribute

[3] You can find the path by looking at “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\DSA Database File”

Summary
I don’t see any value of this actually and maybe this is something left in this release for development and testing purposes so it might disappear in the final release – How did I find this one? Well that qualifies for a post by its own to explain all tests I perform on a new Windows build J

Posted on

How to handle built-in domain groups with ADMT

If you ever performed a migration with the Active Directory Migration Tool (ADMT) it probably has come to your attention that you can’t migrate built-in domain groups, the reason for this is simple – those are groups has well-known SIDs ore more specifically well-known RIDs – meaning that their RIDs are the same in any domain.

The ADMT Migration Log will state the following:

Table 1: Migration Log

ADMT Migration Log – Sample
Active Directory Migration Tool – scripted group migration started.
WRN1:0135 The object ‘Users’ is a built-in or well-known account. The object will not be migrated.
WRN1:0135 The object ‘Domain Guests’ is a built-in or well-known account. The object will not be migrated.
WRN1:0135 The object ‘Group Policy Creator Owners’ is a built-in or well-known account. The object will not be migrated.
WRN1:0135 The object ‘Pre-Windows 2000 Compatible Access’ is a built-in or well-known account. The object will not be migrated.

WRN1:0135 The object ‘Performance Log Users’ is a built-in or well-known account. The object will not be migrated.

WRN1:0135 The object ‘Windows Authorization Access Group’ is a built-in or well-known account. The object will not be migrated.

WRN1:0135 The object ‘Remote Desktop Users’ is a built-in or well-known account. The object will not be migrated.

WRN1:0135 The object ‘Enterprise Admins’ is a built-in or well-known account. The object will not be migrated.

WRN1:0135 The object ‘Schema Admins’ is a built-in or well-known account. The object will not be migrated.

WRN1:0135 The object ‘Guests’ is a built-in or well-known account. The object will not be migrated.

WRN1:0135 The object ‘Domain Users’ is a built-in or well-known account. The object will not be migrated.

WRN1:0135 The object ‘Terminal Server License Servers’ is a built-in or well-known account. The object will not be migrated.

WRN1:0135 The object ‘Domain Admins’ is a built-in or well-known account. The object will not be migrated.

WRN1:0135 The object ‘Administrators’ is a built-in or well-known account. The object will not be migrated

 

The problem
This causes an issue if you want to preserve access to resources in the source domain, once a user account has been migrated to the target domain, this is usually accomplished by using sIDHistory – (Migrated users and groups inherit their source SID in the sIDHistory attribute – You can read more here: Using SID History to Preserve Resource Access)

If you have ACLs (Access Control Lists) that contain ACEs (Access Control Entries) either granting or denying rights on a File Share to for example the “Domain Users” group (a group with a well-known SID) in “source.dom” and a user “John” who is member of “Domain Users” are migrated from “source.dom” to “target.dom” with sIDHistory (since built-in groups aren’t migrated – John would know lose the presence of the “Domain Users” group in “source.dom” in his token and can therefore no longer access resources in “source.dom” that was granted to him by the “Domain Users” group, the example below.

The solution using Security Translation
The Active Directory Migration Tool (ADMT) has a feature called Security Translation – briefly described; Security Translation can translate security references on objects in Windows NT:

  • Files and Folders (NTFS)
  • Shares (Share Permissions)
  • Printers
  • Registry
  • Local Groups
  • User Profiles
  • User Rights

Usually security translation is taking place once a resource has been migrated from the source domain to the target domain, running the security translation will effectively replace SIDs (sIDHistory) for Security Principals from the source domain that has been migrated to SIDs in the target domain, this is referred to as the “Replace” operation.

We are going to use the “Add” operation in order to solve the issue with built-in groups – as we can’t migrate built-in groups.

We’re going to create a “fictive” “Domain Users” group in the target domain (Making source that all users being migrated from the source domain are being members of this group once migrated) and we’re going to use the Security Translation to grant this “fictive” “Domain Users” the exactly same permissions as the “Domain Users” group in the source domain has using the security Translation Wizard.

Step-by-Step

  1. Let’s create group in “target.dom” named: “Source_Domain Users” – obtain the SID of the newly created group, as well obtin the SID of the “Domain Users” group in “source.dom”.Create a SID Mapping file that we’re going to use in ADMT and the Security Translation Wizard.
    The format of this file is <source sid>,<target sid> in this case that is <real domain users group in source>, <fictive domain users group in target>

    Table 2: SIDMappingFile

SIDMappingFile Sample
S-1-5-21-1429931347-2825349181-4189204843-513,S-1-5-21-3530694592-736403133-1610603870-21311

Save the SIDMapping file as “SOURCE_TARGET_Domain Users.txt”

Star the Security Translation Wizard in the Active Directory Migration Tool (ADMT):

  1. Select the “Other objects specified in a file” option and type the path to the mapping file we created in Step 1.
  2. Select the appropriate target and source domains.
  3. You should now select all active member servers within the source domain (because you probably don’t know exactly which servers that contain ACEs (Access Control Entries) with the Domain Users group.Note: I recommend gathering a list of active member servers in the source domain and processing them in batches of 20-30.
  4. Select where you want to translate security.
    Note:I recommend the selected only, as it’s very unlikely that the others contain ACLs/ACEs with Domain Users. (You more you select – the longer time it takes to process each server)
  5. Select the “Add” operation, as we want to grant our “fictive” “SOURCE_Domain Users” group the same permissions as the real “Domain Users” group without replacing/removing it – doing so will instead result in lost access for users that hasn’t been migrated yet.Note:Verify twice that you really have selected the “Add” operation.
  6. You’re now ready to start the operation.Note:You might end-up with security issues if the account you’re running this operation with isn’t member of the local administrators group on all servers you intend to translate security on (e.g all member servers in the source domain – you can accomplish that with GPOs or scripting)

Let’s see now how it works

Once security translation has been performed the “SOURCE_Domain Users” group should now have been granted the same rights as the real “Domain Users” group in the source domain “source.dom” it should look like:
Let’s now migrate “John” from “source.dom” to “target.dom” and make him a member of “SOURCE_Domain Users”

Summary

I’m working on a domain migration and consolidation project where we discovered that it was very common in the source domains that ACLs (Access Control Lists) contained ACEs (Access Control Entries) with the built-in “Domain Users” group – resulting in lost permissions for migrated uses (We applied this workaround on 80~ domains being all migrated to a single domain/forest) – Running Security Translation in domains with 1000+ member servers could take an entire week, I’ve recommend to perform security translations on lager file servers off business hours cause the ADMT agent can consume notable amount of CPU.

If someone is interested we had to write a script to do basically the same on NetApp Filers.

Posted on

How the Active Directory – Data Store Really Works (Inside NTDS.dit) – Code [1-3]

So what is the “Code [1-3]” all about and where is Part 4 of the series that you might expect?
Before I go ahead with Part 4 I thought it would be a good idea to sum up Part 1 to Part 3 with code (So that you know how we figured out all this stuff while we was coding on ESEDump) – Note: This most may be targeted for the developer audience more than the general Active Directory administrator.

Disclaimer: The code samples provided here is code snippets that doesn’t represent any code actual code from the DSA and or any other Microsoft products and technologies, nor those they represent the complete source of ESEDump


ESEHelper – A managed ESE wrapper around the ESE APIs

We decided that we wanted to work with ESE in C# (and when we first started this project EseManaged from codeplex wasn’t around) and even if we could have used it later on – I guess we wanted full control and decided to stick with our own wrapper. So when you see references to “EseHelper” in the code snippets below – you know it’s just a wrapper around: Extensible Storage Engine Native APIs – there is no secrets around this J

JET_RETRIVECOLUMN structure custom methods

We extended the JET_RETRIVECOLUMN structure with some additional methods to retrieve data.

Table 0: JET_RETRIVECOLUMN structure

Code Snippet
// The custom methods in JET_RETRIEVECOLUMN allow us to quickly// interpret each column’s data depending on its data type (string, integer, etc.)

internal
struct
JET_RETRIEVECOLUMN

{

public
int columnid;

public IntPtr pvData; // Pointer to the data block in memory

public
int cbData; // Size of the allocated data block

public
int cbActual; // Size of the actual/used data

public
int grbit;

// Offset to the first byte to be retrieved from a column of type

// JET_coltypLongBinary or JET_coltypLongText

public
int ibLongValue;

// Number of values in a multi-valued column

// Can be used to retrive a specific value

public
int itagSequence;

// The columnid of the tagged, multi-valued, or sparse column

// when all tagged columns are retrieved by passing 0

// as the columnid to JetRetrieveColumn.”

public
int columnidNextTagged;

public
int err;

public
void Initialize(ColumnInfo att)

{

this.Initialize(att.ID, 0);

}

public
void Initialize(ColumnInfo att, int cbData)

{

// Initialize with a data block of cbData size

this.Initialize(att.ID, cbData);

}

public
void Initialize(int columnid)

{

// Initialize with an empty data block

this.Initialize(columnid, 0);

}

public
void Initialize(int columnid, int cbData)

{

// Reset the fields

this.cbActual = 0;

this.cbData = 0;

this.err = 0;

// Make sure to free any previously used memory in this instance

if (this.pvData != IntPtr.Zero)

{

Marshal.FreeHGlobal(this.pvData);

this.pvData = IntPtr.Zero;

}

this.columnid = columnid;

this.itagSequence = 1;

// Allocate a new memory block if necessary (if > 0 bytes requested)

this.cbData = cbData;

if (this.cbData > 0)

this.pvData = Marshal.AllocHGlobal(this.cbData);

}

// Copies the current memory block into a byte array

public
byte[] GetData()

{

byte[] output = new
byte[this.cbActual];

Marshal.Copy(this.pvData, output, 0, output.Length);

return output;

}

// Interpret the inner data as a GUID

public Guid GetGuid()

{

IntPtr pGuid = this.pvData;

byte[] bGuid = new
byte[16];

Marshal.Copy(pGuid, bGuid, 0, bGuid.Length);

return
new Guid(bGuid);

}

// Interpret the inner data as a string (automatically checks ASCII or Unicode encoding)

public
string GetString()

{

byte[] data = this.GetData();

if (IsUnicode(data))

return Encoding.Unicode.GetString(data, 0, data.Length);

else

return Encoding.ASCII.GetString(data, 0, data.Length);

}

// Interpret the inner data as a 32-bit integer

public
int GetInteger()

{

return BitConverter.ToInt32(this.GetData(), 0);

}

// Interpret the inner data as a 64-bit integer

public
long GetLong()

{

return BitConverter.ToInt64(this.GetData(), 0);

}

// Interpret the inner data as a boolean (true/false)

public
bool GetBool()

{

return Marshal.ReadByte(this.pvData) == 1 ? true : false;

}

// Determines if a string in a data block is of Unicode or ASCII encoding

// TODO: International 2-byte characters unsupported?

private
bool IsUnicode(byte[] data)

{

bool isUnicode = false;

// Unicode strings’ data always have an even number of bytes

if (data.Length % 2 == 0)

for (int i = 0; i < data.Length; i += 2)

if (data[i + 1] == ‘\0’)

isUnicode |= true;

return isUnicode;

}

}

// ColumnInfo stores column metadata (ID, type, table owner)

// Used when retrieving JET columns

internal
struct
ColumnInfo

{

public
int ID;

public
string Name;

public
int DataType;

public
string AltName; //added for attribute name

public
int AltId; // added for attribute id

public IntPtr TableId; // added for table support in caching

public ColumnInfo(int id, string name, int type, string altname, int altid, IntPtr tableid)

{

this.ID = id;

this.Name = name;

this.DataType = type;

this.AltName = altname; //added for attribute name

this.AltId = altid;

this.TableId = tableid; //added for table support in caching

}

}

 

Perform Initialization and Attach to NTDS.dit

First thing we had to figure out was how we attached to the database (NTDS.dit) using JetInit, JetBeginSession, JetAttachDatabase and finally calling JetOpenDatabase in addition to those callas we had to set several parameters with JetSetSystemParameter for our usage, e.g there is things that need to be turned off as we attach/open the DB as read-only due to the nature of our application.

Table 1: ESE Initialization

Code Snippet
// E.Check makes sure a JET API call is successful, i.e. JET_errSuccess (0)// If the call fails, we throw an exception/write to the Console

// Initialize ESENT. Setting JetInit will inspect the logfiles to see if the last

// shutdown was clean. If it wasn’t (e.g. the application crashed) recovery will be

// run automatically bringing the database to a consistent state.

err = E.Check(EseHelper.JetSetSystemParameter(ref instance, EseHelper.JET_sesidNil, new
IntPtr(EseHelper.JET_paramDatabasePageSize), new
IntPtr(0x2000), null));

err = E.Check(EseHelper.JetCreateInstance(out instance, “instance”));

// Set up the recovery option (off), the maximum number temporary tables (7) and temporary path

err = E.Check(EseHelper.JetSetSystemParameter(ref instance, EseHelper.JET_sesidNil, new
IntPtr(EseHelper.JET_paramRecovery), IntPtr.Zero, “off”));

err = E.Check(EseHelper.JetSetSystemParameter(ref instance, EseHelper.JET_sesidNil, new
IntPtr(EseHelper.JET_paramEnableOnlineDefrag), IntPtr.Zero, null));

err = E.Check(EseHelper.JetSetSystemParameter(ref instance, EseHelper.JET_sesidNil, new
IntPtr(0xa), IntPtr.Zero, null));

err = E.Check(EseHelper.JetSetSystemParameter(ref instance, EseHelper.JET_sesidNil, new
IntPtr(EseHelper.JET_paramMaxTemporaryTables), new
IntPtr(7), null));

err = E.Check(EseHelper.JetSetSystemParameter(ref instance, EseHelper.JET_sesidNil, new
IntPtr(EseHelper.JET_paramTempPath), IntPtr.Zero, System.IO.Path.GetTempPath()));

// Initialize ESE and begin a session

err = E.Check(EseHelper.JetInit(ref instance));

err = E.Check(EseHelper.JetBeginSession(instance, out sesid, null, null));

// Attach a database

err = E.Check(EseHelper.JetAttachDatabase(sesid, “NTDS.dit”, 1));

err = E.Check(EseHelper.JetOpenDatabase(sesid, “NTDS.dit”, null, out dbid, 1));

 

List the tables inside NTDS.dit

We figured out that by statically opening the “MSysObjects” and positioning over the “RootObjects” index we could enumerate the tables inside the database using the following code snippet.

Table 2: ESE Enumerate tables inside the database

Code Snippet
// Simplified:// Method to obtain a list of tables for a given JET database

private
static
void GetTableNames(IntPtr instance, IntPtr sesid, IntPtr dbid, ref EseErrors err)

{

IntPtr tableid = IntPtr.Zero;

List<string> tables = new List<string>();

// E.Check makes sure a JET API call is successful, i.e. JET_errSuccess (0)

// If the call fails, we throw an exception/write to the Console

err = E.Check(EseHelper.JetOpenTable(sesid, dbid, “MSysObjects”, IntPtr.Zero, 0, 0, out tableid));

// Select the first row in the RootObjects record set

err = E.Check(EseHelper.JetSetCurrentIndex(sesid, tableid, “RootObjects”));

err = E.Check(EseHelper.JetMove(sesid, tableid, EseHelper.JET_MoveFirst, 0));

// Allocate a column array of one element — we only need the name column

EseHelper.JET_RETRIEVECOLUMN[] array = new EseHelper.JET_RETRIEVECOLUMN[1];

// Loop until we reach the end of the record set or an error occurs

while (err == 0)

{

// Allocate 0x41 bytes for this column’s value

// 0x80 is the column ID (table name)

array[0].Initialize(0x80, 0x41);

err = E.Check(EseHelper.JetRetrieveColumns(sesid, tableid, Marshal.UnsafeAddrOfPinnedArrayElement(array, 0), array.Length));

foreach (EseHelper.JET_RETRIEVECOLUMN column in array)

{

if (column.cbData != 0x04)

{

string table = column.GetString();

// Print out the table

Console.WriteLine(table);

}

}

// Select the next record (table info row)

err = E.Check(EseHelper.JetMove(sesid, tableid, EseHelper.JET_MoveNext, 0));

}

// Clean up

err = E.Check(EseHelper.JetCloseTable(sesid, tableid));

Console.WriteLine();

}

 

Retrieving the Ancestors_col

In Part 3, we’re discussing the usage of the “Ancestors_col” column and how it’s used to walk subtrees in the database, the DNTs are stored as bytes within the “Acenstors_col” and are read as in the code snippet below.

Table 3: Ancestors_col

Code Snippet
if (column.err != EseErrors.ColumnNull)
{


string ancestory = null;


char[] ancestor_separator = { ‘,’ };

 


// Walk through every ancestry record in the returned column data


// Construct the ancestry string with the returned DNTs


for (int i = 0; i < column.GetData().Length; i += sizeof(int))

{


int dnt = BitConverter.ToInt32(column.GetData(), i);

ancestory = ancestory + dnt + “,”;

}

 


// Remove any trailing “,” characters

ancestory = ancestory.TrimEnd(ancestor_separator);

output = ancestory;

}

 

Reading an object’s full distinguished name

Note: This is our way to read an objects full distinguished name, given an object’s DNT (Distinguished Name Tag) – But this is not considered safe by the DSA as mentioned in Part 3 as the “Ancestors_col” is being processed by a background task and might not be in-sync all the times. (Safer would be to walk the tree up – by each PDNT until PDNT == 2)

Table 4: Get an objects distinguished name by its DNT

Code Snippet
internal
static
string DBGetDN(IntPtr sesid, IntPtr tableid, int tag, ref
EseErrors err)
{

// 0 (zero) means where already positioned at the obejct.

if (tag != 0)

DBFindDNT(sesid, tableid, tag, ref err);

 

List<string> DN = new
List<string>();

 

EseHelper.JET_RETRIEVECOLUMN[] Ancestors = new
EseHelper.JET_RETRIEVECOLUMN[1];

 

Ancestors[0].Initialize(attid.GetByDisplayName(“Ancestors_col”, tableid), 256 * 6);

 

err = E.Check(EseHelper.JetRetrieveColumns(sesid, tableid, Marshal.UnsafeAddrOfPinnedArrayElement(Ancestors, 0), Ancestors.Length));

 

// Ensure the “Ancestors_col” exist and contains data.

if (Ancestors[0].err != EseErrors.ColumnNull)

{


// Loop thru all ancestors.


for (int i = 0; i < Ancestors[0].GetData().Length; i += sizeof(int))

{


int dnt = BitConverter.ToInt32(Ancestors[0].GetData(), i);


if (dnt != 2) // “2” == $ROOT_OBJECT$ == We hit the top most DN Component.

{


// Move the cursor over the ancestor

DBFindDNT(sesid, tableid, dnt, ref err);

 


// Define a list of attributes we want to read off each ancestor (object)


EseHelper.JET_RETRIEVECOLUMN[] attrList = new
EseHelper.JET_RETRIEVECOLUMN[2];

 

attrList[0].Initialize(attid.GetByDisplayName(“RDNtyp_col”, tableid), 256 * 24);

attrList[1].Initialize(attid.GetByDisplayName(“name”, tableid), 256 * 24);

 

err = E.Check(EseHelper.JetRetrieveColumns(sesid, tableid, Marshal.UnsafeAddrOfPinnedArrayElement(attrList, 0), attrList.Length));

 


// Make sure the object has both a RDNType and a Name


if (attrList[0].err != EseErrors.ColumnNull && attrList[1].err != EseErrors.ColumnNull)

{


string RDNType = DBGetRDNType(tableid, attrList[0].GetInteger());


string Name = attrList[1].GetString();


// Construct this ancestors RDN.

DN.Add(RDNType + “=” + Name);

}

}

}

DN.Reverse();


return
string.Join(“,”, DN.ToArray());

}

return
null;

}

 

Table 5: Get an objects distinguished name (object is referenced by sAMAccountName: ADCH)

ESEDump

 

Posted on

How the Active Directory – Data Store Really Works (Inside NTDS.dit) – Part 3

This is the third post in a series of articles that will describe what’s really inside NTDS.dit and how Active Directory works on the database layer, the past two articles has been about:

Support for onelevel searches

Given the knowledge from the last article in the series on how objects are referred to each other in terms of a parent – child relation (DNT and PDNT) it becomes obvious that it is easy to get all direct-descendent/child objects of a given object by searching for all object’s (rows) that have a specific PDNT (where objects.PDNT == parent.DNT) this can efficiently be archived by using the “PDNT_index”

Table 1: PDNT_Index

PDNT_Index
    PDNT_index

 

        Grbit: IndexUnique, IndexIgnoreNull

        CultureInfo: en-US

        CompareOptions: IgnoreCase, IgnoreNonSpace, IgnoreKanaType, IgnoreWidth

            PDNT_col

                Coltyp: Long

                IsAscending: True

                IsASCII: False

            Name [1.1]

                Coltyp: LongText

                IsAscending: True

                IsASCII: False

[1.1]: “Name” represent the RDN attribute, it’s not stored/named as described in the illustration above, it’s rather stored as “ATTm589825” I choose to represent it as “Name” for simplifying the understanding

Introducing the Ancestors_col and support for subtree searches

Support for subtree searching requires the implementation of another column at the DBLayer, the “Ancestors_col”; the other columns are we already familiar with as of the last post in the series

Table 2: datatable – Simplified for hierarchy representation 2

Name

ESE Data Type

ESE Column Options (grbit)

Description
DNT_col JET_coltypLong JET_bitColumnFixed,
JET_bitColumnAutoincrement		 Every object/phantom within the “datatable” contains a unique DNT value.

 

ESE enforces uniqueness by declaring DNT to be an ESE auto-incrementing column (JET_bitColumnAutoincrement.)

DNT is the primary key of the “datatable”, so objects are clustered in storage by DNT, and access to an object by DNT is more efficient than access via any other column/attribute. Since new objects are created in ascending DNT order, the primary key organization does not slow down the creation of new objects.
PDNT_col JET_coltypLong JET_bitColumnFixed The PDNT column holds the DNT of the parent of an object.

 

The tree structure of objects is not represented by pointers from parent to child, as you might expect given how the tree is normally browsed, but by a pointer in each child object/row to its parent
RDNTyp_col JET_coltypLong JET_bitColumnFixed The RDNTyp_col holds the attributeID to the attribute being used as RDN, typically: cn (Common-Name), ou (Organizational-Unit), dc (Domain-Component), o (Organization)
Ancestors_col JET_coltypBinary JET_bitColumnTagged The Ancestors_col holds the DN path [2.1] (every DNT from the root in the hierarchy to the objects DNT) in a binary blob. This always efficient subtree searches by searching the Ancestors_col with a prefix of DNTs to the root:ed object [2.2]

 

 

[2.1]: The first object/row within the “datatable” always has a NULL ancestory_col value.
[2.2] See “Table 4: Ancestors_index”

Let’s apply “Table 2” to a theoretical sample:

[3.1]: “Name” represent the RDN attribute, it’s not stored/named as described in the illustration above, it’s rather stored as “ATTm589825” I choose to represent it as “Name” for simplifying the understanding

If we wanted to do an subtree search with the “Windows Development” organizational unit as the search base – we would like to use an index over the “ancestors_col” [4.1] and set the prefix to be “2,1787,1788,1789,1790,5520,5521*” that would return a list of all objects (rows) that are subordinated the “Windows Development” organizational unit (e.g. all children).

Table 4: Ancestors_index

PDNT_Index
    Ancestors_index

 

        Grbit: IndexIgnoreNull

        CultureInfo: en-US

        CompareOptions: IgnoreCase, IgnoreNonSpace, IgnoreKanaType, IgnoreWidth

            Ancestors_col

                Coltyp: LongBinary

                IsAscending: True

                IsASCII: False

The ancestors_col and the SDProp (Security Descriptor Propagation Demon) – How are they related?

The “ancestors_col” complicates an object move (Within the same NC/Database, We leave cross-NC/database moves outside this article for simplify understanding)

Given the knowledge from the last article in the series on how objects are referred to each other in terms of a parent – child relation (DNT and PDNT) – It seems easy to implement an object move by simply change the PDNT to the DNT of the new designated parent (e.g. give “Christoffer Andersson” a value of “5521” in his “PDNT_col” and the object is now subordinated the organizational unit “Windows Development” instead of “Users” in the above sample)

After the operation above, the “Ancestors_col” wouldn’t be accurate, so the Ancestors_col needs “fixup”, (e.g. it needs to be adjusted to the new path “2,1787,1788,1789,1790,5520,5521,,5524” – “5522” has to be removed as we moved the object “Christoffer Andersson” from the “Users” organizational unit to the “Windows Development” organizational unit) This might not seem to be an issue at the first glance, but imagine moving a large subtree – the operation wouldn’t fit into an single atomic transaction, therefore ancestry fixup is performed in the background by the SDProp (Security Descriptor Propagation Demon) [4.1]

The most experienced Active Directory administrators at least have heard of “SDProp” that is a short for the Security Propagation Demon and some know that it’s responsible for handling propagation of ACE inheritance, very few probably know that it is in addition also responsible to maintain the acenstors_col at the DBLayer.

Each DSA/DC runs the SDProp (Security Descriptor Propagation Demon) as a background task (TQ_TASK). By default, this task is triggered by the following conditions:

  • Any modification (originating or replicated) of the nTSecurityDescriptor attribute of any object (Except for those modifications done by the SDProp demon)
    • This requires that any new/modified inheritable ACEs are propagated to all descendant objects and that any removed inheritable ACEs are removed from all descendent objects, ACE inheritance is not replicated and is being applied by the SDProp using this process on all DSAs/DCs.
  • Any modification of the “PDNT_col” (object is being moved in the directory) of an object that results in the object having a different parent (Except for those cases where the new parent is a Deleted Objects container)
    • This requires adjustments of the values (DNTs) stored in the “ancestors_col” as the object/row has been moved.
    • This requires that all inheritable ACEs present on the new parent object (and all its ancestors from the top) are being propagated down to the object, and that inheritable ACEs originating from the previous parent (and all its ancestors from the top) has to be removed.


      Table 5: Security Descriptor Propagation Demon – Responsibilities Summary

Name

Description
ACE Inheritance Responsible for propagation of inheritable ACEs, if inheritable ACEs are being added or removed from an object’s SD (Security Descriptor) – SDProp is responsible for propagating those ACEs to all descendant / child objects of the object where the ACEs where added. [4.2]
Ancestors_col fixup If any modification to an object’s parent “PDNT_col” occur (Simply an object move) – SDProp is responsible to adjust the “ancestors_col” to match the full path of DNTs from the top down to the object with respect to its new parent.


[4.1] Since this work is performed in the background and SDProp (that runs on single thread doesn’t implement a limit of how many transactions that is used to perform an operation) there may be a period of time when Active Directory returns inconsistent query results with the tree structure, and inheritable ACEs won’t be accurate.

[4.2]: A common misunderstanding is that the SDProp is responsible for maintaining protection for object’s that is protected by the AdminSDHolder, That’s incorrect. The AdminSDHolder runs in its own background task (TQ_TASK) every 60 minutes on the PDC (if a protected object’s security descriptor mismatches from the once at the adminSDHolder object, inheritance is turned off and the security descriptor is over written with the one at the adminSDHolder object) that makes the AdminSDHolder background task to trigger off SDProp .

You can read more about the SDProp – Security Descriptor Propagation Demon here:
http://msdn.microsoft.com/en-us/library/dd350247(v=prot.13).aspx

Trigger the SDProp from manually from the outside.

It’s possible to use an operational attribute to trigger the SDProp demon to run from the “outside” (e.g be performed by an administrator) – How ever on a fully functional DSA/DC there is no need to invoke the SDProp manually, the SDProp can be trigged globally or for a specific object /row identified by its DNT, The requester must have the “Recalculate-Security-Inheritance” control access right on the nTDSDSA object for the DSA/DC

The following shows an LDIF sample that performs this operation on the entire DIT.

dn:
changetype: modify
add: fixupInheritance
fixupInheritance: 1

The following shows an LDIF sample that performs this operation on a specific object /row

dn:
changetype: modify
add: fixupInheritance
fixupInheritance: dnt:5524

You can read more about the “fixupInheritance” operational attribute here: (it’s actually well documented now days)
http://msdn.microsoft.com/en-us/library/cc223299(v=prot.13).aspx

I think that’s all regarding the “ansectors_col” – I might have missed something.

Posted on

How the Active Directory – Data Store Really Works (Inside NTDS.dit) – Part 2

This is the second post in a series of articles that will describe what’s really inside NTDS.dit and how Active Directory works on the database layer, In an earlier post I explained the tables within NTDS.dit in detail as far as what they are used for, in which release of Active Directory (Windows Server) they were introduced in, as well any major changes being added in later versions: How the Active Directory – Data Store Really Works (Inside NTDS.dit) – Part 1

This post will go into the details of the contents of the “datatable” also known as the object store – that contains all objects and phantoms [1.1] represented as rows (1 object/phantom = 1 row in the table) from any instanced naming context (NC) held as either writable or read-only (until they are physically removed by the garbage-collector) by the Directory System Agent (DSA) hosting the database and where columns represent every [1:3] attribute present in the schema except linked attributes [1:2]

[1.1]: phantoms are references to object’s hosted outside the given database (NTDS.DIT) and the given Directory System Agent (DSA) – (Except structural phantoms)

[1:2] Post-Windows Server 2003 the attribute “ntSecurityDescriptor” is stored in the “sd_table” rather than in the “datatable”

[1:3] Some columns doesn’t reflect attributes and are columns pre-defined in the NTDS.dit template database generated by Microsoft (those are needed for internal states to the DSA)

Maintain the hierarchy of an object tree within a flat object store

The hierarchy in Active Directory is quite obvious to most of us at a simplified layer e.g. daily administrative task such as creating an Organizational Unit and creates several descendent/child objects under neat it, some people may refer to some objects as leaf objects (object that usually don’t contain descendent/child objects) such as object of the class “user” – However the fact is that any object within Active Directory has the possibility (technically) to contain one or more descendent/child objects – this is controlled by schema constrains and more specifically the sum of the following attributes of a given object class and any inherited class (except for auxiliary classes) :

Table 1: Possible Superiors

Attribute

Description
possSuperiors Contains references to object classes that can host the given as a descendent/child object.

 

possSuperios can be modified on both cat1 and cat2 schema class object’s after that they have been instantiated in the schema.
systemPossSuperiors Contains references to object classes that can host the given as a descendent/child object.

 

systemPossSuperiors can’t be modified from the outside, once being instanced after the initial creation of the class within the schema.

Why it’s easy for all objects to host descendants/child objects becomes more obvious when the hierarchy is explained at the DBLayer.

The question remains with the details given above, if one row within the “datatable” represents an object/phantom, how can the hierarchy be maintained? The below table “Table 2” represent columns in the “datatable” that are vital for representing/building the hierarchy in the directory at the DBLayer.

Table 2: datatable – Simplified for hierarchy representation 1

Name

ESE Data Type

ESE Column Options (grbit)

Description
DNT_col JET_coltypLong JET_bitColumnFixed,
JET_bitColumnAutoincrement		 Every object/phantom within the “datatable” contains a unique DNT value.

 

DNT is a short for distinguished name tag.

ESE enforces uniqueness by declaring DNT to be an ESE auto-incrementing column (JET_bitColumnAutoincrement.)

DNT is the primary key of the “datatable”, so [2.1] objects are clustered in storage by DNT, and access to an object by DNT is more efficient than access via any other column/attribute. Since new objects are created in ascending DNT order, the primary key organization does not slow down the creation of new objects.
PDNT_col JET_coltypLong JET_bitColumnFixed The PDNT column holds the DNT of the parent of an object [2.2].

 

PDNT is a short for parent distinguished name tag.

The tree structure of objects is not represented by pointers from parent to child, as you might expect given how the tree is normally browsed, but by a pointer in each child object/row to its parent
RDNTyp_col JET_coltypLong JET_bitColumnFixed The RDNTyp_col holds the attributeID to the attribute being used as RDN, typically: cn (Common-Name), ou (Organizational-Unit), dc (Domain-Component), o (Organization) [2:3]

[2.1]: The maximum numbers of objects/phantoms that ever can be created on a given DSA (Domain Controller) for its entire life time is 2 billion objects (147,483,393 (231 minus 255)). Note that this count against objects/phantoms ever introduced to the local DSA as part of any naming context (NC) writable or partial ever hosted by the DSA. * If the DSA is promoted by using IFM (Install from Media it inheritance the count of already allocated DNTs from the former DSA) – When the maximum numbers of auto-increment values has been used (the limit mention above have been hit) the following error are returned at the DBLayer: JET_errOutOfAutoincrementValues -1076 from the outside we will notice: “Error: Add: Operations Error. <1> Server error: 000020EF: SvcErr: DSID-0208044C, problem 5012 (DIR_ERROR), data -1076.” Read more about Active Directory Limits: http://technet.microsoft.com/en-us/library/active-directory-maximum-limits-scalability(v=ws.10).aspx#BKMK_Objects

[2.2]: The first row introduced in the “datatable” isn’t a real object nor is it a phantom and is named “$NOT_AN_OBJECT1$” and have its PDNT_col set to NULL.
The “PDNT_col” is indexed so becomes very easy to drive an object’s direct-descendants/child objects (not all descendants) by simply query who that has the object’s DNT in their PDNT_col.

[2:3] The DSA has an in-memory cache for the most common RDNs (the ones mentioned above)
Active Directory allows us to use a custom attribute as RDN as well by specify the attributeID of that custom attribute as the rDNAttID for the particular class. * A RDN attribute must have the syntax string **customization may not be supported by all LDAPv3 clients. *** rDNAttID should preferably be set before any objects of the given class is instanced in the directory (changes won’t apply to already instanced/existing objects)

Let’s apply “Table 2” to a theoretical sample:

[3.1]: “Name” represent the RDN attribute, it’s not stored/named as described in the illustration above, it’s rather stored as “ATTm589825” I choose to represent it as “Name” for simplifying the understanding of the hierarchy in this case.

[3.2] Structural Phantom – (Different from a phantom used for reference integrity to real object’s hosted outside the given DIT) is used to represent the full distinguished name of the domain e.g “DC=ntdev,DC=corp,DC=chrisse,DC=com”

In the next article – we will continue the deep-dive into the content and the structure of the “datatable” – going thru things like ancestors, the difference between phantoms and real objects, tombstones and the garbage collector on the DBLayer and much more.

Posted on

How the Active Directory – Data Store Really Works (Inside NTDS.dit) – Part 1

You might as I have asked yourself many times – What is inside NTDS.dit? (Most experienced Active Directory admins knows that NTDS.dit is the database and the physical on disk store that Active Directory uses to store information – most of you have probably got in touch with NTDS.dit during backup and restore scenarios)

Long story in a short version – I wasn’t satisfy not knowing – neither was I after being reading the following article:
(That I actually think isn’t that bad – but is also probably the most detailed public available information on the subject)
[1] http://technet.microsoft.com/en-us/library/cc772829(WS.10).aspx

So I decided with a very good friend of mine Stanimir Stoyanov (Microsoft Visual C# MVP) to go ahead and build a tool that could read NTDS.dit and decode its internals, and then we started a journey that has given us invaluable knowledge at this part of Active Directory, this is the first article in a series of articles that will describe what’s really inside NTDS.dit and how Active Directory works on the database layer.

The illustration below has been presented in various documentations since Active Directory was initially released over 10 years ago; a similar illustration is also available in (However after this research project it’s actually turning out to be inaccurate in some aspects – in the way the DRA/REPL communicates with the DBLayer) [1]

Table 1: DSA Components (Simplified for the DBLayer)

Component

Description
Ntdsa.dll – Directory System Agent The DSA, which runs as Ntdsa.dll on each domain controller, provides the interfaces through which directory clients and other directory servers gain access to the directory database (the DBLayer). In addition, the DSA enforces directory semantics, maintains the schema, guarantees object identity, and enforces data types on attributes.
Esent.dll – Extensible Storage Engine (ESE) APIs The Extensible Storage Engine (ESE) is an advanced indexed and sequential access method (ISAM) storage technology. ESE enables applications to store and retrieve data from tables using indexed or sequential cursor navigation. It supports denormalized schemas including wide tables with numerous sparse columns, multi-valued columns, and sparse and rich indexes. It enables applications to enjoy a consistent data state using transacted data update and retrieval.

 

ESE was formerly known as Joint Engine Technology (JET) Blue, The DBLayer uses the ESE APIs documented here: http://msdn.microsoft.com/en-us/library/windows/desktop/gg269259(v=exchg.10).aspx
NTDS.dit The on physical-disk file that represent the ESE/JetBlue database that holds the information store for the given DSA/Active Directory Domain Controller.

Data Store Physical Structure / Inside NTDS.dit – Tables

Finally we can start looking into the content/internal structure of NTDS.dit – but first let’s take a look on what has been reveled before, the illustration below is from [1] and is accurate as far as outside the white box that represent the tables within the database, the tables do exist (Except for * “sd_table” on Windows 2000 DSAs) – but there is more tables that isn’t mentioned in this example.

So it’s about time to reveal the real table structure of an NTDS.dit database file – It’s time to use the tool we produced to first discover this:

Table 2: NTDS.DIT – Tables

Table

Description

Minimum DSA Version
Datatable Contains all objects and phantoms [2.1] represented as rows (1 object/phantom = 1 row in the table) from any instanced naming context (NC) held as either writable or read-only by the Directory System Agent (DSA) hosting the database and where columns represent every [2:3] attribute present in the schema except linked attributes [2:2]

 

[2.1]: phantoms are references to object’s hosted outside the given database (NTDS.DIT) and the given Directory System Agent (DSA)

[2:2] Post-Windows Server 2003 the attribute “ntSecurityDescriptor” is stored in the “sd_table” rather than in the “datatable”

[2:3] Some columns doesn’t reflect attributes and are columns pre-defined in the NTDS.dit template database generated by Microsoft (those are needed for internal states to the DSA)

 Windows 2000 Server

 

Note: Windows Server 2008 R2 added a column to support the “is-Recycled” state
Hiddentable Contains one row but several columns that defines the state of the database as well the [2:2] DNT (reference) of the NTDSA-Settings object that represents this DSA (used for finding config information specific to this domain controller.)

 

[2:4] The concept of DNTs (Distinguished Name Tags)

 Windows 2000 Server
Note: Windows Server 2003 Introduced additional state columns such as backupexpiration_col
Link_table Contains link-pair references (DNT, DNT), the link base (link id >> 1) and possibly a binary blob (In case of DN-binary, DN-string syntax) Windows 2000 Server

 

Note: Windows Server 2008 R2 added a column to support deactivated links for recycle-bin
Sd_table Contains single-instance-stored SDs (Security Descriptors) that pre-Windows Server 2003 was stored in the ntSecurityDescriptor attribute in the “datatable” – those are now instead referenced to the SDs in the “sd_table” that is, if more than one object has exactly the same security defined (Security Descriptor) both objects are referenced to the same row in the “sd_table”, hence the single-instance-storage and reducing the size needed to store Security Descriptors. Windows Server 2003.
Sdpropcounttable Used by the Security Descriptor Propagation Demon (SDProp) responsible for Security Descriptor inheritance down the tree, within the local database  
Sdproptable Used by the Security Descriptor Propagation Demon (SDProp) responsible for Security Descriptor inheritance down the tree, within the local database Windows 2000 Server
Quota_rebuild_progress_table Contains temporary information during quota tracking rebuild, for the Active Directory quota feature introduced in Windows Server 2003 – this allows the demon to keep track of processed objects. Windows Server 2003
Quota_table Contains quota tracking information, for the Active Directory quota feature introduced in Windows Server 2003, quota tracking is peer naming context (NC) and for a given security principal identified by its SID. Windows Server 2003
MSysObjects ESE Internals – out of scope for this article N/A
MSysObjectsShadow ESE Internals – out of scope for this article N/A
MSysUnicodeFixupVer2 ESE Internals – out of scope for this article N/A

In the next article – we will take a deep-dive into the content and the structure of the “datatable” also known as the object-store.

Posted on

How install from media (IFM) really works (Part 2)

This is the second and last blog post (If someone really cares about the differences for ADAM /AD LDS I can point that out to – just send me and e-mail) that completes a series of posts covering how the “Install from media” feature really works, it’s an in-depth very technical series of posts that explains what happens under the hood and this second post explains the changed regarding to this feature that was introduced with Windows Server 2008 (Most of the changes made are to support RODCs as you will noticed if you counties to read).

Note: This article is Windows Server 2003 Install From Media (IFM) functionality + Changes made in Windows Server 2008 and later (This article doesn’t go through the list of functionality that has been left unchanged from Windows Server 2003 again, therefore I recommend to read part 1 first: How install from media (IFM) really works (Part 1)

Background

Install from media was first introduced in Windows Server 2003, as a solution to improve the installation experience of newly promoted domain controllers in branch offices mainly (or sites with slow-links where the initial replication could take significant time to complete), but it is actually an important component in many disaster recovery plans I have designed for various customers over the years, As it is a fast and efficient way to re-install a domain controller and get it up to sync, (that’s the proper way to handle a faulting replicas/domain controllers in most cases). The feature has been mostly changed in Windows Server 2008 and later to address the new type of DCs – Read Only Domain Controllers to be supported by Install from media (IFM) or as sometimes referred to as rifm, there has also been some improvements in the ability to produce install from media (IFM) without taking a regular backup.

What dose Install from media (IFM) consist of

Install from media (IFM) contains two important things.

  • NTDS.DIT (Active Directory Database) – at the time the IFM is generated (Regardless of Windows Server 2003, Windows Server 2008 or later –the NTDS.dit is pretty much unchanged until DCPROMO makes a lot of changes at the becoming domain controller that takes use of the database – it will change the DSA reference and update related “instance specific” information in the hidden table) – How ever this excepts from rifm’s or Read-Only Domain Controller install from media.
  • SYSVOL (SYSVOL GPT Storage)
  • Registry (Contains the SYSKEY used to decrypt the PEK (also known as Password Encryption Key) that efficiently ensure that the protection for sensitive information stored in the Active Directory database (Such as Password Hashes) are unique to each instance of the database (read each domain controller) –Note: This doesn’t apply to RODCs .

Sourcing install from media (IFM) using System State and VSS

Sourcing the media used by IFM is different in Windows Server 2003 (all versions) and Windows Server 2008 and later, the difference is the technology used to gather information required. Windows Server 2008 and later is using VSS and VSS Writers for NTDS (Active Directory Domain Services) and a Registry VSS writer to source the required information to construct an IFM – Note: the Registry doesn’t apply to RODCs

Table 1: VSS Writers used by install from media

Name

Description

Guid
Registry Writer The registry writer is responsible for the Windows registry. Beginning with Windows Vista and Windows Server 2008, the registry writer now performs in-place backups and restores of the registry. On versions of Windows prior to Windows Vista, the registry writer used an intermediate repository file (sometimes called a “spit file”) to store registry data.

In Windows Vista and later, the registry writer does not report user hives.


 The writer ID for the registry writer is AFBAB4A2-367D-4D15-A586-71DBB18F8485.
NTDS Writer Beginning with Windows Server 2003, this writer reports the NTDS database file (ntds.dit) and the associated log files. These files are required to restore the Active Directory correctly.

There is only one ntds.dit file per domain controller, and it is reported in the writer metadata as in the following example:

<DATABASE_FILES path=”C:WindowsNTDS”

filespec=”ntds.dit”

filespecBackupType=”3855″/>

Here is an example that shows how to list components in the writer’s metadata:

<BACKUP_LOCATIONS>

<DATABASE logicalPath=”C:_Windows_NTDS”

componentName=”ntds”

caption=”” restoreMetadata=”no”

notifyOnBackupComplete=”no”

selectable=”no”

selectableForRestore=”no”

componentFlags=”3″>

<DATABASE_FILES path=”C:WindowsNTDS”

filespec=”ntds.dit”

filespecBackupType=”3855″/>

<DATABASE_LOGFILES path=”C:WindowsNTDS”

filespec=”edb*.log”

filespecBackupType=”3855″/>

<DATABASE_LOGFILES path=”C:WindowsNTDS”

filespec=”edb.chk”

filespecBackupType=”3855″/>

</DATABASE>

</BACKUP_LOCATIONS>

At backup time, the writer sets the backup expiration time in the writer’s backup metadata. Requesters should retrieve this metadata by using IVssComponent::GetBackupMetadata to determine whether the database has expired. Expired databases cannot be restored.

If the computer that contains the NTDS database is a domain controller, the backup application should always perform a system state backup across all volumes containing critical system state information. At restore time, the application should first restart the computer in Directory Services Restore Mode and then perform a system state restore.


 The writer ID for this writer is B2014C9E-8711-4C5C-A5A9-3CF384484757.

Sourcing install from media (IFM) in Windows Server 2008 and later with NTDSUTIL

Windows Server 2008 introduces a new context in the NTDSUTIL command line tool to give us a built-in tool to produce install from media instead of having us to perform and restore a backup as in Windows Server 2003.

The new context is named “IFM” and is designed to produce install form media IFM for the following cases.

Table 2: NTDSUTIL IFM options

Name

Notes

Source

Destination
Full IFM Note: If the SYSVOL tree is located at the same volume as the database, its log files and/or the registry – it will still be included in the snapshot but not copied into the IFM media.

Note: The NTDS VSS writer is invoked and as a result of this we can see that the ‘state_col’ in the ‘hiddentable’ in the ntds.dit database are changed to 4. This means a status of a Backed up database. This flag is only set if the NTDS VSS Writer is used and not for legacy backups.

 This command can only be performed on a full/writable DCs This IFM media can only be used to promote full DCs (Technically unless instanceType is changed recursive on all NCs)
Full IFM with SYSVOL Same as above +

Note: The full SYSVOL tree will be copied to the IFM media, except DfsrPrivate Folders.

 This command can only be performed on a full/writable DCs This IFM media can only be used to promote full DCs (Technically unless instanceType is changed recursive on all NCs)
RODC IFM Note: The NTDS VSS writer is passed with a “special” secrets flag in this case, performing a delete of all columns in the database that contains a secret attribute, hidden attribute or an attribute that has been marked as “filtered attribute set” in the schema.

Note: That the PEK (Password Encryption Key) is stored in the “pekList” is not marked secret, (it can’t be as it’s the master key used to protect secret attributes/columns) but it’s rather marked as a ‘hidden attribute’ meaning that we at this state has cleared out the master key for decrypting any other secrets in the DB, however those has just also been cleared so this makes it “safe safe”

NTDSUTIL will remove link values for linked attributes that are marked as “filtered attribute set” in the schema (This is not done by the NTDS VSS writer) and if the command is performed on a full/writable DC all objects including NC heads will recursively change InstanceType – clearing the 0x4 – Write Flag.

Note: In Windows Server 2008 R2 a check is performed against the domain functional level (DFL) and is failing the command if we’re below Windows Server 2003 as DFL with “Can’t produce RODC IFM media for down-level instances) According to be this is an error/mistake in the product as RODCs requires Windows Server 2003 forest functional level (FFL) and not DFL – furthermore there is no issues performing RODC IFM media while the DFL is in Windows 2000 Native for example and then rise the FFL to Windows Server 2003 and introduce a RODC using that media created prior to that the FFL was Windows Server 2003.

Experienced administrators can actually by pass this check, but I won’t include those steps here.

 This command can be performed on a full/writable DCs as well Read-Only DCs This IFM media can only be used to promote RODCs
(Technically unless instanceType is changed recursive on all NCs)
RODC IFM with SYSVOL Same as above +

Note: The full SYSVOL tree will be copied to the IFM media, except DfsrPrivate Folders.

 This command can be performed on a full/writable DCs as well Read-Only DCs This IFM media can only be used to promote RODCs
(Technically unless instanceType is changed recursive on all NCs)

Once the snapshot is performed by the VSS writers, the snapshotted volumes are mounted. There would be one mount point entry for each drive that contains one of the following:

  • NTDS.dit – Active Directory Database
  • Log Files – Active Directory Database Log files (Technically never included/copied to the IFM media, but needed by the NTDS writer itself).
  • Registry – The SYSTEM and the REGISTERY hives (Those are not copied for Read-Only Domain Controller IFM or rifm)
  • SYSVOL – SYSVOL is only included if request (Full IFM with SYSVOL or RODC IFM with SYSVOL)

That means if all of the above (A,B,C,D) are located at the C: drive, there will only be one mount point for the C: drive mounted, once one or more mount points has been created, the data listed in (A,B,C,D) are copied over to the following structure:

Table 3: IFM on disk structure

Folder Name

Content
Active Directory NTDS.dit – Active Directory database
Registry SYSTEM and SECURITY – registry hives (Except for RODCs)
SYSVOL The full SYSVOL tree – only if requested in NTDSUTIL


Conversations taking place in DCPROMO for Read-Only Domain Controllers.

If a Read-Only Domain Controller (RODC) is being promoted using a RODC media generated from a full DC the following conversations has to take place.

  1. The attribute msDS-hasMasterNCs are moved into msDS-hasFullReplicaNCs
  2. The binary portion of msDsHasInstantiatedNCs is changed from indicating to have writable NCs instanced to have none-writable NCs instanced.
  3. Update msds-NCType:
    1. Schema NC are updated to contain: NCT_SPECIAL_SECRET_PROCESSING
    2. Domain, Configuration NC and any hosted NDNCs are updated to contain: NCT_SPECIAL_SECRET_PROCESSING | NCT_FILTERED_ATTRIBUTE_SET
    3. On any partial NCs (If the DC the IFM was sourced from was a Full DC and GC) are updated to contain: NCT_FILTERED_ATTRIBUTE_SET

Note: Those conversations are only necessary when a RODC is being promoted by IFM media that was converted from a full DC.

Preventing an invalid database to be used by IFM

There are several checks taking place during a DCRPOMO IFM to determine that the database that is used during IFM is valid according to a few rules.

  1. Preventing a Read-Only Domain Controller (RODC) promotion using a none-converted IFM media from a full DC: This is prevented by looking at the instanceType at the domain NC head, If it contains it WRITE flag and a promotion of an RODC is in progress, the promotion fails with: The Install-From-Media promotion of a Read-Only DC cannot start because the specified source database is not allowed. Only databases from other RODCs can be used for IFM promotion of a RODC. (8200)
  2. Preventing a Full/Writable Domain Controller (DC) promotion using a RODC IFM media: This is prevented by looking at the instanceType at the doman NC head, if it doesn’t contain the it WRITE flag and a promotion of a Full/Writable DC is in progress the promotion fails with an error similar to the one above.
  3. If the schema version in the IFM media database used to promote the DC is different from the local machines schema.ini, the builds between the source and the current operating system the IFM media is being used on are considered a mismatch and the promotion will fail.

Preventing secrets to sneak into Read-Only Domain Controller (RDOC) being promoted using IFM

Even if the NTDSUTIL tool makes sure that IFM produced for Read-Only Domain Controllers (RODCs) are completely removed from secret attributes, hidden attributes and attributes that contain the “Filtered Attribute Set” flag, In fact the columns representing those attributes in NTDS.dit and the ‘datatable’ are actually completely removed, and any linked attributes that contain the “Filtered Attribute Set” flag will have their rows deleted from the ‘link_table’.

Note: Those are all physical deleted and don’t end up in the Deleted Objects container in either the IsDeleted or IsRecycled state.

DCPROMO performs an additional check, and ensure that no secrets are present in the NTDS.dit while a RODC is being promoted from IFM, if there is columns in the databse representing any secrets (as mentioned above) those will be deleted.

A final solution to make sure that if there are any secrets left in the DB, the last allocated USN to the database before it was IFM’ed are stored as in every database in the ‘hiddentable’ in the ‘usn_col’ column.

(After that the DB is fully initialized and accepts updates again, new USNs are allocated and this is reflected in the ‘usn_col’) However prior to that the Directory System Agent (DSA) accepts updates again, the value in the ‘usn_col’ are copied over to a new column named ‘usnatrifm’ this column will maintain the USN prior to the promotion of the DC using the IFM media, and remain the same as it was when the IFM media was produced for the entire life time of the database (until the DC is demoted)

This allows replication of secrets to compare the ‘usnatrifm’ with the metadata of the attribute containing the secret being replicated in, if that USN has a higher value than the ‘usnatrifm’ column, the current secret in the database is considered not cached (consider to not be valid) and will be replicated in/writing over the old secret that made it in with the IFM.

The reason for why the ‘usnatrifm’ has to remain for the entire life time of the DC is that secret caching happens on-demand meaning that a secret may made it in with IFM for user account ‘ChristofferA’ and ‘ChristofferA’ moves into the branch where the RODC is placed and authenticates 5 years after it was promoted.

Posted on

How install from media (IFM) really works (Part 1)

This is the first blog post in a series of posts covering how the “Install from media” feature really works, it’s an in-depth very technical post that explains what happens under the hood and this first part focuses on how it works in Windows Server 2003.

Background

Install from media was first introduced in Windows Server 2003, as a solution to improve the installation experience of newly promoted domain controllers in branch offices mainly (or sites with slow-links where the initial replication could take significant time to complete), but it is actually an important component in many disaster recovery plans I have designed for various customers over the years, As it is a fast and efficient way to re-install a domain controller and get it up to sync, (that’s the proper way to handle a faulting replicas/domain controllers in most cases). There is some common misunderstandings of the concept “Install from media” I terms of if the operation could be performed entirely offline or online, the short answer is: No. It can’t be performed offline; you have to be online with at least one writable domain controller in the same domain as the IFM source is taken from and even then you may not be able to be fully efficient and cause replication to happen over the network anyway, this need some future explanation.

What dose Install from media (IFM) consist of

Install from media (IFM) contains two important things.

  • NTDS.DIT (Active Directory Database) – at the time the IFM is generated (Regardless of Windows Server 2003, Windows Server 2008 or later –the NTDS.dit is pretty much unchanged until DCPROMO makes a lot of changes at the becoming domain controller that takes use of the database – it will change the DSA reference and update related “instance specific” information in the hidden table )
  • SYSVOL (SYSVOL GPT Storage)
  • Registry (Contains the SYSKEY used to decrypt the PEK (also known as Password Encryption Key) that efficiently ensure that the protection for sensitive information stored in the Active Directory database (Such as Password Hashes) are unique to each instance of the database (read each domain controller) –Note: This doesn’t apply to RODCs .

Sourcing install from media (IFM) using System State and VSS

Sourcing the media used by IFM is different in Windows Server 2003 (all versions) and Windows Server 2008 and later, the difference is the technology used to gather information required.

  • Windows Server 2003 IFM media is generated by performing a system state backup, the reason for this is that we can get a copy of ntds.dit while we’re up and running DsIsNTDSOnline=True (Active Directory is operational) this is archived by the DsBackup API: http://msdn.microsoft.com/en-us/library/ms675896(VS.85).aspx.We will also get a copy of SYSVOL since a system state backup contains the following:
    • Active Directory
    • The SYSVOL tree
    • The Boot.ini file
    • The COM+ class registration database
    • The registry

    To be more specific the following is required in the system state in order to be able to source IFM:

    • Active Directory is required.
    • The SYSVOL tree may be optionally removed. (A specific configuration is required to source the SYSVOL tree during IFM promotion)
    • The Boot.ini file may be removed.
    • The COM+ class registration database may be removed.
    • The registry folder is required. Registry components are required as follows:
      • The Default file in the Registry folder may be removed.
      • The SAM file is required.
      • The SECURITY folder is required.
      • The SOFTWARE file may be removed.
      • The SYSTEM file is required.

    We’re are responsible for doing a restore of a system state backup or selves to an alternative location, and ensure we gather the required information above. (Optionally if we care about disk size optimization we can not only select the specific components required in the system state backup, but we could also perform a offline defragmentation of the ntds.dit database)

    We can only use IFM to promote domain controllers in the same domain as we sourced it from, as well the target domain controller has to be running the same operating system, including service pack and architecture (x86/x64)

    Sourcing the NTDS.DIT – Active Directory database from IFM in Windows Server 2003

    As explained earlier in this article the Active Directory database (NTDS.dit) is being backup either by the DsBackup API or VSS API – The Active Directory database are unique to each DC and contains the NCs hosted by the DC typically Domain, Schema, Configuration and in some cases NDNC’s also known as application partitions such as DomainDNSZones and ForestDNSZones, if the DC is also a GC it will contain partial information about every object in all domain NCs in the entire forest (a multiple domain forest) – even thou if a DC are a GC in a single domain environment, the domain NC is only stored once in the database, if the sourced database was a GC, the computer being promoted to a DC using the source will also become a GC.

    During the backup itself (even if the intent are for IFM) – No changes are made to the database except setting the backup expiration date and backup usn in the hidden record. (Also known has the hidden table). Changes are instead modified/adjusted during DCPROMO on the computer that are about to become a DC using the sourced database from IFM.

    Before we can have an in-depth look at the changes that take place inside the Active Directory database during IFM we need to understand the physical layout of the database and some key concepts. (Note: For the purpose of this article the NTDS.dit physical layout will be rather simplified than the real/exact layout for display purposes)

    The Active Directory database (NTDS.DIT) contains the following tables.

    Table 1: NTDS.DIT Database Layout (Simplified Version)

Table

Description
datatable Contain: Domain, Schema, Config and NDNCs as well partial NCs and basically every object and phantom.
hiddentable Contains the DSA identity and various related information
linktable N/A – Out of scope for this article
quota_rebuild_progress_table N/A – Out of scope for this article
quota_table N/A – Out of scope for this article
sdproptable N/A – Out of scope for this article
sd_table N/A – Out of scope for this article
  • Ensure the database isn’t out-dated aka older than Tombstone Life Time (TSL)
    During DCPROMO the sourced IFM database are verified that it hasn’t passed the Tombstone Life Time (TSL) in the forest, if the database is older than the TSL, the promotion is aborted. This is more specifically measured against the object who had the last USN Change and it’s when-changed date.

  • Change/Adjust the DSA – Directory Service Agent IdentityThe “hiddentable” also known as the hidden record contains the identity of the local DSA – Directory Services Agent (DC) hosting the instance of the database, this identity points to the DCs NTDSA object within the directory stored within the “datatable”, the ntdsa object in its turn contains necessary information such as the DMD to be able to read the schema and soon. (I will write another article on how this works – more in detail). When it comes to IFM, the DSA identity stored in the hidden record/hidden table still points to the identity of the DC the IFM was sourced from, this cause an issue.The following illustrates the relationship between the DSA identity in the “hiddentable” and the NTDS Settings object stored in the “datatable” – that’s actually the NTDS Settings object you will see under the server object in Active Directory Sites and Services.

    Table 2: NTDS.DIT “datatale” (Simplified Version)

    So in this example NTTEST-SCH-01 is the DC where we sourced the IFM (backed up the database) – Now how do we get the new identity of the computer being promoted to a DC using the sourced IFM media? Well the new DSA for the computer being promoted is actually created remotely during DCPROMO on another DC (at the DC we perform the initial replication with, specified in the unattended answer file as “ReplicationSourceDC” parameter) , before we can change the record in the “hiddentable” to point to it – we must replicate in the newly created DSA by replicating the Configuration NC – this can cause an issue by itself – If we’re preforming the initial replication with the DC we sourced the IFM from, we’re temporarily presenting us self as the very same DC and replication will fail.

    So how are this solved? We’re creating a temporary “dummy” DSA with a corresponding server object in the database, retire invocation IDs, copying all references to NCs hosted by the old DSA (Doing not will have KCC to trigger a deletion of those sourced NCs later on), the temporary “dummy” DSA are created in the any site and the first site in alphabetical order identified.

    Now we can re-initialize and successfully replicate in the remotely created DSA by replicate the configuration NC using the temporary “dummy” DSA identity, retire invocation IDs, copying all references to NCs hosted by the temporary “dummy” DSA (Doing not will have KCC to trigger a deletion of those sourced NCs later on) we can now remove the temporary “dummy” DSA and store the real identity of the computer being promoted in the “hiddentable”

    We can now re-initialize as our self’s with our real DSA identity and continue processing.

  • Remove None-Replicated attributes from the database.
    Non-replicated attributes (Containing bit 0x00000001 in System-Flags) , such as badPwdCount, Last-Logon, and Last-Logoff are stored on each domain controller, but are not replicated. The non-replicated attributes are attributes that pertain to a particular domain controller, as those attributes contains local information associated with the DC that the IFM was sourced from, those are deleted except the following:
  1. DS-Core-Propagation-Data: http://msdn.microsoft.com/en-us/library/ms675655(v=VS.85).aspx
  2. Obj-Dist-Name: http://msdn.microsoft.com/en-us/library/ms675516(v=VS.85).aspx
  3. ms-DS-ReplicationEpoch: http://msdn.microsoft.com/en-us/library/ms677478(v=VS.85).aspx
  4. Object-GUID: http://msdn.microsoft.com/en-us/library/ms679021(v=VS.85).aspx
  5. Partial-Attribute-Deletion-List: http://msdn.microsoft.com/en-us/library/ms679106(v=VS.85).aspx
  6. Partial-Attribute-Set: http://msdn.microsoft.com/en-us/library/ms679107(v=VS.85).aspx
  7. Prefix-Map: http://msdn.microsoft.com/en-us/library/ms679371(v=VS.85).aspx
  8. Repl-Property-Meta-Data: http://msdn.microsoft.com/en-us/library/ms679448(v=VS.85).aspx
  9. Repl-UpToDate-Vector: http://msdn.microsoft.com/en-us/library/ms679450(v=VS.85).aspx
  10. Sub-Refs: http://msdn.microsoft.com/en-us/library/ms679895(v=VS.85).aspx
  11. USN-Changed: http://msdn.microsoft.com/en-us/library/ms680871(v=VS.85).aspx
  12. USN-Created: http://msdn.microsoft.com/en-us/library/ms680924(v=VS.85).aspx
  13. When-Changed: http://msdn.microsoft.com/en-us/library/ms680921(v=VS.85).aspx
  14. Pek-List: http://msdn.microsoft.com/en-us/library/ms679109(v=VS.85).aspx
  15. msDS-NcType: http://msdn.microsoft.com/en-us/library/cc220312(v=prot.10).aspx
    • Decrypt and Re-encrypt the Password Encryption Key (PEK)
      Secret Data stored within the Active Directory database (NTDS.DIT) such as the password hashes are additionally protected by a Password Encryption Key (PEK) – the PEK are encrypted by the SYSKEY of the DC and are therefore unique to each DC, the sourced NTDS.dit from the IFM contains a PEK encrypted by the SYSKEY from the DC on which the IFM was generated (the computer where the NTDS.dit was backed up). DCPROMO will decrypt the PEK using the SYSKEY (from the DC the IFM was sourced from) from the registry in the restored IFM information as the SYSKEY are stored in the registry (that’s one reason why we need to include parts of the registry in IFM) and then re-encrypt the PEK with the SYSKEY of the computer being promoted to DC.

 

  • Diagnostics and Logging
    IFM promotions can be identified in the Dcpromo.log and Dcpromoui.log files that are located in the %systemroot%debug folder. There are several entries that can be used to verify that the database where sourced from the IFM and that the promotion did use IFM.

    Table 3: DCPROMO.log

DCPROMO.log
07/03 06:35:29 [INFO] Copying restored Active Directory files from C:IFM_MEDIAActive Directoryntds.dit to C:WINDOWSntdsntds.dit…
07/03 06:35:29 [INFO] Copying restored Active Directory files from C:IFM_MEDIAActive Directoryedb00001.log to C:WINDOWSntdsedb00001.log…
07/03 06:35:29 [INFO] Active Directory is initializing the restored database files. This might take several minutes.

Table 4: DCPROMOUI.log

DCPROMOUI.log
dcpromoui AAC.AB0 0271 Enter State::ReplicateFromMedia true
dcpromoui AAC.AB0 0272 Enter State::GetReplicationSourcePath C:IFM_MEDIA

Sourcing NDNCs with Windows Server 2003 is only supported by Windows Server 2003 SP1 or later under the following conditions:

  • Both the DC your souring the IFM from must be running Windows Server 2003 SP1 or later and as well the machine intending to become a DC using the source IFM.
  • The forest functional level (FFL) has to be: Windows Server 2003 (Pre-Windows Server 2003 FFL adding replicas to NCs has to be done on the Domain Naming Master – FSMO)
    Note: The promotion completes with the sourced IFM even if the forest functional level (FFL) is less than Windows Server 2003 but NDNCs aren’t sourced from the IFM and the following will happen:
  1. The following will be logged in the Directory Services Log: “The forest functional level is not high enough to complete addition of application directory partitions during installation of the directory. Therefore specified application directory partitions will not be added to this domain controller during installation. If you would like to make this server a replica of an application directory partition, you could re-add these application partition after the installation is complete.”
  2. The following will be logged in the Directory Services Log, and the DC will begin the process of physically removing the NDCs source from the IFM in the DCs database: “The local domain controller is no longer configured to host the following directory partition. As a result, the objects in this directory partition will be removed from the local Active Directory database.” Note: In this case the DomainDNSZones and ForestDNSZones NDNCs.
  3. The DomainDNSZones and ForestDNSZones are begin replicated in again over the wire using normal replication, as the promoted DC (Sourced from IFM) hosts the DNS Service: As a result of this the DC (Sourced from IFM) has to obtain a new invocationID once again (It has already done this once for using the sourced IFM database instance):
    Note: This can be confirmed by running the repadmin /showsig command

    Default-First-Site-NameNTTEST-SCH-02

    Current DC invocationID: 7bbd4543-cf19-44e3-9638-96907ceb8a36 ß Current InvocationID obtained cause of removing/adding NDCs.

    28081325-eee8-40b0-9587-9c02867040bc retired on 2011-07-03 07:16:41 at USN 32780 ß New InvocationID representing the sourced IFM restored/promoted as a new instance on the current DC

    b7633426-242b-47bf-852c-a07466ef937f retired on 2011-07-03 06:35:39 at USN 16397ß InvocationID representing the instance on the DC where the IFM where sourced

  • You have to use an unattended answer file specifying the ReplicateFromMedia=Yes parameter as well define the ApplicationPartitionsToReplicate parameter, note this can be used to include specific NDNCs or you can simply include them all by specific a wildcard, Here are some samples:
    • ApplicationPartitionsToReplicate=*
    • ApplicationPartitionsToReplicate=”DC=DomainDNSZones,DC=corp,DC=company,DC=com” “DC=ForestDNSZones,DC=corp,DC=company,DC=com”

Sourcing SYSVOL with IFM at Windows Server 2003:

The File Replication Service (FRS) can source files and folders from the restored system state backup on the first restart after a DCPROMO IFM promotion if the strict dependencies that the File Replication Service (FRS) requires are fulfilled.

  • The system state backup must contain MD5 checksum data that is used by the File Replication Service (FRS) to determine if a restored file or folder is the same as the file versions on existing domain controllers in the domain.
  • The File Replication Service (FRS) must have constructed MD5 checksum data for the files in the SYSVOL tree.
    For MD5 checksums to exist, files and folders in the SYSVOL tree must have been replicated at least one time after there were two or more domain controllers in the domain (Note: The SYSVOL can never be efficiently sourced from a IFM media also known as System State Backup in Windows Server 2003 unless there is at least two DCs in the domain already present at the point when the IFM media also known as System State Backup is generated). You can trigger FRS to store the MD5 checksum of all files in the SYSVOL tree my writing a script that modifies the files, for example set/un-set the hidden attribute, that simply will tiger a replication.Furthermore the MD5 checksum data is stored in ntfrs.jdb ESE database that’s by default located in “%SystemRoot%ntfrsjet”. The ntfrs.jdb ese database are using 4k pages and have the following layout.Table 5: NTFRS.JDB Database Layout
Table

Columns
ConfigTable N/A – Out of scope for this article
CXTIONTable00000 N/A – Out of scope for this article
CXTIONTable00001 N/A – Out of scope for this article
DIRTable00000 N/A – Out of scope for this article
DIRTable00001 N/A – Out of scope for this article
IDTable00000 FileGuidFileID

ParentGuid

ParentFileID

VersionNumber

EventTime

OriginatorGuid

OriginatorVSN

CurrentFileUsn

FileCreateTime

FileWriteTime

FileSize

FileObjID

FileIsDir

FileAttributes

Flags

ReplEnabled

TombStoneGC

OutLogSeqNum

Spare1Ull

Spare2Ull

Spare1Guid

Spare2Guid

FileName

Spare1Wcs

Spare2Wcs

Spare1Bin
IDTable00001 N/A – Out of scope for this article
INLOGTable00000 N/A – Out of scope for this article
INLOGTable00001 N/A – Out of scope for this article
OUTLOGTable00000 SequenceNumberFlags
IFlags
State
ContentCmd
Lcmd
FileAttributes
FileVersionNumber
PartnerAckSeqNumber
FileSize
FileOffset
FrsVsn
FileUsn
JrnlUsn
JrnlFirstUsn
OriginalReplica
NewReplica
ChangeOrderGuid
OriginatorGuid
FileGuid
OldParentGuid
NewParentGuid
CxtionGuid
Spare1Ull
Spare2Ull
Spare1Guid
Spare2Guid
EventTime
FileNameLength
Spare1Wcs
Spare2Wcs
Spare1Bin
Spare2Bin
FileName
OUTLOGTable00001 N/A – Out of scope for this article
VVTable00000 N/A – Out of scope for this article
VVTable00001 N/A – Out of scope for this article

The MD5 checksum are stored in the IDTable0000X and are stored in the column Spare1Bin.
You can validate the existence of MD5 checksums by using “ntfrsutil idtable > MD5Hash.txt” and search for entries missing hashes.


Table 6: NTFRS.JDB IDTable

IDTable
Table Type: ID Table for DOMAIN SYSTEM VOLUME (SYSVOL SHARE) (1)
FileGuid : 790adf00-7709-447d-9a756b655931151b

FileID : 00030000 00002bb9
ParentGuid : 9fe6d1af-dac8-40be-bd06e6030f4a6ae0

ParentFileID : 00050000 00002874

VersionNumber : 00000003

EventTime : Sun Sep 11, 2005 03:36:18

OriginatorGuid : 5ab8b3e3-27e8-4c9e-a5e152246f57df61

OriginatorVSN : 01c5b0b1 ad0a2413

CurrentFileUsn : 00000000 011a3e88

FileCreateTime : Sun Aug 7, 2005 23:40:01

FileWriteTime : Sun Sep 11, 2005 03:36:18

FileSize : 00000000 00002000

FileObjID : 00000000-0000-0000-0000000000000000

FileName : CorpLogon.vbs

FileIsDir : 00000000

FileAttributes : 00000020 Flags [ARCHIVE ]

Flags : 00000000 Flags [<Flags Clear>]

ReplEnabled : 00000001

TombStoneGC : Sun Aug 7, 2005 23:42:01

OutLogSeqNum : 00000000 00000000

Spare1Ull : 00000000 00000000

MD5CheckSum : MD5: b9552637 1973e4d6 84dd6de7 f3125a64

RetryCount : 0

FirstTryTime :
Table Type: ID Table for DOMAIN SYSTEM VOLUME (SYSVOL SHARE) (1)
FileGuid : 127f0701-7e51-4624-b0ef801fa93bb925

FileID : 00040000 00002b7f

ParentGuid : 8bebf734-7fd2-4d66-b8548820dcb17bad

ParentFileID : 00020000 00002b7b

VersionNumber : 0000000b

EventTime : Fri Nov 13, 2009 14:46:45

OriginatorGuid : 5ab8b3e3-27e8-4c9e-a5e152246f57df61

OriginatorVSN : 01c96785 4bd8b2c5

CurrentFileUsn : 00000001 df606a48

FileCreateTime : Sat Aug 6, 2005 23:27:54

FileWriteTime : Fri Nov 13, 2009 14:46:45

FileSize : 00000000 00000020

FileObjID : 127f0701-7e51-4624-b0ef801fa93bb925

FileName : gpt.ini

FileIsDir : 00000000

FileAttributes : 00000020 Flags [ARCHIVE ]

Flags : 00000000 Flags [<Flags Clear>]

ReplEnabled : 00000001

TombStoneGC : Sat Aug 6, 2005 23:28:01

OutLogSeqNum : 00000000 00000000

Spare1Ull : 00000000 00000000

MD5CheckSum : MD5: 959cfc82 54e6ccbf 3dc74a29 7fe46a42

RetryCount : 0

FirstTryTime :
  • The SYSVOL part of the IFM media also known as the System State Backup must be restored to the same volume that is chosen to host the SYSVOL tree when you run DCPROMO, or it has to be specified to the same value in your unattended answer file.


Seeding the SYSVOL with IFM at Windows Server 2003:

Even if the SYSVOL are sourced with IFM delta changes are about to be replicated in over the network, there are certain requirements to ensure that this process are being efficient and that not the entire SYSVOL tree are replicated over the network again, once of the requirements has already been discussed in this article regarding MD5 checksums: The File Replication Service (FRS) must have constructed MD5 checksum data for the files in the SYSVOL tree.

The Domain Controller (DC) or File Replication Service Replica (FRS Replica) that the initial replication of the SYSVOL tree takes place with must meet the following requirements

  • The Domain Controller (DC) or File Replication Service Replica (FRS Replica) that the initial replication takes place with are identified by specifying the “ReplicationSourceDC” parameter in the unattended dcpromo answer file. (Note this can’t be done using the UI)

    How to best select the Domain Controller (DC) or File Replication Service Replica (FRS Replica) to perform initial replication with:

    • Locate a domain controller that has a low number of inbound and outbound connections. This domain controller must not be a significant originator or forwarder of change orders to downstream partners in SYSVOL or FRS-replicated DFS replica sets
    • Locate a domain controller that doesn’t act as Bridgehead server (those typically have many replication partners)
  • The File Replication Service (FRS) outbound log on the DC/FRS replica that is used to seed the SYSVOL tree with must be cleared so that a full vvjoin is triggered when the initial synchronization of SYSVOL with the IFM promoted DC occur, the reason for this is that if the outbound log contain cached items, an optimized vvjoin is performed and optimized vvjoin’s doesn’t support pre-staging content, this results in a full replication of the entire SYSVOL tree over the network with the IFM promoted DC instead of delta changes and new files.

    How to verify and clear the outlog:

    • At the intended helper DC (DC chosen to perform initial replication with) run:
      ntfrsutil outlog to show current entries in the outlog change cache.
      Note: See Table 5: NTFRS.JDB Database Layout for the outlog table layout earlier in this article
    • If the ntfrsutil outlog show’s entries, the outlog needs to be trimmed/reset or if the period of time specified as the “Outlog Change History in Minutes” (by default 7 days) has passed since the IFM media was generated.
      Changes the Outlog Change History In Minutes value in the following registry subkey:
      HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesNtFrsParameters to 0 (zero)
    • Run ntfrsutl poll /now
    • Restart the FRS Service on the actual DC/FRS replica: net stop ntfrs | net start ntfrs
    • Run the ntfrsutil outlog again, the contents of the current outbound log must contain only files that have been modified after you changed the registry and restarted the FRS.

      Note: Don’t forget to reset the Outlog Change History In Minutes registry type back to the seven-day default while you’re done with the IFM operations (e.g. all DCs intended to be promoted with IFM has been promoted)

  • Configure Debug and Analysis logging on the computer that is to be promoted using IFM

    Configure Debug Severity on the computer that is about to be promoted using IFM media:

    • To be able to determine whether files in the SYSVOL tree are being moved in from the pre-staged folder on the local computer or are being replicated over the network from an upstream partner, set the registry value for Debug Log Severity to 4 on the computer being promoting using IFM media in the following registry subjey:
      HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesNtFrsParameters to 4
      Note: This has to be configured before you promote the computer to a DC using IFM.
    • Verify whether the files in the SYSVOL tree was seeded from the pre-stage folder (the restored IFM media) or replicated over the network.

To find all the files that were replicated from the initial replication partner over the wire (files that wasn’t seeded from IFM), type:

findstr /I “RcsReceivingStageFile” NtFrs_000X.log where X should be the number of the log, in case if multiple logs e.g. NtFrs_0001.log, NtFrs_0002.log etc run the command against both files.

To find all files that were sourced from the pre-staged system state backup, type:

Findstr /I “(218)” NTFRS_000X.log where X should be the number of the log, in case if multiple logs e.g. NtFrs_0001.log, NtFrs_0002.log

Posted on

ConfigMgr: Task sequence that installs many updates stops responding


Back in
2007 I was working on a fairly large System Center Configuration Manager 2007
project (an upgrade from SMS2003 with 60 000 + clients) and a Windows 2000/Windows
XP to Windows Vista upgrade, The project last for 2 entire years before it’s completion
and I decide it to be my first and last ConfigMgr project for me at least in
the Enterprise market. However one thing we never got satisfied with was the
ability to install software updates during an operating system deployment (OSD)
while we were capturing our reference images (we actually did this daily during
the development phase) – the result was very inconsistent: Sometimes it did
work like a charm, other days we could have a few updates integrated other
none.

Microsoft has now finally released a hotfix to resolve this issue and at least
it works out for me J

 

Download: Task sequence that installs many updates stops responding on a System Center
Configuration Manager 2007 SP2 client