Month: February 2012

So what is the “Code [1-3]” all about and where is Part 4 of the series that you might expect?
Before I go ahead with Part 4 I thought it would be a good idea to sum up Part 1 to Part 3 with code (So that you know how we figured out all this stuff while we was coding on ESEDump) – Note: This most may be targeted for the developer audience more than the general Active Directory administrator.

Disclaimer: The code samples provided here is code snippets that doesn’t represent any code actual code from the DSA and or any other Microsoft products and technologies, nor those they represent the complete source of ESEDump

ESEHelper – A managed ESE wrapper around the ESE APIs

We decided that we wanted to work with ESE in C# (and when we first started this project EseManaged from codeplex wasn’t around) and even if we could have used it later on – I guess we wanted full control and decided to stick with our own wrapper. So when you see references to “EseHelper” in the code snippets below – you know it’s just a wrapper around: Extensible Storage Engine Native APIs – there is no secrets around this J

JET_RETRIVECOLUMN structure custom methods

We extended the JET_RETRIVECOLUMN structure with some additional methods to retrieve data.

Table 0: JET_RETRIVECOLUMN structure

Perform Initialization and Attach to NTDS.dit

First thing we had to figure out was how we attached to the database (NTDS.dit) using JetInit, JetBeginSession, JetAttachDatabase and finally calling JetOpenDatabase in addition to those callas we had to set several parameters with JetSetSystemParameter for our usage, e.g there is things that need to be turned off as we attach/open the DB as read-only due to the nature of our application.

Table 1: ESE Initialization

List the tables inside NTDS.dit

We figured out that by statically opening the “MSysObjects” and positioning over the “RootObjects” index we could enumerate the tables inside the database using the following code snippet.

Table 2: ESE Enumerate tables inside the database

Retrieving the Ancestors_col

In Part 3, we’re discussing the usage of the “Ancestors_col” column and how it’s used to walk subtrees in the database, the DNTs are stored as bytes within the “Acenstors_col” and are read as in the code snippet below.

Table 3: Ancestors_col

Reading an object’s full distinguished name

Note: This is our way to read an objects full distinguished name, given an object’s DNT (Distinguished Name Tag) – But this is not considered safe by the DSA as mentioned in Part 3 as the “Ancestors_col” is being processed by a background task and might not be in-sync all the times. (Safer would be to walk the tree up – by each PDNT until PDNT == 2)

Table 4: Get an objects distinguished name by its DNT

Table 5: Get an objects distinguished name (object is referenced by sAMAccountName: ADCH)

ESEDump

This is the third post in a series of articles that will describe what’s really inside NTDS.dit and how Active Directory works on the database layer, the past two articles has been about:

• Explains the tables within NTDS.dit in detail as far as what they are used for, in which release of Active Directory (Windows Server) they were introduced in, as well any major changes being added in later versions: How the Active Directory – Data Store Really Works (Inside NTDS.dit) – Part 1

• Explains how the object tree hierarchy is maintained at the database layer and the concept of DNTs and PDNTs and how they make up the relation between parent and descendent objects: How the Active Directory – Data Store Really Works (Inside NTDS.dit) – Part 2

Support for onelevel searches

Given the knowledge from the last article in the series on how objects are referred to each other in terms of a parent – child relation (DNT and PDNT) it becomes obvious that it is easy to get all direct-descendent/child objects of a given object by searching for all object’s (rows) that have a specific PDNT (where objects.PDNT == parent.DNT) this can efficiently be archived by using the “PDNT_index”

Table 1: PDNT_Index

[1.1]: “Name” represent the RDN attribute, it’s not stored/named as described in the illustration above, it’s rather stored as “ATTm589825” I choose to represent it as “Name” for simplifying the understanding

Introducing the Ancestors_col and support for subtree searches

Support for subtree searching requires the implementation of another column at the DBLayer, the “Ancestors_col”; the other columns are we already familiar with as of the last post in the series

Table 2: datatable – Simplified for hierarchy representation 2

[2.1]: The first object/row within the “datatable” always has a NULL ancestory_col value.
[2.2] See “Table 4: Ancestors_index”

Let’s apply “Table 2” to a theoretical sample:

[3.1]: “Name” represent the RDN attribute, it’s not stored/named as described in the illustration above, it’s rather stored as “ATTm589825” I choose to represent it as “Name” for simplifying the understanding

If we wanted to do an subtree search with the “Windows Development” organizational unit as the search base – we would like to use an index over the “ancestors_col” [4.1] and set the prefix to be “2,1787,1788,1789,1790,5520,5521*” that would return a list of all objects (rows) that are subordinated the “Windows Development” organizational unit (e.g. all children).

Table 4: Ancestors_index

The ancestors_col and the SDProp (Security Descriptor Propagation Demon) – How are they related?

The “ancestors_col” complicates an object move (Within the same NC/Database, We leave cross-NC/database moves outside this article for simplify understanding)

Given the knowledge from the last article in the series on how objects are referred to each other in terms of a parent – child relation (DNT and PDNT) – It seems easy to implement an object move by simply change the PDNT to the DNT of the new designated parent (e.g. give “Christoffer Andersson” a value of “5521” in his “PDNT_col” and the object is now subordinated the organizational unit “Windows Development” instead of “Users” in the above sample)

After the operation above, the “Ancestors_col” wouldn’t be accurate, so the Ancestors_col needs “fixup”, (e.g. it needs to be adjusted to the new path “2,1787,1788,1789,1790,5520,5521,,5524” – “5522” has to be removed as we moved the object “Christoffer Andersson” from the “Users” organizational unit to the “Windows Development” organizational unit) This might not seem to be an issue at the first glance, but imagine moving a large subtree – the operation wouldn’t fit into an single atomic transaction, therefore ancestry fixup is performed in the background by the SDProp (Security Descriptor Propagation Demon) [4.1]

The most experienced Active Directory administrators at least have heard of “SDProp” that is a short for the Security Propagation Demon and some know that it’s responsible for handling propagation of ACE inheritance, very few probably know that it is in addition also responsible to maintain the acenstors_col at the DBLayer.

Each DSA/DC runs the SDProp (Security Descriptor Propagation Demon) as a background task (TQ_TASK). By default, this task is triggered by the following conditions:

• Any modification (originating or replicated) of the nTSecurityDescriptor attribute of any object (Except for those modifications done by the SDProp demon)
  • This requires that any new/modified inheritable ACEs are propagated to all descendant objects and that any removed inheritable ACEs are removed from all descendent objects, ACE inheritance is not replicated and is being applied by the SDProp using this process on all DSAs/DCs.
• Any modification of the “PDNT_col” (object is being moved in the directory) of an object that results in the object having a different parent (Except for those cases where the new parent is a Deleted Objects container)
  • This requires adjustments of the values (DNTs) stored in the “ancestors_col” as the object/row has been moved.
  • This requires that all inheritable ACEs present on the new parent object (and all its ancestors from the top) are being propagated down to the object, and that inheritable ACEs originating from the previous parent (and all its ancestors from the top) has to be removed.

    
    Table 5: Security Descriptor Propagation Demon – Responsibilities Summary
    

[4.1] Since this work is performed in the background and SDProp (that runs on single thread doesn’t implement a limit of how many transactions that is used to perform an operation) there may be a period of time when Active Directory returns inconsistent query results with the tree structure, and inheritable ACEs won’t be accurate.

[4.2]: A common misunderstanding is that the SDProp is responsible for maintaining protection for object’s that is protected by the AdminSDHolder, That’s incorrect. The AdminSDHolder runs in its own background task (TQ_TASK) every 60 minutes on the PDC (if a protected object’s security descriptor mismatches from the once at the adminSDHolder object, inheritance is turned off and the security descriptor is over written with the one at the adminSDHolder object) that makes the AdminSDHolder background task to trigger off SDProp .

You can read more about the SDProp – Security Descriptor Propagation Demon here:
http://msdn.microsoft.com/en-us/library/dd350247(v=prot.13).aspx

Trigger the SDProp from manually from the outside.

It’s possible to use an operational attribute to trigger the SDProp demon to run from the “outside” (e.g be performed by an administrator) – How ever on a fully functional DSA/DC there is no need to invoke the SDProp manually, the SDProp can be trigged globally or for a specific object /row identified by its DNT, The requester must have the “Recalculate-Security-Inheritance” control access right on the nTDSDSA object for the DSA/DC

The following shows an LDIF sample that performs this operation on the entire DIT.

dn:
changetype: modify
add: fixupInheritance
fixupInheritance: 1

The following shows an LDIF sample that performs this operation on a specific object /row

dn:
changetype: modify
add: fixupInheritance
fixupInheritance: dnt:5524

You can read more about the “fixupInheritance” operational attribute here: (it’s actually well documented now days)
http://msdn.microsoft.com/en-us/library/cc223299(v=prot.13).aspx

I think that’s all regarding the “ansectors_col” – I might have missed something.

This is the second post in a series of articles that will describe what’s really inside NTDS.dit and how Active Directory works on the database layer, In an earlier post I explained the tables within NTDS.dit in detail as far as what they are used for, in which release of Active Directory (Windows Server) they were introduced in, as well any major changes being added in later versions: How the Active Directory – Data Store Really Works (Inside NTDS.dit) – Part 1

This post will go into the details of the contents of the “datatable” also known as the object store – that contains all objects and phantoms [1.1] represented as rows (1 object/phantom = 1 row in the table) from any instanced naming context (NC) held as either writable or read-only (until they are physically removed by the garbage-collector) by the Directory System Agent (DSA) hosting the database and where columns represent every [1:3] attribute present in the schema except linked attributes [1:2]

[1.1]: phantoms are references to object’s hosted outside the given database (NTDS.DIT) and the given Directory System Agent (DSA) – (Except structural phantoms)

[1:2] Post-Windows Server 2003 the attribute “ntSecurityDescriptor” is stored in the “sd_table” rather than in the “datatable”

[1:3] Some columns doesn’t reflect attributes and are columns pre-defined in the NTDS.dit template database generated by Microsoft (those are needed for internal states to the DSA)

Maintain the hierarchy of an object tree within a flat object store

The hierarchy in Active Directory is quite obvious to most of us at a simplified layer e.g. daily administrative task such as creating an Organizational Unit and creates several descendent/child objects under neat it, some people may refer to some objects as leaf objects (object that usually don’t contain descendent/child objects) such as object of the class “user” – However the fact is that any object within Active Directory has the possibility (technically) to contain one or more descendent/child objects – this is controlled by schema constrains and more specifically the sum of the following attributes of a given object class and any inherited class (except for auxiliary classes) :

Table 1: Possible Superiors

Why it’s easy for all objects to host descendants/child objects becomes more obvious when the hierarchy is explained at the DBLayer.

The question remains with the details given above, if one row within the “datatable” represents an object/phantom, how can the hierarchy be maintained? The below table “Table 2” represent columns in the “datatable” that are vital for representing/building the hierarchy in the directory at the DBLayer.

Table 2: datatable – Simplified for hierarchy representation 1

[2.1]: The maximum numbers of objects/phantoms that ever can be created on a given DSA (Domain Controller) for its entire life time is 2 billion objects (147,483,393 (231 minus 255)). Note that this count against objects/phantoms ever introduced to the local DSA as part of any naming context (NC) writable or partial ever hosted by the DSA. * If the DSA is promoted by using IFM (Install from Media it inheritance the count of already allocated DNTs from the former DSA) – When the maximum numbers of auto-increment values has been used (the limit mention above have been hit) the following error are returned at the DBLayer: JET_errOutOfAutoincrementValues -1076 from the outside we will notice: “Error: Add: Operations Error. <1> Server error: 000020EF: SvcErr: DSID-0208044C, problem 5012 (DIR_ERROR), data -1076.” Read more about Active Directory Limits: http://technet.microsoft.com/en-us/library/active-directory-maximum-limits-scalability(v=ws.10).aspx#BKMK_Objects

[2.2]: The first row introduced in the “datatable” isn’t a real object nor is it a phantom and is named “$NOT_AN_OBJECT1$” and have its PDNT_col set to NULL.
The “PDNT_col” is indexed so becomes very easy to drive an object’s direct-descendants/child objects (not all descendants) by simply query who that has the object’s DNT in their PDNT_col.

[2:3] The DSA has an in-memory cache for the most common RDNs (the ones mentioned above)
Active Directory allows us to use a custom attribute as RDN as well by specify the attributeID of that custom attribute as the rDNAttID for the particular class. * A RDN attribute must have the syntax string **customization may not be supported by all LDAPv3 clients. *** rDNAttID should preferably be set before any objects of the given class is instanced in the directory (changes won’t apply to already instanced/existing objects)

Let’s apply “Table 2” to a theoretical sample:

[3.1]: “Name” represent the RDN attribute, it’s not stored/named as described in the illustration above, it’s rather stored as “ATTm589825” I choose to represent it as “Name” for simplifying the understanding of the hierarchy in this case.

[3.2] Structural Phantom – (Different from a phantom used for reference integrity to real object’s hosted outside the given DIT) is used to represent the full distinguished name of the domain e.g “DC=ntdev,DC=corp,DC=chrisse,DC=com”

In the next article – we will continue the deep-dive into the content and the structure of the “datatable” – going thru things like ancestors, the difference between phantoms and real objects, tombstones and the garbage collector on the DBLayer and much more.

You might as I have asked yourself many times – What is inside NTDS.dit? (Most experienced Active Directory admins knows that NTDS.dit is the database and the physical on disk store that Active Directory uses to store information – most of you have probably got in touch with NTDS.dit during backup and restore scenarios)

Long story in a short version – I wasn’t satisfy not knowing – neither was I after being reading the following article:
(That I actually think isn’t that bad – but is also probably the most detailed public available information on the subject)
[1] http://technet.microsoft.com/en-us/library/cc772829(WS.10).aspx

So I decided with a very good friend of mine Stanimir Stoyanov (Microsoft Visual C# MVP) to go ahead and build a tool that could read NTDS.dit and decode its internals, and then we started a journey that has given us invaluable knowledge at this part of Active Directory, this is the first article in a series of articles that will describe what’s really inside NTDS.dit and how Active Directory works on the database layer.

The illustration below has been presented in various documentations since Active Directory was initially released over 10 years ago; a similar illustration is also available in (However after this research project it’s actually turning out to be inaccurate in some aspects – in the way the DRA/REPL communicates with the DBLayer) [1]

Table 1: DSA Components (Simplified for the DBLayer)

Data Store Physical Structure / Inside NTDS.dit – Tables

Finally we can start looking into the content/internal structure of NTDS.dit – but first let’s take a look on what has been reveled before, the illustration below is from [1] and is accurate as far as outside the white box that represent the tables within the database, the tables do exist (Except for * “sd_table” on Windows 2000 DSAs) – but there is more tables that isn’t mentioned in this example.

So it’s about time to reveal the real table structure of an NTDS.dit database file – It’s time to use the tool we produced to first discover this:

Table 2: NTDS.DIT – Tables

In the next article – we will take a deep-dive into the content and the structure of the “datatable” also known as the object-store.