You might as I have asked yourself many times – What is inside NTDS.dit? (Most experienced Active Directory admins knows that NTDS.dit is the database and the physical on disk store that Active Directory uses to store information – most of you have probably got in touch with NTDS.dit during backup and restore scenarios)
Long story in a short version – I wasn’t satisfy not knowing – neither was I after being reading the following article:
(That I actually think isn’t that bad – but is also probably the most detailed public available information on the subject)
[1] http://technet.microsoft.com/en-us/library/cc772829(WS.10).aspx
So I decided with a very good friend of mine Stanimir Stoyanov (Microsoft Visual C# MVP) to go ahead and build a tool that could read NTDS.dit and decode its internals, and then we started a journey that has given us invaluable knowledge at this part of Active Directory, this is the first article in a series of articles that will describe what’s really inside NTDS.dit and how Active Directory works on the database layer.
The illustration below has been presented in various documentations since Active Directory was initially released over 10 years ago; a similar illustration is also available in (However after this research project it’s actually turning out to be inaccurate in some aspects – in the way the DRA/REPL communicates with the DBLayer) [1]
Table 1: DSA Components (Simplified for the DBLayer)
|
Component |
Description |
| Ntdsa.dll – Directory System Agent | The DSA, which runs as Ntdsa.dll on each domain controller, provides the interfaces through which directory clients and other directory servers gain access to the directory database (the DBLayer). In addition, the DSA enforces directory semantics, maintains the schema, guarantees object identity, and enforces data types on attributes. |
| Esent.dll – Extensible Storage Engine (ESE) APIs | The Extensible Storage Engine (ESE) is an advanced indexed and sequential access method (ISAM) storage technology. ESE enables applications to store and retrieve data from tables using indexed or sequential cursor navigation. It supports denormalized schemas including wide tables with numerous sparse columns, multi-valued columns, and sparse and rich indexes. It enables applications to enjoy a consistent data state using transacted data update and retrieval.
ESE was formerly known as Joint Engine Technology (JET) Blue, The DBLayer uses the ESE APIs documented here: http://msdn.microsoft.com/en-us/library/windows/desktop/gg269259(v=exchg.10).aspx |
| NTDS.dit | The on physical-disk file that represent the ESE/JetBlue database that holds the information store for the given DSA/Active Directory Domain Controller. |
Data Store Physical Structure / Inside NTDS.dit – Tables
Finally we can start looking into the content/internal structure of NTDS.dit – but first let’s take a look on what has been reveled before, the illustration below is from [1] and is accurate as far as outside the white box that represent the tables within the database, the tables do exist (Except for * “sd_table” on Windows 2000 DSAs) – but there is more tables that isn’t mentioned in this example.
So it’s about time to reveal the real table structure of an NTDS.dit database file – It’s time to use the tool we produced to first discover this:
Table 2: NTDS.DIT – Tables
| Table |
Description |
Minimum DSA Version |
| Datatable | Contains all objects and phantoms [2.1] represented as rows (1 object/phantom = 1 row in the table) from any instanced naming context (NC) held as either writable or read-only by the Directory System Agent (DSA) hosting the database and where columns represent every [2:3] attribute present in the schema except linked attributes [2:2]
[2.1]: phantoms are references to object’s hosted outside the given database (NTDS.DIT) and the given Directory System Agent (DSA) [2:2] Post-Windows Server 2003 the attribute “ntSecurityDescriptor” is stored in the “sd_table” rather than in the “datatable” [2:3] Some columns doesn’t reflect attributes and are columns pre-defined in the NTDS.dit template database generated by Microsoft (those are needed for internal states to the DSA) |
Windows 2000 Server
Note: Windows Server 2008 R2 added a column to support the “is-Recycled” state |
| Hiddentable | Contains one row but several columns that defines the state of the database as well the [2:2] DNT (reference) of the NTDSA-Settings object that represents this DSA (used for finding config information specific to this domain controller.)
[2:4] The concept of DNTs (Distinguished Name Tags) |
Windows 2000 Server Note: Windows Server 2003 Introduced additional state columns such as backupexpiration_col |
| Link_table | Contains link-pair references (DNT, DNT), the link base (link id >> 1) and possibly a binary blob (In case of DN-binary, DN-string syntax) | Windows 2000 Server
Note: Windows Server 2008 R2 added a column to support deactivated links for recycle-bin |
| Sd_table | Contains single-instance-stored SDs (Security Descriptors) that pre-Windows Server 2003 was stored in the ntSecurityDescriptor attribute in the “datatable” – those are now instead referenced to the SDs in the “sd_table” that is, if more than one object has exactly the same security defined (Security Descriptor) both objects are referenced to the same row in the “sd_table”, hence the single-instance-storage and reducing the size needed to store Security Descriptors. | Windows Server 2003. |
| Sdpropcounttable | Used by the Security Descriptor Propagation Demon (SDProp) responsible for Security Descriptor inheritance down the tree, within the local database | |
| Sdproptable | Used by the Security Descriptor Propagation Demon (SDProp) responsible for Security Descriptor inheritance down the tree, within the local database | Windows 2000 Server |
| Quota_rebuild_progress_table | Contains temporary information during quota tracking rebuild, for the Active Directory quota feature introduced in Windows Server 2003 – this allows the demon to keep track of processed objects. | Windows Server 2003 |
| Quota_table | Contains quota tracking information, for the Active Directory quota feature introduced in Windows Server 2003, quota tracking is peer naming context (NC) and for a given security principal identified by its SID. | Windows Server 2003 |
| MSysObjects | ESE Internals – out of scope for this article | N/A |
| MSysObjectsShadow | ESE Internals – out of scope for this article | N/A |
| MSysUnicodeFixupVer2 | ESE Internals – out of scope for this article | N/A |
In the next article – we will take a deep-dive into the content and the structure of the “datatable” also known as the object-store.
Hi Christoffer,
I have a doubt about AD database. If ntds.dit is a database, why can’t we use MS SQL database for AD ? Why do we need another database for AD?
Venkat.
Active Directory was being developed for quite some years before its initial release in Windows 2000 Server and is based on ESE97 (even that it uses some new features in more recent releases of ESE) at that time (in 1996 where the development started) I guess only SQL Server 6.0 was available at that time. SQL faced a limitation until some recent release (if I recall correctly) on numbers of columns in a single table that ESE overcomes and that was necessary for the X.500 data store. I know that tests has been made to put AD on SQL and the results shown degradation in performance, other than that it’s a license discussion internally at Microsoft (as it would require windows to ship with SQL Server to put AD on it) – I can pretty much guarantee that we never will see Active Directory (as we know it today) on SQL Server, and I don’t see any need for it either ESE dose it work perfectly fine.
Can we use ESE without AD just like SQL DB? Say for my new application can I use ESE instead of SQL DB?
Kindly ignore my ignorance as I am new to this field?
Thanks,
Manohar
Yes, sure, have a look at http://managedesent.codeplex.com/
Hi, i wish to see the file that contains the password history.. where do i find that…just a curious questions as i have set some 12 pwds and i need to know where that file is stored.
Hari
Hi Christopher !
I have just started to read AD DS and got attracted to it. Its beautiful.I had worked with IBM for 2 years on DRA without knowing what I was doing . (I was playing around with AD I guess). Now I am going through a detailed study about AD DS. I want to pursue it until end.
I would appreciate if you walk me through the path of learning AD DS step by step !
Tnx,
Anil
Hi Christoffer,
As we know, restoring AD requires System State backup so I have a question in mind what is architecture level difference in system state backup than full or any other backup? It would be great if you could help on this.
Thanks in advance.
Abhijit W.
A system state restore is performing a call into the database to make it a aware that it has been restored before the DS is started.
Excellent dear, nice information. I didn’t know this before.
Thanks