{"id":762,"date":"2012-02-11T01:22:00","date_gmt":"2012-02-11T01:22:00","guid":{"rendered":"http:\/\/web03.partners.extranet.chrisse.com\/wordpress\/?p=762"},"modified":"2023-06-06T17:06:31","modified_gmt":"2023-06-06T15:06:31","slug":"how-the-active-directory-data-store-really-works-inside-ntds-dit-part-1","status":"publish","type":"post","link":"https:\/\/blog.chrisse.se\/?p=762","title":{"rendered":"How the Active Directory \u2013 Data Store Really Works (Inside NTDS.dit) \u2013 Part 1"},"content":{"rendered":"\n

You might as I have asked yourself many times \u2013 What is inside NTDS.dit? (Most experienced Active Directory admins knows that NTDS.dit is the database and the physical on disk store that Active Directory uses to store information \u2013 most of you have probably got in touch with NTDS.dit during backup and restore scenarios)<\/p>\n\n\n\n

Long story in a short version \u2013 I wasn’t satisfy not knowing \u2013 neither was I after being reading the following article:
(That I actually think isn’t that bad \u2013 but is also probably the most detailed public available information on the subject)
[1] http:\/\/technet.microsoft.com\/en-us\/library\/cc772829(WS.10).aspx<\/a><\/p>\n\n\n\n

So I decided with a very good friend of mine Stanimir Stoyanov<\/a> (Microsoft Visual C# MVP) to go ahead and build a tool that could read NTDS.dit and decode its internals, and then we started a journey that has given us invaluable knowledge at this part of Active Directory, this is the first article in a series of articles that will describe what’s really inside NTDS.dit and how Active Directory works on the database layer.<\/p>\n\n\n\n

The illustration below has been presented in various documentations since Active Directory was initially released over 10 years ago; a similar illustration is also available in (However after this research project it’s actually turning out to be inaccurate in some aspects \u2013 in the way the DRA\/REPL communicates with the DBLayer) [1]<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Table 1: DSA Components (Simplified for the DBLayer)
<\/span><\/p>\n\n\n\n

\n<\/colgroup>\n\n\n\n\n\n
\n

Component<\/span><\/p>\n<\/td>\n

\n

Description<\/span><\/p>\n<\/td>\n<\/tr>\n

Ntdsa.dll \u2013 Directory System Agent<\/span><\/td>\nThe DSA, which runs as Ntdsa.dll on each domain controller, provides the interfaces through which directory clients and other directory servers gain access to the directory database (the DBLayer). In addition, the DSA enforces directory semantics, maintains the schema, guarantees object identity, and enforces data types on attributes.<\/span><\/td>\n<\/tr>\n
Esent.dll \u2013 Extensible Storage Engine (ESE) APIs<\/span><\/td>\nThe Extensible Storage Engine (ESE) is an advanced indexed and sequential access method (ISAM) storage technology. ESE enables applications to store and retrieve data from tables using indexed or sequential cursor navigation. It supports denormalized schemas including wide tables with numerous sparse columns, multi-valued columns, and sparse and rich indexes. It enables applications to enjoy a consistent data state using transacted data update and retrieval.
<\/span>\n

 <\/p>\n

ESE was formerly known as Joint Engine Technology (JET) Blue, The DBLayer uses the ESE APIs documented here: http:\/\/msdn.microsoft.com\/en-us\/library\/windows\/desktop\/gg269259(v=exchg.10).aspx<\/a>
<\/span><\/p>\n<\/td>\n<\/tr>\n

NTDS.dit<\/span><\/td>\nThe on physical-disk file that represent the ESE\/JetBlue database that holds the information store for the given DSA\/Active Directory Domain Controller.<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n\n\n\n

Data Store Physical Structure \/ Inside NTDS.dit – Tables
<\/strong><\/p>\n\n\n\n

Finally we can start looking into the content\/internal structure of NTDS.dit \u2013 but first let’s take a look on what has been reveled before, the illustration below is from [1] and is accurate as far as outside the white box that represent the tables within the database, the tables do exist (Except for * “sd_table” on Windows 2000 DSAs) \u2013 but there is more tables that isn’t mentioned in this example.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

So it’s about time to reveal the real table structure of an NTDS.dit database file \u2013 It’s time to use the tool we produced to first discover this:<\/p>\n\n\n\n

Table 2: NTDS.DIT – Tables
<\/span><\/p>\n\n\n\n

\n<\/colgroup>\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Table<\/span><\/td>\n\n

Description<\/span><\/p>\n<\/td>\n

\n

Minimum DSA Version<\/span><\/p>\n<\/td>\n<\/tr>\n

Datatable<\/span><\/td>\nContains all objects and phantoms [2.1] represented as rows (1 object\/phantom = 1 row in the table) from any instanced naming context (NC) held as either writable or read-only by the Directory System Agent (DSA) hosting the database and where columns represent every [2:3] attribute present in the schema except linked attributes [2:2]<\/span>\n

 <\/p>\n

[2.1]: phantoms are references to object’s hosted outside the given database (NTDS.DIT) and the given Directory System Agent (DSA)<\/p>\n

[2:2] Post-Windows Server 2003 the attribute “ntSecurityDescriptor” is stored in the “sd_table” rather than in the “datatable”<\/p>\n

[2:3] Some columns doesn’t reflect attributes and are columns pre-defined in the NTDS.dit template database generated by Microsoft (those are needed for internal states to the DSA)<\/p>\n<\/td>\n

Windows 2000 Server
<\/span>\n

 <\/p>\n

Note: Windows Server 2008 R2 added a column to support the “is-Recycled” state<\/span><\/p>\n<\/td>\n<\/tr>\n

Hiddentable<\/span><\/td>\nContains one row but several columns that defines the state of the database as well the [2:2] DNT (reference) of the NTDSA-Settings object that represents this DSA (used for finding config information specific to this domain controller.)<\/span>\n

 <\/p>\n

[2:4] The concept of DNTs (Distinguished Name Tags)<\/p>\n<\/td>\n

Windows 2000 Server
Note: Windows Server 2003 Introduced additional state columns such as backupexpiration_col<\/span><\/td>\n<\/tr>\n
Link_table<\/span><\/td>\nContains link-pair references (DNT, DNT), the link base (link id >> 1) and possibly a binary blob (In case of DN-binary, DN-string syntax)<\/span><\/td>\nWindows 2000 Server<\/span>\n

 <\/p>\n

Note: Windows Server 2008 R2 added a column to support deactivated links for recycle-bin<\/p>\n<\/td>\n<\/tr>\n

Sd_table<\/span><\/td>\nContains single-instance-stored SDs (Security Descriptors) that pre-Windows Server 2003 was stored in the ntSecurityDescriptor attribute in the “datatable” \u2013 those are now instead referenced to the SDs in the “sd_table” that is, if more than one object has exactly the same security defined (Security Descriptor) both objects are referenced to the same row in the “sd_table”, hence the single-instance-storage and reducing the size needed to store Security Descriptors.<\/span><\/td>\nWindows Server 2003.<\/span><\/td>\n<\/tr>\n
Sdpropcounttable<\/span><\/td>\nUsed by the Security Descriptor Propagation Demon (SDProp) responsible for Security Descriptor inheritance down the tree, within the local database<\/span><\/td>\n <\/td>\n<\/tr>\n
Sdproptable<\/span><\/td>\nUsed by the Security Descriptor Propagation Demon (SDProp) responsible for Security Descriptor inheritance down the tree, within the local database<\/span><\/td>\nWindows 2000 Server<\/span><\/td>\n<\/tr>\n
Quota_rebuild_progress_table<\/span><\/td>\nContains temporary information during quota tracking rebuild, for the Active Directory quota feature introduced in Windows Server 2003 \u2013 this allows the demon to keep track of processed objects.<\/span><\/td>\nWindows Server 2003<\/span><\/td>\n<\/tr>\n
Quota_table<\/span><\/td>\nContains quota tracking information, for the Active Directory quota feature introduced in Windows Server 2003, quota tracking is peer naming context (NC) and for a given security principal identified by its SID.<\/span><\/td>\nWindows Server 2003<\/span><\/td>\n<\/tr>\n
MSysObjects<\/span><\/td>\nESE Internals \u2013 out of scope for this article<\/span><\/td>\nN\/A<\/span><\/td>\n<\/tr>\n
MSysObjectsShadow<\/span><\/td>\nESE Internals \u2013 out of scope for this article<\/span><\/td>\nN\/A<\/span><\/td>\n<\/tr>\n
MSysUnicodeFixupVer2<\/span><\/td>\nESE Internals \u2013 out of scope for this article<\/span><\/td>\nN\/A<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n\n\n\n

In the next article \u2013 we will take a deep-dive into the content and the structure of the “datatable” also known as the object-store.<\/p>\n","protected":false},"excerpt":{"rendered":"

You might as I have asked yourself many times \u2013 What is inside NTDS.dit? (Most experienced Active Directory admins knows that NTDS.dit is the database and the physical on disk store that Active Directory uses to store information \u2013 most of you have probably got in touch with NTDS.dit during backup and restore scenarios) Long … <\/p>\n

Continue reading “How the Active Directory \u2013 Data Store Really Works (Inside NTDS.dit) \u2013 Part 1”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[1],"tags":[6,14,16],"class_list":["post-762","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-active-directory","tag-extensible-storage-engine-ese","tag-ntds-dit"],"_links":{"self":[{"href":"https:\/\/blog.chrisse.se\/index.php?rest_route=\/wp\/v2\/posts\/762","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.chrisse.se\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.chrisse.se\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.chrisse.se\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.chrisse.se\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=762"}],"version-history":[{"count":2,"href":"https:\/\/blog.chrisse.se\/index.php?rest_route=\/wp\/v2\/posts\/762\/revisions"}],"predecessor-version":[{"id":1072,"href":"https:\/\/blog.chrisse.se\/index.php?rest_route=\/wp\/v2\/posts\/762\/revisions\/1072"}],"wp:attachment":[{"href":"https:\/\/blog.chrisse.se\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=762"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.chrisse.se\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=762"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.chrisse.se\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=762"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}