How install from media (IFM) really works (Part 1)

This is the first blog post in a series of posts covering how the “Install from media” feature really works, it’s an in-depth very technical post that explains what happens under the hood and this first part focuses on how it works in Windows Server 2003.

Background

Install from media was first introduced in Windows Server 2003, as a solution to improve the installation experience of newly promoted domain controllers in branch offices mainly (or sites with slow-links where the initial replication could take significant time to complete), but it is actually an important component in many disaster recovery plans I have designed for various customers over the years, As it is a fast and efficient way to re-install a domain controller and get it up to sync, (that’s the proper way to handle a faulting replicas/domain controllers in most cases). There is some common misunderstandings of the concept “Install from media” I terms of if the operation could be performed entirely offline or online, the short answer is: No. It can’t be performed offline; you have to be online with at least one writable domain controller in the same domain as the IFM source is taken from and even then you may not be able to be fully efficient and cause replication to happen over the network anyway, this need some future explanation.

What dose Install from media (IFM) consist of

Install from media (IFM) contains two important things.

• NTDS.DIT (Active Directory Database) – at the time the IFM is generated (Regardless of Windows Server 2003, Windows Server 2008 or later –the NTDS.dit is pretty much unchanged until DCPROMO makes a lot of changes at the becoming domain controller that takes use of the database – it will change the DSA reference and update related “instance specific” information in the hidden table )
• SYSVOL (SYSVOL GPT Storage)
• Registry (Contains the SYSKEY used to decrypt the PEK (also known as Password Encryption Key) that efficiently ensure that the protection for sensitive information stored in the Active Directory database (Such as Password Hashes) are unique to each instance of the database (read each domain controller) –Note: This doesn’t apply to RODCs .

Sourcing install from media (IFM) using System State and VSS

Sourcing the media used by IFM is different in Windows Server 2003 (all versions) and Windows Server 2008 and later, the difference is the technology used to gather information required.

• Windows Server 2003 IFM media is generated by performing a system state backup, the reason for this is that we can get a copy of ntds.dit while we’re up and running DsIsNTDSOnline=True (Active Directory is operational) this is archived by the DsBackup API: http://msdn.microsoft.com/en-us/library/ms675896(VS.85).aspx.We will also get a copy of SYSVOL since a system state backup contains the following:
  • Active Directory
  • The SYSVOL tree
  • The Boot.ini file
  • The COM+ class registration database
  • The registry

  To be more specific the following is required in the system state in order to be able to source IFM:

  • Active Directory is required.
  • The SYSVOL tree may be optionally removed. (A specific configuration is required to source the SYSVOL tree during IFM promotion)
  • The Boot.ini file may be removed.
  • The COM+ class registration database may be removed.
  • The registry folder is required. Registry components are required as follows:
    • The Default file in the Registry folder may be removed.
    • The SAM file is required.
    • The SECURITY folder is required.
    • The SOFTWARE file may be removed.
    • The SYSTEM file is required.

  We’re are responsible for doing a restore of a system state backup or selves to an alternative location, and ensure we gather the required information above. (Optionally if we care about disk size optimization we can not only select the specific components required in the system state backup, but we could also perform a offline defragmentation of the ntds.dit database)

  We can only use IFM to promote domain controllers in the same domain as we sourced it from, as well the target domain controller has to be running the same operating system, including service pack and architecture (x86/x64)

  Sourcing the NTDS.DIT – Active Directory database from IFM in Windows Server 2003
  

  As explained earlier in this article the Active Directory database (NTDS.dit) is being backup either by the DsBackup API or VSS API – The Active Directory database are unique to each DC and contains the NCs hosted by the DC typically Domain, Schema, Configuration and in some cases NDNC’s also known as application partitions such as DomainDNSZones and ForestDNSZones, if the DC is also a GC it will contain partial information about every object in all domain NCs in the entire forest (a multiple domain forest) – even thou if a DC are a GC in a single domain environment, the domain NC is only stored once in the database, if the sourced database was a GC, the computer being promoted to a DC using the source will also become a GC.

  During the backup itself (even if the intent are for IFM) – No changes are made to the database except setting the backup expiration date and backup usn in the hidden record. (Also known has the hidden table). Changes are instead modified/adjusted during DCPROMO on the computer that are about to become a DC using the sourced database from IFM.

  Before we can have an in-depth look at the changes that take place inside the Active Directory database during IFM we need to understand the physical layout of the database and some key concepts. (Note: For the purpose of this article the NTDS.dit physical layout will be rather simplified than the real/exact layout for display purposes)

  The Active Directory database (NTDS.DIT) contains the following tables.

  Table 1: NTDS.DIT Database Layout (Simplified Version)
  

• Ensure the database isn’t out-dated aka older than Tombstone Life Time (TSL)
  During DCPROMO the sourced IFM database are verified that it hasn’t passed the Tombstone Life Time (TSL) in the forest, if the database is older than the TSL, the promotion is aborted. This is more specifically measured against the object who had the last USN Change and it’s when-changed date.

• Change/Adjust the DSA – Directory Service Agent IdentityThe “hiddentable” also known as the hidden record contains the identity of the local DSA – Directory Services Agent (DC) hosting the instance of the database, this identity points to the DCs NTDSA object within the directory stored within the “datatable”, the ntdsa object in its turn contains necessary information such as the DMD to be able to read the schema and soon. (I will write another article on how this works – more in detail). When it comes to IFM, the DSA identity stored in the hidden record/hidden table still points to the identity of the DC the IFM was sourced from, this cause an issue.The following illustrates the relationship between the DSA identity in the “hiddentable” and the NTDS Settings object stored in the “datatable” – that’s actually the NTDS Settings object you will see under the server object in Active Directory Sites and Services.

  Table 2: NTDS.DIT “datatale” (Simplified Version)
  

  So in this example NTTEST-SCH-01 is the DC where we sourced the IFM (backed up the database) – Now how do we get the new identity of the computer being promoted to a DC using the sourced IFM media? Well the new DSA for the computer being promoted is actually created remotely during DCPROMO on another DC (at the DC we perform the initial replication with, specified in the unattended answer file as “ReplicationSourceDC” parameter) , before we can change the record in the “hiddentable” to point to it – we must replicate in the newly created DSA by replicating the Configuration NC – this can cause an issue by itself – If we’re preforming the initial replication with the DC we sourced the IFM from, we’re temporarily presenting us self as the very same DC and replication will fail.

  So how are this solved? We’re creating a temporary “dummy” DSA with a corresponding server object in the database, retire invocation IDs, copying all references to NCs hosted by the old DSA (Doing not will have KCC to trigger a deletion of those sourced NCs later on), the temporary “dummy” DSA are created in the any site and the first site in alphabetical order identified.

  

  Now we can re-initialize and successfully replicate in the remotely created DSA by replicate the configuration NC using the temporary “dummy” DSA identity, retire invocation IDs, copying all references to NCs hosted by the temporary “dummy” DSA (Doing not will have KCC to trigger a deletion of those sourced NCs later on) we can now remove the temporary “dummy” DSA and store the real identity of the computer being promoted in the “hiddentable”

  

  We can now re-initialize as our self’s with our real DSA identity and continue processing.

• Remove None-Replicated attributes from the database.
  Non-replicated attributes (Containing bit 0x00000001 in System-Flags) , such as badPwdCount, Last-Logon, and Last-Logoff are stored on each domain controller, but are not replicated. The non-replicated attributes are attributes that pertain to a particular domain controller, as those attributes contains local information associated with the DC that the IFM was sourced from, those are deleted except the following:

1. DS-Core-Propagation-Data: http://msdn.microsoft.com/en-us/library/ms675655(v=VS.85).aspx
2. Obj-Dist-Name: http://msdn.microsoft.com/en-us/library/ms675516(v=VS.85).aspx
3. ms-DS-ReplicationEpoch: http://msdn.microsoft.com/en-us/library/ms677478(v=VS.85).aspx
4. Object-GUID: http://msdn.microsoft.com/en-us/library/ms679021(v=VS.85).aspx
5. Partial-Attribute-Deletion-List: http://msdn.microsoft.com/en-us/library/ms679106(v=VS.85).aspx
6. Partial-Attribute-Set: http://msdn.microsoft.com/en-us/library/ms679107(v=VS.85).aspx
7. Prefix-Map: http://msdn.microsoft.com/en-us/library/ms679371(v=VS.85).aspx
8. Repl-Property-Meta-Data: http://msdn.microsoft.com/en-us/library/ms679448(v=VS.85).aspx
9. Repl-UpToDate-Vector: http://msdn.microsoft.com/en-us/library/ms679450(v=VS.85).aspx
10. Sub-Refs: http://msdn.microsoft.com/en-us/library/ms679895(v=VS.85).aspx
11. USN-Changed: http://msdn.microsoft.com/en-us/library/ms680871(v=VS.85).aspx
12. USN-Created: http://msdn.microsoft.com/en-us/library/ms680924(v=VS.85).aspx
13. When-Changed: http://msdn.microsoft.com/en-us/library/ms680921(v=VS.85).aspx
14. Pek-List: http://msdn.microsoft.com/en-us/library/ms679109(v=VS.85).aspx
15. msDS-NcType: http://msdn.microsoft.com/en-us/library/cc220312(v=prot.10).aspx

• Decrypt and Re-encrypt the Password Encryption Key (PEK)
  Secret Data stored within the Active Directory database (NTDS.DIT) such as the password hashes are additionally protected by a Password Encryption Key (PEK) – the PEK are encrypted by the SYSKEY of the DC and are therefore unique to each DC, the sourced NTDS.dit from the IFM contains a PEK encrypted by the SYSKEY from the DC on which the IFM was generated (the computer where the NTDS.dit was backed up). DCPROMO will decrypt the PEK using the SYSKEY (from the DC the IFM was sourced from) from the registry in the restored IFM information as the SYSKEY are stored in the registry (that’s one reason why we need to include parts of the registry in IFM) and then re-encrypt the PEK with the SYSKEY of the computer being promoted to DC.

• Diagnostics and Logging
  IFM promotions can be identified in the Dcpromo.log and Dcpromoui.log files that are located in the %systemroot%debug folder. There are several entries that can be used to verify that the database where sourced from the IFM and that the promotion did use IFM.

  Table 3: DCPROMO.log
  

Table 4: DCPROMOUI.log

Sourcing NDNCs with Windows Server 2003 is only supported by Windows Server 2003 SP1 or later under the following conditions:

• Both the DC your souring the IFM from must be running Windows Server 2003 SP1 or later and as well the machine intending to become a DC using the source IFM.
• The forest functional level (FFL) has to be: Windows Server 2003 (Pre-Windows Server 2003 FFL adding replicas to NCs has to be done on the Domain Naming Master – FSMO)
  Note: The promotion completes with the sourced IFM even if the forest functional level (FFL) is less than Windows Server 2003 but NDNCs aren’t sourced from the IFM and the following will happen:

1. The following will be logged in the Directory Services Log: “The forest functional level is not high enough to complete addition of application directory partitions during installation of the directory. Therefore specified application directory partitions will not be added to this domain controller during installation. If you would like to make this server a replica of an application directory partition, you could re-add these application partition after the installation is complete.”
   
2. The following will be logged in the Directory Services Log, and the DC will begin the process of physically removing the NDCs source from the IFM in the DCs database: “The local domain controller is no longer configured to host the following directory partition. As a result, the objects in this directory partition will be removed from the local Active Directory database.” Note: In this case the DomainDNSZones and ForestDNSZones NDNCs.
   
3. The DomainDNSZones and ForestDNSZones are begin replicated in again over the wire using normal replication, as the promoted DC (Sourced from IFM) hosts the DNS Service: As a result of this the DC (Sourced from IFM) has to obtain a new invocationID once again (It has already done this once for using the sourced IFM database instance):
   Note: This can be confirmed by running the repadmin /showsig command
   

   Default-First-Site-NameNTTEST-SCH-02
   

   Current DC invocationID: 7bbd4543-cf19-44e3-9638-96907ceb8a36 ß Current InvocationID obtained cause of removing/adding NDCs.
   

   28081325-eee8-40b0-9587-9c02867040bc retired on 2011-07-03 07:16:41 at USN 32780 ß New InvocationID representing the sourced IFM restored/promoted as a new instance on the current DC
   

   b7633426-242b-47bf-852c-a07466ef937f retired on 2011-07-03 06:35:39 at USN 16397ß InvocationID representing the instance on the DC where the IFM where sourced
   

• You have to use an unattended answer file specifying the ReplicateFromMedia=Yes parameter as well define the ApplicationPartitionsToReplicate parameter, note this can be used to include specific NDNCs or you can simply include them all by specific a wildcard, Here are some samples:
  • ApplicationPartitionsToReplicate=*
    
  • ApplicationPartitionsToReplicate=”DC=DomainDNSZones,DC=corp,DC=company,DC=com” “DC=ForestDNSZones,DC=corp,DC=company,DC=com”
    

Sourcing SYSVOL with IFM at Windows Server 2003:

The File Replication Service (FRS) can source files and folders from the restored system state backup on the first restart after a DCPROMO IFM promotion if the strict dependencies that the File Replication Service (FRS) requires are fulfilled.

• The system state backup must contain MD5 checksum data that is used by the File Replication Service (FRS) to determine if a restored file or folder is the same as the file versions on existing domain controllers in the domain.
• The File Replication Service (FRS) must have constructed MD5 checksum data for the files in the SYSVOL tree.
  For MD5 checksums to exist, files and folders in the SYSVOL tree must have been replicated at least one time after there were two or more domain controllers in the domain (Note: The SYSVOL can never be efficiently sourced from a IFM media also known as System State Backup in Windows Server 2003 unless there is at least two DCs in the domain already present at the point when the IFM media also known as System State Backup is generated). You can trigger FRS to store the MD5 checksum of all files in the SYSVOL tree my writing a script that modifies the files, for example set/un-set the hidden attribute, that simply will tiger a replication.Furthermore the MD5 checksum data is stored in ntfrs.jdb ESE database that’s by default located in “%SystemRoot%ntfrsjet”. The ntfrs.jdb ese database are using 4k pages and have the following layout.Table 5: NTFRS.JDB Database Layout
  

The MD5 checksum are stored in the IDTable0000X and are stored in the column Spare1Bin.
You can validate the existence of MD5 checksums by using “ntfrsutil idtable > MD5Hash.txt” and search for entries missing hashes.

Table 6: NTFRS.JDB IDTable

• The SYSVOL part of the IFM media also known as the System State Backup must be restored to the same volume that is chosen to host the SYSVOL tree when you run DCPROMO, or it has to be specified to the same value in your unattended answer file.

Seeding the SYSVOL with IFM at Windows Server 2003:

Even if the SYSVOL are sourced with IFM delta changes are about to be replicated in over the network, there are certain requirements to ensure that this process are being efficient and that not the entire SYSVOL tree are replicated over the network again, once of the requirements has already been discussed in this article regarding MD5 checksums: The File Replication Service (FRS) must have constructed MD5 checksum data for the files in the SYSVOL tree.

The Domain Controller (DC) or File Replication Service Replica (FRS Replica) that the initial replication of the SYSVOL tree takes place with must meet the following requirements

• The Domain Controller (DC) or File Replication Service Replica (FRS Replica) that the initial replication takes place with are identified by specifying the “ReplicationSourceDC” parameter in the unattended dcpromo answer file. (Note this can’t be done using the UI)

  How to best select the Domain Controller (DC) or File Replication Service Replica (FRS Replica) to perform initial replication with:
  

  • Locate a domain controller that has a low number of inbound and outbound connections. This domain controller must not be a significant originator or forwarder of change orders to downstream partners in SYSVOL or FRS-replicated DFS replica sets
  • Locate a domain controller that doesn’t act as Bridgehead server (those typically have many replication partners)
• The File Replication Service (FRS) outbound log on the DC/FRS replica that is used to seed the SYSVOL tree with must be cleared so that a full vvjoin is triggered when the initial synchronization of SYSVOL with the IFM promoted DC occur, the reason for this is that if the outbound log contain cached items, an optimized vvjoin is performed and optimized vvjoin’s doesn’t support pre-staging content, this results in a full replication of the entire SYSVOL tree over the network with the IFM promoted DC instead of delta changes and new files.

  How to verify and clear the outlog:

  • At the intended helper DC (DC chosen to perform initial replication with) run:
    ntfrsutil outlog to show current entries in the outlog change cache.
    Note: See Table 5: NTFRS.JDB Database Layout for the outlog table layout earlier in this article
  • If the ntfrsutil outlog show’s entries, the outlog needs to be trimmed/reset or if the period of time specified as the “Outlog Change History in Minutes” (by default 7 days) has passed since the IFM media was generated.
    Changes the Outlog Change History In Minutes value in the following registry subkey:
    HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesNtFrsParameters to 0 (zero)
  • Run ntfrsutl poll /now
  • Restart the FRS Service on the actual DC/FRS replica: net stop ntfrs | net start ntfrs
  • Run the ntfrsutil outlog again, the contents of the current outbound log must contain only files that have been modified after you changed the registry and restarted the FRS.

    Note: Don’t forget to reset the Outlog Change History In Minutes registry type back to the seven-day default while you’re done with the IFM operations (e.g. all DCs intended to be promoted with IFM has been promoted)

• Configure Debug and Analysis logging on the computer that is to be promoted using IFM

  Configure Debug Severity on the computer that is about to be promoted using IFM media:
  

  • To be able to determine whether files in the SYSVOL tree are being moved in from the pre-staged folder on the local computer or are being replicated over the network from an upstream partner, set the registry value for Debug Log Severity to 4 on the computer being promoting using IFM media in the following registry subjey:
    HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesNtFrsParameters to 4
    Note: This has to be configured before you promote the computer to a DC using IFM.
  • Verify whether the files in the SYSVOL tree was seeded from the pre-stage folder (the restored IFM media) or replicated over the network.

To find all the files that were replicated from the initial replication partner over the wire (files that wasn’t seeded from IFM), type:

findstr /I “RcsReceivingStageFile” NtFrs_000X.log where X should be the number of the log, in case if multiple logs e.g. NtFrs_0001.log, NtFrs_0002.log etc run the command against both files.

To find all files that were sourced from the pre-staged system state backup, type:

Findstr /I “(218)” NTFRS_000X.log where X should be the number of the log, in case if multiple logs e.g. NtFrs_0001.log, NtFrs_0002.log