Understanding Windows VM initialization on Google Cloud

On Google Cloud, a Linux instance boots in about 30 seconds while a Windows instance takes a full 2 minutes to turn up – why is Windows so much slower? Pundits might be quick to point to Windows being more bloated and slower as the reason, but the truth is that there is simply a lot more going on during VM initialization – and as it turns out, and a lot of it has to do with the trouble of cloning Windows installations.

The trouble with cloning Windows installations

When you install Windows, Windows Setup generates and assigns a unique SID to the machine. Naturally, if you later clone the Windows installation, each clone ends up with the same SID.

Conventional wisdom tells that duplicate SIDs are a problem – and that having multiple machines with the same machine SID in a network poses a security risk. Luckily, this myth has long been debunked by Mark Russinovich:

So is having multiple computers with the same machine SID a problem? The only way it would be is if Windows ever references the machine SIDs of other computers. [...] However as we reviewed, Windows doesn’t allow you to authenticate to another computer using an account known only to the local computer. Instead, you have to specify credentials for either an account local to the remote system or to a Domain account for a Domain the remote computer trusts. The remote computer retrieves the SIDs for a local account from its own Security Accounts Database (SAM) and for a Domain account from the Active Directory database on a Domain Controller (DC). The remote computer never references the machine SID of the connecting computer.

In other words, it’s not the SID that ultimately gates access to a computer, but an account’s user name and password: simply knowing the SID of an account on a remote system doesn’t allow you access to the computer or any resources on it. As further evidence that a SID isn’t sufficient, remember that built-in accounts like the Local System account have the same SID on every computer, something that would be a major security hole if it was.

So at least from a security perspective, cloning a machine seems harmless – unless one of the clones is a domain controller:

Every Domain has a unique Domain SID that’s the machine SID of the system that became the Domain’s first DC, and all machine SIDs for the Domain’s DCs match the Domain SID. So in some sense, that’s a case where machine SIDs do get referenced by other computers. That means that Domain member computers cannot have the same machine SID as that of the DCs and therefore Domain.

If you know exactly what the clones are going to be used for, odds are that duplicate SIDs will not cause any issues. But if you want to play it safe, it is best to follow the Microsoft policy for disk duplication of Windows installations which demands:

When you deploy a duplicated or imaged Windows installation, it is required that the System Preparation (Sysprep) tool is used before the capture of the image. Sysprep prepares an installation of Microsoft Windows for duplication, auditing, and customer delivery.

A positive side-effect of using sysprep is that it also cleans up additional system state like settings and the event log.

Base images on Google Cloud

Google provides public images for all recent versions of Windows Server. Given that Google Cloud has no idea what a customer might be using these images for, allowing machine SID duplications to happen would be risky. So Google Cloud plays it safe by retaining the public images in a generalized state. That means that the final stages of Windows Setup – including the one where the SID is generated and assigned – have not been run yet, and are only run when you first start up a new VM instance.

Before digging deeper into what exactly happens when you first start a VM instance, it is worth reviewing the role of Windows Setup configuration passes.

Windows Setup configuration passes

When you run Windows Setup, then one of the first (and most important) things that happen is that Windows Setup initializes the installation disk and applies the installation image (install.wim). Once that step has completed, the installation is almost done – it is not quite done yet because the installation is still very generic and no machine-specific configuration has been applied yet. Windows calls this state generalized state.

The next stage in the Windows Setup is the specialize configuration pass. In this pass, machine-specific configuration is applied. After the specialize pass, you can optionally boot into audit mode. During the auditSystem and auditUser configuration passes that follow, you can manually apply additional customizations.

Finally, the system enters the oobeSystem configuration pass, where the user is prompted for all sorts of preferences that are then applied to the system. Once this pass has completed, the installation has been fully customized to the machine – the installation is now said to be in specialized state.

Automation Hooks

By default, the Windows Setup is GUI-driven and especially during the oobeSystem pass, Setup prompts for a lot of user input. But Windows Setup also provides some hooks that allow you to fully automate the installation process:

• Answer files: An answer file, usually named unattend.xml, is an XML file that essentially contains all “answers” that Setup would usually prompt for. Answer files primarily come into play during the specialize and oobeSystem phase.
• Hijacking CmdLine in HKLM:\SYSTEM\Setup: Windows Setup uses the CmdLinevalue in HKLM:\SYSTEM\Setup to define what command should be run after the next boot. By replacing the value with a custom script (which in turn invokes the original command), you can inject custom logic into the setup process.
• SetupComplete.cmd/ErrorHandler.cmd: These scripts are run after the Windows Setup has completed or failed. They can be used to automate any post-install tasks such as installing custom applications.

Sysprep and GCESysprep

Once an installation is in a specialized state, you can turn it back to a generalized state by running sysprep /generalize /oobe. Although that does not mean that everything that was done previously during specialize or oobeSystem is being undone, enough of the state is being cleared that the installation can be legitimately considered to be in generalized state again.

Initially, I said that the Google Cloud public images are in a generalized state – so does that simply mean that Google ran sysprep /generalize /oobe before publishing the images? As it turns out, that is not the case – instead, Google runs GCESysprep before publishing an image.

As the documentation explains, GCESysprep is a wrapper around sysprep that ensures that a few extra things happen on first boot, including:

• Set the hostname to match the VM instance name.
• Run user-provided startup and specialize scripts.
• Activate Windows.
• Configure RDP and WinRM.

Let’s see how that actually works.

Conveniently, GCESysprep is open source and its sources are on GitHub. Here’s the gist of what the script does:

• Clean up logs, temp files, certificates, and some scheduled tasks.
• Run sysprep /generalize /oobe and instruct it to use a (rather uninteresting) default unattend.xml for the specialize and oobeSystem configuration passes that will follow the next time the system is booted. You can also use a custom unattend.xml if you like.
• Hijack CmdLine in HKLM:\SYSTEM\Setup so that windeploy.cmd is run on first boot.
• Reboot (unless run with -NoShutdown).

As you see, GCESysprep uses two of the three automation hooks mentioned before. While the custom unattend.xml is not too interesting, the windeploy.cmd hijack will play a crucial role during VM initialization.

First boot using a GCESysprep’ed image

When you first boot an image that has been prepared by using GCESysprep, the HKLM:\SYSTEM\Setup hijack kicks in and causes Windeploy.cmd to be run.

Windeploy.cmd first invokes windeploy.exe (which is the command that would usually be run if the hijack were not in place). Windeploy resumes Setup and runs through the specialize and oobeSystem configuration passes. Because an unattend.xml file is in place, these configuration passes complete silently.

Once windeploy.exe completes, control is returned to the Windeploy.cmd script, which then runs instance_setup.ps1 -specialize. This script performs a number of GCE-specific tasks, including:

• Configure networking settings.
• Set the hostname to match the VM instance name. Renaming a computer requires a reboot, but conveniently, a reboot is pending anyway, so this is a great time to perform the rename.
• Generate a custom SetupComplete.cmd file (taking advantage of the third kind of automation hook). This script causes instance_setup.ps1 to be run again after the next reboot, this time without the -specialize flag.
• Run GCEMetadataScripts specialize, which in turn causes any user-provided specialize script to be run.

What is worth noticing is that any user-provided specialize scripts run after the Windows Setup has completed the specialize and oobeSystem configuration passes. Any attempts to alter the unattend.xml from within a user-provided specialize script is therefore futile.

Second boot

After the second boot, Windows Setup is complete and will therefore run the custom SetupComplete.cmd that has been generated previously. This script invokes instance_setup.ps1, which applies some final configuration before declaring the VM instance ready:

• Activate Windows. This is done using the activate_instance.ps1 helper script.
• Enable RDP and WinRM.
• Trigger a scheduled task to run any user-provided startup scripts.

At this point, the VM instance is ready and you can connect to it by using RDP or WinRM.

The fact that a VM has to boot twice during initialization makes it significantly more time-consuming to spin up a Windows VM than a Linux VM – but as we have seen, the two boots happen for a good reason.

• Share this article: