Archive for October, 2009

LTCG issues with the WIN7/amd64 environment of WDK 7600

Now that Windows 7 is out, we all sooner or later have to upgrade to WDK 7600. I am still reluctant to move away from WDK 6000/6001 because of the dropped W2K support, but this is a different issue.

However, as one cfix user who has obviously already adopted WDK 7600 kindly pointed out to me, linking a kernel mode unit test against cfix using WDK 7600 and the WIN7/amd64 environment fails reproducibly with the following error message:

error fatal error C1047: The object or library file ‘…\lib\amd64\cfixkdrv.lib’ was created with an older compiler than other objects; rebuild old objects and libraries

In contrast, building the same driver for WIN7/x86 works fine.

As the documentation for C1047 indicates, this error is usually related to inconsistent usage of Link Time Code Generation (LTCG): As soon as you use LTCG, all objects and libraries must be compiled with /GL — this normally is not a big deal, but as this WDK page rightfully explains, it means that libraries built this way are not suitable for redistribution because of their dependency on a specific compiler/linker version. But of couse, it also means that a library not built using /GL cannot be used easily when you build your program using LTCG.

Before Windows 7, all WDK build environment configurations I am aware of did not use LTCG. Neither did cfix, so everything worked fine even if your compiler/linker versions did not match the ones used for building cfix.

With WDK 7600, this situation changes: While WLH and other downlevel build environments still do not seem to use LTCG, the WIN7 environment, at least for amd64, enables LTCG by default.

What this means is that as soon as you link against a library which is not part of the WDK and therefore likely to be built using a different compiler version, you’ll get C1047. It is thus no surprise that attempting to link against cfixkdrv.lib, which is the library all kernel mode unit tests have to link against and which itself has been built using WDK 6000, leads to the error quoted above.

However, once you have figured this out, the workaround for this issue is trivial: Disable LTCG for your test driver by adding the following line to your SOURCES file:


Link time code generation is a very powerful optimization technique and I encourage everybody to make use of it if possible. However, for the compatibility reasons outlined above, I consider it a rather stupid idea to have the WDK enable LTCG by default. Rather, I had much preferred to see an opt-in switch for LTCG.

Anyway, for the next version, I will consider shipping both, a 7600-compatible LTCG-enabled and a non-LTCG-enabled version of cfixkdrv.lib. Until then, the workaround described above will do the trick.



Having given my presentation on NTrace today at the WCRE in Lille/France, I have also opened to the public. NTrace, in case you have missed my previous posts, is a dynamic function boundary tracing system for Windows/x86 I initially developed as part of my Master’s thesis that is capable of performing DTrace-like tracing of both user and kernel mode components.

On the NTrace page, you will now find the paper itself as being published as part of the WCRE proceedings (mind the copyright notice, please) along with two screencasts: One showing how NTrace can be used to trace kernel mode components such as NTFS, and one demonstrating NTrace for user mode tracing.

If you have questions about NTrace or are interested in more details, please feel free to write me an email — my address is jpassing at acm org.

Mixing 32 and 64-bit components in a single MSI

Definetely one my pet peeves about Windows Installer is how it deals with instruction set architectures (ISAs). Looking at Windows NT history, supported ISAs have come (amd64, IA-64) and gone (Alpha, PowerPC, MIPS) — yet most of the time, there was more than one ISA being officially supported. Having to ship binaries for multiple ISAs therefore always has been on the agenda for many ISVs.

Needless to say, supporting multiple ISAs requires special consideration when developing setup packages and providing separate packages — one for each ISA — has become common practice to approach this. This approach makes perfect sense: Given the incompatibility of most ISAs, nobody needs Alpha binaries on a MIPS system or amd64 binaries on a IA-64 machine, so there seems little reason to mix ISAs within a single package.

Unsurprisingly, Windows Installer, which was created somewhere around 2000, also goes this route and encourages developers to provide separate packages for each ISA.

However, with the advent of amd64/x64/IA-32e/Intel 64/whateveryoucallit, the situation has changed: Because i386 and amd64 are so closely related and compatible, there are now plenty of situations where combining binaries of differing ISAs (i.e. amd64 and i386) in a single installer package makes perfect sense. Examples for this include:

  • A package comprises a shell extension as well as a standalone App. For certain reasons (maybe the use of VB6), there only is a 32 bit version of the App. The shell extension, in contrast, is available for both, i386 and amd64. Whether you put everything into one package or provide separate packages for each ISA, one of them will comprise a mixture of ISAs.
  • SDKs for unmanaged code usually include .lib and .dll files for multiple architectures. Shipping separate packages for i386 and amd64 (containing different binaries but the same headers, docs, etc.) may please the Windows Installer gods, but seems redundant, a waste of disk space, and user-unfriendly.

Thanks to the msidbComponentAttributes64bit flag, mixing architectures in a single MSI package is technically possible: You mark the package as being 32 bit and set said flag for all 64-bit components. Rather than splitting your setup into multiple packages, you can conveniently combine everything into one.

When reading the documentation (and ICE requirements, more on this later) carefully though, it turns out that this is not quite what the Windows Installer team invented this flag for. Anyway, it works fine, problem solved.


If only there was not ICE80.

ICE80, alas, is critical if you intend to conform to the Requirements for the Windows Vista Logo Program for Software:

Applications must use the Windows Installer (MSI) or ClickOnce for installation. Windows Installation packages must not receive any errors from the Internal Consistency Evaluators (ICEs) listed here:

1-24, 27-31, 33-36, 38, 40-57, 59, 61-63, 65, 67-72, 74-84, 86-87, 89-94, 96-99

ICE80 mainly states that (1) you should not install 64 bit components to 32 bit directories (e.g. Program Files vs. Program Files (x86)) and (2) you should not use 64 bit components in a 32 bit package.

(1) is fair enough, although it raises the question where you should install your software to without splitting it in two or violating other ICE rules. Worse yet, (2) effectively means that said way to create multi-ISA packages, creating 32 bit packages with some components marked with msidbComponentAttributes64bit, is illegal alltogether.

So to be logo’ed, there seems to be no other way than providing separate packages, maybe along with (urgh!) a meta-package that installs the other two.

If there are more important things on your schedule than getting a Vista logo, ICE80 seems like something that can safely be ignored. Indeed, this is what I have done several times, including in case of the cfix installer.

Anyway, let’s ignore ICE80 once more and hold on to the plan of building a 32-bit package containing both, 32-bit and 64-bit components.


For an SDK that is installed on 64-bit Windows, it will usually make sense to install both, 32 and 64 bit .lib and .dll files etc. On 32-bit Windows, installing 64-bit components may seem odd, but due to the existence of amd64 compilers for i386, it still makes sense to install them or at least offer them as optional feature.

So far, so good. Things get interesting, though, when COM registration comes into play. Naturally, a 32 bit installer package sees the system like any other 32 bit application does. Most importantly, this means that Registry Reflection and File System Redirection applies.

Now consider a package that contains both a 32-bit and a 64-bit version of some COM server, each installed to a separate directory. COM Registration either be performed through the Class or the Registry table. Provided that the msidbComponentAttributes64bit flag has been used properly, such a package will work great on 64 bit systems thanks to Registry Reflection: The regsitry entries will be written to the proper (reflected) locations and both COM servers will work properly.

Now think what happens on 32-bit Windows: (1) There is no Registry Reflection and (2) Windows Installer silently ignores msidbComponentAttributes64bit flags. Result: The installation will run just as smooth as on the 64-bit system. However, while installing the files continues to works flawlessly, the registry will be left in a less-than-optimal state: Due to the nonexistence of Registry Reflection, the registration entries of both COM servers will have been written to the same location!

Needless to say, the server whose registration entries were written first will now be unusable.

In a way, Windows Installer has taken its revenge for breaking the rules.

Bottom line: Mixing 32 and 64-bit components in a single MSI works fine in many cases, but is against the MSI rules and can lead to further problems. And while I am still convinced that providing separate, ISA-specific packages is wrong or at least inconvenient in certain situations, it is definitely the safer and “right” way to go.

(Note: Windows Installer 4.5 introduced multi-package transactions, which allow reliable and transactional multi-package setups to be built so that splitting a setup into multiple packages can be implemented without much pain. However, very few users already have Windows Installer 4.5 installed and Windows 2000 is not even supported by this release. For many of us, relying on this feature therefore is not really an option.)

I’ll be at WCRE 2009 presenting NTrace

Next week, the 16th Working Conference on Reverse Engineering (WCRE) will be held in Lille, France. I will be there presenting NTrace: Function Boundary Tracing for Windows on IA-32.

NTrace is a dynamic function boundary tracing toolkit for IA-32/x86 that can be used to trace both kernel and user mode Windows components — examples for components that can be traced include the kernel itself (ntoskrnl), drivers like NTFS as well as user mode components such as kernel32, shell32 or even explorer.exe.

NTrace implements a novel approach to instrumenting IA-32 machine code and integrating with the Structured Exception Handling facility of Windows. Using this approach, NTrace is not only capable of tracing nearly the entire Windows kernel and system libraries, it is also faster than Solaris DTrace FBT on IA-32!

Details on how exactly NTrace works will be publiched in the paper, which will be made available soon. I will also publish more details on NTrace both here and on a dedicated NTrace website.

The work, by the way, is basically the result of my Master’s thesis I wrote back in 2008.


About me

Johannes Passing lives in Berlin, Germany and works as a Solutions Architect at Google Cloud.

While mostly focusing on Cloud-related stuff these days, Johannes still enjoys the occasional dose of Win32, COM, and NT kernel mode development.

He also is the author of cfix, a C/C++ unit testing framework for Win32 and NT kernel mode, Visual Assert, a Visual Studio Unit Testing-AddIn, and NTrace, a dynamic function boundary tracing toolkit for Windows NT/x86 kernel/user mode code.

Contact Johannes: jpassing (at) hotmail com

LinkedIn Profile
Xing Profile
Github Profile