WDK « Johannes Passing's Blog

Archive for the 'WDK' Category

What a weirdo: How the /analyze switch changes its behavior depending on its environment

In Visual Studio 2005 Team System (VSTS), the “ultimate” SKU of Visual Studio 2005, Microsoft introduced the /analyze compiler switch. When the /analyze switch is used, the cl compiler not only does its regular checks, but performs a much more thorough static code analysis.

While /analyze is very useful indeed, it was only available in the top SKU — the Standard and Professional versions of Visual Studio lacked support for this compiler switch (this has changed by now, Professional now also supports this feature). As some smart people quickly figured out though, the compilers shipped as part of the Windows SDK did support /analyze, too.

So given that some compilers do support /analyze while other do not, you may well expect that there are two slightly different types of binaries, one that the SDK and VSTS uses, and one that is shipped with other Visual Studio SKUs.

At least this was what I expected. As it turns out though, this is not quite the case.

Where’s /analyze?

For the past two years, I have been developing using Visual Studio 2005 Team System along with Windows SDK 6.0 and WDK 6000 on a Vista x64 machine. Using this setup, I was able to use the /analyze switch in both, “regular” Visual Studio projects and WDK (build.exe-driven) projects. That led me to the conclusion that the WDK 6000 compilers, like the SDK compilers were in fact /analyze-enabled binaries as well.

Switching to a Windows 7 machine with VSTS 2005 and 2008, SDK 7.0, and WDK 6000 did not change this — /analyze kept working fine in all environments.

Then I set up a build server, installed WDK 6000 and Windows SDK 7.0 and attempted to perform a build — to my surprise, though, I got plenty of complaints about the /anayze switch not being supported.

I verified that the right compilers (WDK 6000) were used and compared cl versions between the build machine and my development machine — both were 14.00.50727.220, so everything seemed right. Running cl.exe /? on both machines, however, I noticed that despite versions being the same, this Code Analsis section was missing in the output on the build machine:

                         -CODE ANALYSIS-

/analyze[:WX-] enable code analysis
    WX- - code analysis warnings should not be treated as errors even if /WX is invoked

So obviously, Code Analysis support is enabled or disabled depending on external factors — not the binary itself, but the environment somehow determines whether the /analyze switch is supported or not.

Observing cl.exe /? with Process Monitor on my development machine resulted in the following output:

This trace leaves little room for interpretation: The code analysis features must (mainly) be implemented in c1xxast.dll. c1xxast.dll, however, is not shipped with the WDK itself, nor is it shipped with the non-VSTS SKUs of Visual Studio. So by default, the WDK’s cl will fail to locate the DLL and will revert to “/analyze-disabled mode”.

If, however, you have VSTS or the Windows SDK installed on your machine and your %PATH% happens to include the right directories, cl’s search for c1xxast.dll will succeeded and — tada — /analyze suddenly works. On my development machine, this obviously was the case, whilst on the build machine, it was not.

Compiler version mish-mash

I added the Windows SDK’s bin directory to the build machine’s %PATH% and rerun the build. As I expected, /analyze now worked fine — what I did not quite expect though was that I was now getting dozens of compilation warnings like:

warning C6309: Argument '1' is null: this does not adhere to function
specification of 'CfixCreateThread'

The reason for this was simple: The WDK cl.exe (remember, version 14.00.50727.220), thanks to a proper %PATH%, now used c1xxast.dll from SDK 7 to perform code analysis — despite the fact that c1xxast.dll actually “belonged” to cl version 15.00.30729.01. So the c1xxast.dll was one generation ahead of the WDK I was using.

The really, really cool thing about cl being able to work with a newer c1xxast.dll is that you can continue using WDK 6000 or 6001 (with W2K support!) and still benefit from the latest-and-greatest static code analysis features.

The reason for getting several warnings on the build machine while not getting similar warnings on my development machine was simply that on my development machine, the VS 2005 directory preceded the SDK directory in my %PATH%. Once I switched the order, I got the same wanings on both machines. This leads me to:

The ugly thing about this, however, is that a tiny change in the order of directories in %PATH% can suddenly make a huge difference w.r.t. code analysis. This is not quite what you’d normally expect.

(The additional compiler warnings, by the way, were a result of the improved analysis checks in cl 15: cl 14 routinely failed to verify the usage of __in vs. __in_opt parameters; cl 15 has become much more precise here and found several mis-attributed function signatures.)

LTCG issues with the WIN7/amd64 environment of WDK 7600

Published October 21, 2009 WDK , cfix Leave a Comment
Tags: ddk, library, ltcg, WDK, win7

Now that Windows 7 is out, we all sooner or later have to upgrade to WDK 7600. I am still reluctant to move away from WDK 6000/6001 because of the dropped W2K support, but this is a different issue.

However, as one cfix user who has obviously already adopted WDK 7600 kindly pointed out to me, linking a kernel mode unit test against cfix using WDK 7600 and the WIN7/amd64 environment fails reproducibly with the following error message:

error fatal error C1047: The object or library file ‘…\lib\amd64\cfixkdrv.lib’ was created with an older compiler than other objects; rebuild old objects and libraries

In contrast, building the same driver for WIN7/x86 works fine.

As the documentation for C1047 indicates, this error is usually related to inconsistent usage of Link Time Code Generation (LTCG): As soon as you use LTCG, all objects and libraries must be compiled with /GL — this normally is not a big deal, but as this WDK page rightfully explains, it means that libraries built this way are not suitable for redistribution because of their dependency on a specific compiler/linker version. But of couse, it also means that a library not built using /GL cannot be used easily when you build your program using LTCG.

Before Windows 7, all WDK build environment configurations I am aware of did not use LTCG. Neither did cfix, so everything worked fine even if your compiler/linker versions did not match the ones used for building cfix.

With WDK 7600, this situation changes: While WLH and other downlevel build environments still do not seem to use LTCG, the WIN7 environment, at least for amd64, enables LTCG by default.

What this means is that as soon as you link against a library which is not part of the WDK and therefore likely to be built using a different compiler version, you’ll get C1047. It is thus no surprise that attempting to link against cfixkdrv.lib, which is the library all kernel mode unit tests have to link against and which itself has been built using WDK 6000, leads to the error quoted above.

However, once you have figured this out, the workaround for this issue is trivial: Disable LTCG for your test driver by adding the following line to your SOURCES file:

USER_C_FLAGS=/GL-

Link time code generation is a very powerful optimization technique and I encourage everybody to make use of it if possible. However, for the compatibility reasons outlined above, I consider it a rather stupid idea to have the WDK enable LTCG by default. Rather, I had much preferred to see an opt-in switch for LTCG.

Anyway, for the next version, I will consider shipping both, a 7600-compatible LTCG-enabled and a non-LTCG-enabled version of cfixkdrv.lib. Until then, the workaround described above will do the trick.

AuxKlibGetImageExportDirectory and forwarders

Published April 15, 2008 Kernel , WDK Leave a Comment
Tags: Kernel, pe

One of the newer additions to the DDK is the aux_klib library, which, among others, offers the routine AuxKlibGetImageExportDirectory. As its name suggests, AuxKlibGetImageExportDirectory offers a handy way to obtain a pointer to the export directory of a kernel module.

There is, however, one issue that — at least in my opinion — renders AuxKlibGetImageExportDirectory pretty much useless in most scenarios: Dealing with forwaders.

The primary motivation to call AuxKlibGetImageExportDirectory is to either enumerate the exports of a module or to find a specific export. In both cases, the code is likely to call at least one of the exported routines. To maintain binary compatibility, it would be risky for such code to rely on the fact that all exports that it aims to call are in fact ‘real’ exports and not forwarders. Rather, it is crucial to be prepared to find both types — exports and forwarders — in the export directory and handle each of them appropropriately.

So we need to tell an export from a forwarder. As it turns out, this is not quite as easy as checking some flag. Quoting the Microsoft Portable Executable and Common Object File Format Specification on the content of the export address table:

Each entry in the export address table is a field that uses one of two formats in the following table. If the address specified is not within the export section (as defined by the address and length that are indicated in the optional header), the field is an export RVA, which is an actual address in code or data. Otherwise, the field is a forwarder RVA, which names a symbol in another DLL.

And this exactly is the problem — only being provided the PIMAGE_EXPORT_DIRECTORY pointer, we do not know the start and end RVA of the export section. As a consequence, identifying forwarders is infeasible when using AuxKlibGetImageExportDirectory — which in turn makes it a pretty much useless function.

Workaround

Although AuxKlibGetImageExportDirectory is handy, the work it performs is rather trivial. Therefore, it is not hard to come up with code that, given the Load Address of a module, finds the export directory and properly checks for the existance of forwarders. The following code shows how:

PIMAGE_DATA_DIRECTORY ExportDataDir;
PIMAGE_EXPORT_DIRECTORY ExportDirectory;
PIMAGE_DOS_HEADER DosHeader = ( PIMAGE_DOS_HEADER ) LoadAddress;
PIMAGE_NT_HEADERS NtHeader; 

PULONG FunctionRvaArray;
PUSHORT OrdinalsArray;

ULONG Index;

//
// Peek into PE image to obtain exports.
//
NtHeader = ( PIMAGE_NT_HEADERS )
  PtrFromRva( DosHeader, DosHeader->e_lfanew );
if( IMAGE_NT_SIGNATURE != NtHeader->Signature )
{
  //
  // Unrecognized image format.
  //
  return ...;
}

ExportDataDir = &NtHeader->OptionalHeader.DataDirectory
    [ IMAGE_DIRECTORY_ENTRY_EXPORT ];

ExportDirectory = ( PIMAGE_EXPORT_DIRECTORY ) PtrFromRva(
  LoadAddress,
  ExportDataDir->VirtualAddress );

if ( ExportDirectory->AddressOfNames == 0 ||
   ExportDirectory->AddressOfFunctions == 0 ||
   ExportDirectory->AddressOfNameOrdinals == 0 )
{
  //
  // This module does not have any exports.
  //
  return ...;
}

FunctionRvaArray = ( PULONG ) PtrFromRva(
  LoadAddress,
  ExportDirectory->AddressOfFunctions );

OrdinalsArray = ( PUSHORT ) PtrFromRva(
  LoadAddress,
  ExportDirectory->AddressOfNameOrdinals );

for ( Index = 0; Index <
      ExportDirectory->NumberOfNames; Index++ )
{
  //
  // Get corresponding export ordinal.
  //
  USHORT Ordinal = ( USHORT ) OrdinalsArray[ Index ]
    + ( USHORT ) ExportDirectory->Base;

  //
  // Get corresponding function RVA.
  //
  ULONG FuncRva =
    FunctionRvaArray[ Ordinal - ExportDirectory->Base ];

  if ( FuncRva >= ExportDataDir->VirtualAddress &&
     FuncRva < ExportDataDir->VirtualAddress
       + ExportDataDir->Size )
  {
    //
    // It is a forwarder.
    //
  }
  else
  {
    //
    // It is an export.
    //
  }
}

Creating and embedding message tables with the WDK/build.exe

Published February 25, 2008 WDK , Win32 Leave a Comment
Tags: build mc

Although message tables play an important role in Windows, their tool support has always be somewhat limited — at least compared to string tables, for which Visual Studio even provides a graphical editor.

When in comes to creating and embedding message tables into a binary built with the WDK, documentation is light. However, the WDK tool chain provides support for mc files and using it requires only a few steps.

1. Create a message file

Unsurprisingly, the first step is to write a message file — I will name it foobarmsg.mc. Here is an example file:

;
; The default is NTSTATUS -- but HRESULT works just as well.
;
MessageIdTypedef=HRESULT

SeverityNames=(
  Success=0x0
  Informational=0x1
  Warning=0x2
  Error=0x3
)

FacilityNames=(
  Interface=4
)

LanguageNames=(English=0x409:MSG00409)

;//--------------------------------------------------------------------
MessageId		= 0x9000
Severity		= Warning
Facility		= Interface
SymbolicName	= FOOBAR_E_WEIRDFAILURE
Language		= English
Some weird failure has occured.
.

Updating the SOURCES file

The message file must be compiled (done by mc.exe). If we include the mc file in the SOURCES macro, build.exe will arange this for us:

SOURCES=
	foobar.c
	foobarmsg.mc

To tell mc where to place the result files (i.e. the header and the resources), the following two macros can be used in the SOURCES file:

PASS0_HEADERDIR=....include
PASS0_SOURCEDIR=obj$(BUILD_ALT_DIR)$(TARGET_DIRECTORY)

As the names of the macros suggest, mc.exe is run during pass 0 (i.e. before any sources are compiled) — therefore, it is no problem to include the generated header file (foobar.h) in the source files.

Updating the rc file

Assuming the project already includes a .rc file for versioning information, we can use this file and refer to the generated message table resources. At the end of your project’s rc file, include the following line:

#include "foobarmsg.rc"

That’s it. The resulting binary will contain a proper message table.

How to use manifests with build.exe

Published February 1, 2008 WDK , Win32 1 Comment
Tags: SXS, UAC, WDK

As of Windows Vista, basically all applications require a manifest in order to at least declare UAC compliance. Visual Studio has builtin support for creating and embedding manifests, so when using VS to build applications, using manifests is straightforward. However, when building a user mode application with the WDK and build.exe, things are a little different. Looking at the WDK documentation, manifests remain unmentioned — both in the context of UAC and SXS. Judging from documentation, it seems that the WDK does not provide any support for embedding manifests — which would mean that you are left with having to poke with the makefiles in order to invoke mt somewhere.

Looking at makefile.new, however, reveals that there are plenty of manifest-related rules and as it turns out, there indeed is support for manifests. On lines 1866 to 1925 (WDK 6000), makefile.new even contains a short documentation about the usage of manifests — so whether manifest support being unmentioned in the official docs is intentional or not is not quite clear. However, using the information in makefile.new, it is straightforward to get build.exe to embed manifests in a binary.

To embed a manifest, first create the manifest file and name it myapp.manifest. In the SOURCES file, include these lines:

SXS_APPLICATION_MANIFEST=myapp.manifest
SXS_ASSEMBLY_VERSION=1.0
SXS_ASSEMBLY_NAME=MyApp
SXS_ASSEMBLY_LANGUAGE=0000

That’s it.

In order not to have to provide a separate manifest file for different processor builds, there is some preprocessing taking place before the manifest is embedded. The following macros are available for use in the manifest:

SXS_ASSEMBLY_NAME (set in SOURCES)
SXS_ASSEMBLY_VERSION (set in SOURCES, defaults to 5.1.0.0)
SXS_ASSEMBLY_LANGUAGE (set in SOURCES, LCID or 0000 for neutral)
SXS_PROCESSOR_ARCHITECTURE (set automatically)

When using these macros, note that they will be replaced by quoted text — as a consequence, you have to use them — a bit unintuitively — as follows (Note the missing quotes!):

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly
     xmlns="urn:schemas-microsoft-com:asm.v1"
     manifestVersion="1.0">
  <assemblyIdentity version=SXS_ASSEMBLY_VERSION
     processorArchitecture=SXS_PROCESSOR_ARCHITECTURE
     name=SXS_ASSEMBLY_NAME/>
  <description>Example</description>
  <trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
    <security>
      <requestedPrivileges>
        <requestedExecutionLevel level="asInvoker"
          uiAccess="false"/>
      </requestedPrivileges>
    </security>
  </trustInfo>
</assembly>

After preprocessing, the file is embedded into the binary as follows:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!--  Copyright (c) Microsoft Corporation -->
<assembly
     xmlns="urn:schemas-microsoft-com:asm.v1"
     manifestVersion="1.0">
  <assemblyIdentity version="1.0.0.0"
     processorArchitecture="AMD64"
     name="MyApp" />
  <description>Example</description>
  <trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
  <security>
    <requestedPrivileges>
      <requestedExecutionLevel level="asInvoker"
        uiAccess="false" />
    </requestedPrivileges>
  </security>
  </trustInfo>
</assembly>

About me

Johannes Passing, M.Sc., living in Berlin, Germany.

Johannes mainly focusses on Win32, COM, and NT kernel mode development, along with Java and .Net. He also is the author of cfix, a C/C++ unit testing framework for Win32 and NT kernel mode, Visual Assert, a Visual Studio Unit Testing-AddIn, and NTrace, a dynamic function boundary tracing toolkit for Windows NT/x86 kernel/user mode code.

Contact Johannes: jpassing (at) acm org

Johannes' GPG fingerprint is BBB1 1769 B82D CD07 D90A 57E8 9FE1 D441 F7A0 1BB1.

LinkedIn Profile

Xing Profile

Follow me on Twitter (new)

Johannes Passing's Blog