Archive for February, 2008

When HEAP_NO_SERIALIZE heap operations are not quite ‘not serialized’

While the RTL Heap performs all heap operation in an interlocked manner by default, it can be requested not to serialize operations by passing the HEAP_NO_SERIALIZE flag to HeapCreate or HeapAlloc. In this case, the caller is in charge to provide proper synchronization.

As it turns out, things change when Application Verifier heap checks are enabled for this process. To check heap operations, Application Verifier intercepts certain operations. Presumably to protect its own internal data structures, at some points, Avrf enters a critical section, as shown in this stack trace:

0:009> k
ChildEBP RetAddr  
0840f540 77d9cfad ntdll!ZwWaitForSingleObject+0x15
0840f5a4 77d9cf78 ntdll!RtlpWaitOnCriticalSection+0x154
0840f5cc 6db25f92 ntdll!RtlEnterCriticalSection+0x152
0840f5f0 6db29d99 vfbasics!AVrfpFreeMemLockChecks+0x42 
0840f60c 6db2bae8 vfbasics!AVrfpFreeMemNotify+0x39 
0840f668 77731d27 vfbasics!AVrfpRtlFreeHeap+0x148 
0840f67c 6d242757 kernel32!HeapFree+0x14
...

Entering a critical section is no big thing — nontheless, it effectvely means that heap operations, whether they are conducted with HEAP_NO_SERIALIZE flag set or not, must be considered to be at least partially serialized by Application Verifier.

Although this somewhat contradicts the idea of the HEAP_NO_SERIALIZE flag, this should not be a problem in most cases — especially when HEAP_NO_SERIALIZE has been passed for performance reasons only. However, if HEAP_NO_SERIALIZE has been used for correctness reasons, this might indeed bite you. In my specific case, I used HEAP_NO_SERIALIZE and did the synchronization myself in order to break a deadlock situation. And while the application ran fine (and deadlock-free) without Avrf enabled, it began deadlocking randomly as soon as I enabled Avrf heap checks — for the very reason just described. This is especially annoying as I am now left with two less-than-optimal choices: Leave Avrf enabled for my test suite and experience random deadlocks or forgo Avrf’s helpful heap checks in order not to deadlock. Grrr, I need more choices!

Advertisements

Introducing cfix, a unit testing framework for C/C++ on Win32

I am happy to announce that the unit testing framework cfix I have developed over the past weeks and months is now available on Sourceforge as a first release candidate. It is licensed under the GPL, both binaries and source code are available.

Background

cfix is a framework for authoring and running xUnit-like testcases written in C or C++. The aim of the tool is to provide a development experience very similar to frameworks such as JUnit or NUnit. Due to the nature of C and C++, current unit testing frameworks for C and C++ hardly reach the ease of use of JUnit or NUnit. In particular, it is noticable that significantly more code has to be written to implement a test suite.

Languages like Java and the various .Net languages, as well as scripting languages, all provide reflection facilities. Unit testing frameworks for these languages can therefore rely on reflective features in order to minimize the amount of code required to define a test suite. Provided a library, the framework can find and identify test cases and is able to selectively run them.

Lacking similar reflective facilities, the route most unit testing frameworks for C and C++ have chosen is to oblige the developer to explicitly define test cases and fixtures. Taking CUnit as an example, the developer has to make explictit function calls to define a test suite, add test cases to the suite and to finally run this suite. CppUnit simplifies this a bit, but still requires the developer to implement quite some amount of initialization code. Another important drawback of this approach is the fact that no real separation between test code and test runner is done. Often, even the choice whether to use a graphical or a console frontend for running test is tied to this initialization code.

The aim of cfix is to overcome these limitations and to provide an easy to use, yet powerful unit testing framework for C and C++. Rather than having to write tedious initialization code, the developer merely has to define a fixture using the macros shown in the example below.

Like JUnit and NUnit (but unlike all frameworks for C/C++, with WinUnit being the notable exception), cfix implements a separation in between test code and the test runner. Unit tests are compiled into a DLL rather than an EXE file. This DLL thus only contains the code to be tested. The entire logic and user interface required to actually run unit tests is contained in the cfix-provided testrunner, cfix32.exe/cfix64.exe.

Moreover, cfix has been designed to make debugging of unit tests as easy as possible — the frameworks notices when a debugger is present and allows you to break in as soon as some assertion fails.

Example

So let’s have a look at a minimalistic cfix unit test:

#include 

void Test1()
{
  int a = 1;
  int b = 1;
  CFIX_ASSERT( a + b == 2 );
}

CFIX_BEGIN_FIXTURE( MyMinimalisticFixture )
  CFIX_FIXTURE_ENTRY( Test1 )
CFIX_END_FIXTURE()

After compiling and linking the code into a DLL using

cl /Zi /LD cfix-sample.c

the unit test can be run with the testrunner cfix32 (or cfix64):

[Success]      cfix-sample.dll.MyMinimalisticFixture.Test1

Needless to say, cfix is not limited to such simple tests — have a look at the Tutorial to learn more about the features, installation and usage of cfix.

Enough background information — go ahead and…

Of course, I’d love to read your feedback — you can reach me via passing at users.sourceforge.net!

Creating and embedding message tables with the WDK/build.exe

Although message tables play an important role in Windows, their tool support has always be somewhat limited — at least compared to string tables, for which Visual Studio even provides a graphical editor.

When in comes to creating and embedding message tables into a binary built with the WDK, documentation is light. However, the WDK tool chain provides support for mc files and using it requires only a few steps.

1. Create a message file

Unsurprisingly, the first step is to write a message file — I will name it foobarmsg.mc. Here is an example file:

;
; The default is NTSTATUS -- but HRESULT works just as well.
;
MessageIdTypedef=HRESULT

SeverityNames=(
  Success=0x0
  Informational=0x1
  Warning=0x2
  Error=0x3
)

FacilityNames=(
  Interface=4
)

LanguageNames=(English=0x409:MSG00409)

;//--------------------------------------------------------------------
MessageId		= 0x9000
Severity		= Warning
Facility		= Interface
SymbolicName	= FOOBAR_E_WEIRDFAILURE
Language		= English
Some weird failure has occured.
.
Updating the SOURCES file

The message file must be compiled (done by mc.exe). If we include the mc file in the SOURCES macro, build.exe will arange this for us:

SOURCES=
	foobar.c 
	foobarmsg.mc

To tell mc where to place the result files (i.e. the header and the resources), the following two macros can be used in the SOURCES file:

PASS0_HEADERDIR=....include
PASS0_SOURCEDIR=obj$(BUILD_ALT_DIR)$(TARGET_DIRECTORY)

As the names of the macros suggest, mc.exe is run during pass 0 (i.e. before any sources are compiled) — therefore, it is no problem to include the generated header file (foobar.h) in the source files.

Updating the rc file

Assuming the project already includes a .rc file for versioning information, we can use this file and refer to the generated message table resources. At the end of your project’s rc file, include the following line:

#include "foobarmsg.rc"

That’s it. The resulting binary will contain a proper message table.

Glitch in documentation of SYMBOL_INFO

The documentation of the SYMBOL_INFO structure used by dbghelp contains a glitch. As the structure is of variable length, the documentation for SizeOfStruct states:

Size of the structure, in bytes. This member must be set to sizeof(SYMBOL_INFO). Note that the total size of the data is the SizeOfStruct + MaxNameLen – 1. The reason to subtract one is that the first character in the name is accounted for in the size of the structure.

While this is true for SYMBOL_INFO (the ANSI version), it is wrong for SYMBOL_INFOW (the Unicode version). As MaxNameLen contains the character count rather than the byte count of the Name field, it should read:

Size of the structure, in bytes. This member must be set to sizeof(SYMBOL_INFO). Note that the total size of the data is SizeOfStruct + ( MaxNameLen – 1 ) * sizeof( TCHAR ). The reason to subtract one is that the first character in the name is accounted for in the size of the structure.

I have reported this bug, let’s see how soon it gets fixed…

Update: The documentation has meanwhile been fixed.

Application Verifier: Thread cannot own a critical section

Whereas most Verifier error messages are rather descriptive, this one is not really:

VERIFIER STOP 00000200 : pid 0x2B4: 
Thread cannot own a critical section. 

0000104C : Thread ID.
05274FDC : Critical section address.
09D88FE0 : Critical section debug information address.
010135D4 : Critical section initialization stack trace.

Why should not this thread be allowed to own a critical section? What is Application Verifier trying to tell us? Luckily, the stack trace gives a hint:

ntdll.dll!_DbgBreakPoint@0() 	
vrfcore.dll!VerifierStopMessageEx(...)
vfbasics.dll!VfBasicsStopMessage()
vfbasics.dll!AVrfpCheckCriticalSection()
vfbasics.dll!AVrfpCheckCriticalSectionSplayNode()
vfbasics.dll!AVrfpCheckCriticalSectionTree()
vfbasics.dll!AVrfCheckForOrphanedCriticalSections()
vfbasics.dll!AVrfpCheckThreadTermination()
vfbasics.dll!AVrfpCheckCurrentThreadTermination()
vfbasics.dll!AVrfpStandardThreadFunction()
kernel32.dll!@BaseThreadInitThunk@12()
ntdll.dll!__RtlUserThreadStart@8()

So in fact, Application Verifier has detected that the thread currently being terminated is still owning a critical section. What a stupid error message for a simple problem…

How to use manifests with build.exe

As of Windows Vista, basically all applications require a manifest in order to at least declare UAC compliance. Visual Studio has builtin support for creating and embedding manifests, so when using VS to build applications, using manifests is straightforward. However, when building a user mode application with the WDK and build.exe, things are a little different. Looking at the WDK documentation, manifests remain unmentioned — both in the context of UAC and SXS. Judging from documentation, it seems that the WDK does not provide any support for embedding manifests — which would mean that you are left with having to poke with the makefiles in order to invoke mt somewhere.

Looking at makefile.new, however, reveals that there are plenty of manifest-related rules and as it turns out, there indeed is support for manifests. On lines 1866 to 1925 (WDK 6000), makefile.new even contains a short documentation about the usage of manifests — so whether manifest support being unmentioned in the official docs is intentional or not is not quite clear. However, using the information in makefile.new, it is straightforward to get build.exe to embed manifests in a binary.

To embed a manifest, first create the manifest file and name it myapp.manifest. In the SOURCES file, include these lines:

SXS_APPLICATION_MANIFEST=myapp.manifest
SXS_ASSEMBLY_VERSION=1.0
SXS_ASSEMBLY_NAME=MyApp
SXS_ASSEMBLY_LANGUAGE=0000

That’s it.

In order not to have to provide a separate manifest file for different processor builds, there is some preprocessing taking place before the manifest is embedded. The following macros are available for use in the manifest:

  • SXS_ASSEMBLY_NAME (set in SOURCES)
  • SXS_ASSEMBLY_VERSION (set in SOURCES, defaults to 5.1.0.0)
  • SXS_ASSEMBLY_LANGUAGE (set in SOURCES, LCID or 0000 for neutral)
  • SXS_PROCESSOR_ARCHITECTURE (set automatically)

When using these macros, note that they will be replaced by quoted text — as a consequence, you have to use them — a bit unintuitively — as follows (Note the missing quotes!):

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<assembly 
     xmlns="urn:schemas-microsoft-com:asm.v1" 
     manifestVersion="1.0"> 
  <assemblyIdentity version=SXS_ASSEMBLY_VERSION
     processorArchitecture=SXS_PROCESSOR_ARCHITECTURE
     name=SXS_ASSEMBLY_NAME/> 
  <description>Example</description>
  <trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
    <security>
      <requestedPrivileges>
        <requestedExecutionLevel level="asInvoker" 
          uiAccess="false"/>
      </requestedPrivileges>
    </security>
  </trustInfo>
</assembly>

After preprocessing, the file is embedded into the binary as follows:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!--  Copyright (c) Microsoft Corporation --> 
<assembly 
     xmlns="urn:schemas-microsoft-com:asm.v1" 
     manifestVersion="1.0">
  <assemblyIdentity version="1.0.0.0" 
     processorArchitecture="AMD64" 
     name="MyApp" /> 
  <description>Example</description> 
  <trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
  <security>
    <requestedPrivileges>
      <requestedExecutionLevel level="asInvoker" 
        uiAccess="false" /> 
    </requestedPrivileges>
  </security>
  </trustInfo>
</assembly>

Categories




About me

Johannes Passing, M.Sc., living in Berlin, Germany.

Besides his consulting work, Johannes mainly focusses on Win32, COM, and NT kernel mode development, along with Java and .Net. He also is the author of cfix, a C/C++ unit testing framework for Win32 and NT kernel mode, Visual Assert, a Visual Studio Unit Testing-AddIn, and NTrace, a dynamic function boundary tracing toolkit for Windows NT/x86 kernel/user mode code.

Contact Johannes: jpassing (at) acm org

Johannes' GPG fingerprint is BBB1 1769 B82D CD07 D90A 57E8 9FE1 D441 F7A0 1BB1.

LinkedIn Profile
Xing Profile
Github Profile