Although Hotpatching has caught a bit of attention, suprisingly little information has been published about its inner workings. As the technology is patented, however, there is quite a bit of information that can be obtained by reading the patent description. Moreover, there is this (admittedly very terse) discussion about the actual implementation of hotpatching.

Armed with this information, it is possible to get into more detail by looking what is actually happening under the hood when a hoftix is applied: I did so and chose KB911897 as an example, which fixes some flaw in mrxsmb.sys and rdbss.sys. I have also gone through the hassle of translating key parts of the respective assembly code back to C.

Preparing the machine

First, we need a proper machine image which can be used for the experiment. Unfortunately, KB911897 is an SP1 package, so we have to use an old Win 2003 Server SP1 system to apply this update. Once we have the machine running, we can attach the kernel debugger and see what is happening when the hotfix is installed.

Observing the update

When launched with /hotpatch:enable, after some initialization work, the updater calls NtSetSystemInformation (which delegates to ExApplyCodePatch) to apply the hotpatch. Hotpatching includes a coldpatch, which I do not care about here and the actual hotpatch. The first two calls to NtSetSystemInformation (and thus to ExApplyCodePatch) are coldpatching-related and I will thus ignore them here. The third call, however, is made to apply the actual hotpatch, so let’s observe this one further.

Requiring a kernel mode-patch, ExApplyCodePatch then calls MmHotPatchRoutine, which is where the fun starts. Expressed in C, MmHotPatchRoutine, MmHotPatchRoutine roughly looks like this (reverse engineered from assembly, might be slightly incorrect):

NTSTATUS MmHotPatchRoutine(
  __in PSYSTEM_HOTPATCH_CODE_INFORMATION RemoteInfo
  )
{
  UNICODE_STRING ImageFileName;
  DWORD Flags = RemoteInfo->Flags;
  PVOID ImageBaseAddress;
  PVOID ImageHandle;
  NTSTATUS Status, LoadStatus;
  KTHREAD CurrentThread;

  ImageFileName.Length = RemoteInfo->KernelInfo.NameLength;
  ImageFileName.MaximumLength = RemoteInfo->KernelInfo.NameLength;
  ImageFileName.Buffer = ( PBYTE ) RemoteInfo + NameOffset;

  CurrentThread = KeGetCurrentThread();
  KeEnterCriticalRegion( CurrentThread );

  KeWaitForSingleObject(
    MmSystemLoadLock,
    WrVirtualMemory,
    0,
    0,
    0 );

  LoadStatus = MmLoadSystemImage(
    &ImageFileName,
    0,
    0,
    0,
    &ImageHandle,
    &ImageBaseAddress );
  if ( NT_SUCCESS( Status ) || Status == STATUS_IMAGE_ALREADY_LOADED )
  {

    Status = MiPerformHotPatch(
      ImageHandle,
      ImageBaseAddress,
      Flags );
    
    if ( NT_SUCCESS( Status ) || LoadStatus == STATUS_IMAGE_ALREADY_LOADED )
    {
      NOTHING;
    }
    else
    {
      MmUnloadSystemImage( ImageHandle );
    }
    
    LoadStatus = Status;
  }


  KeReleaseMutant(
    MmSystemLoadLock,
    1,  // increment
    FALSE,
    FALSE );

  KeLeaveCriticalRegion( CurrentThread );

  return LoadStatus;
}

As you see in the code, MmHotPatchRoutine will try load the hotpatch image — we can verify this in the debugger:

kd> bp nt!MmLoadSystemImage

kd> g
Breakpoint 3 hit
nt!MmLoadSystemImage:
808ec4b5 6878010000      push    178h

kd> k
ChildEBP RetAddr  
f6acbb28 80990c9e nt!MmLoadSystemImage
f6acbb68 809b2d67 nt!MmHotPatchRoutine+0x59
f6acbba8 808caeff nt!ExApplyCodePatch+0x191
f6acbd50 8082337b nt!NtSetSystemInformation+0xa1e
f6acbd50 7c82ed54 nt!KiFastCallEntry+0xf8
0006bc50 7c821f24 ntdll!KiFastSystemCallRet
0006bd44 7c8304c9 ntdll!ZwSetSystemInformation+0xc
[...]

kd> dt _UNICODE_STRING poi(@esp+4)
ntdll!_UNICODE_STRING
 "\??\c:\windows\system32\drivers\hpf3.tmp"
   +0x000 Length           : 0x50
   +0x002 MaximumLength    : 0x50
   +0x004 Buffer           : 0x81623fa8  "\??\c:\windows\system32\drivers\hpf3.tmp"
   
kd> gu

kd> lm
start    end        module name
[...]           
f6ba4000 f6bad000   hpf3       (deferred)  
[...]
f95cb000 f9641000   mrxsmb     (deferred)  
f9641000 f9671000   rdbss      (deferred)      
[...]

Having loaded the hotpatch image, MmHotPatchRoutine proceeds be calling MiPerformHotPatch, which looks about like this:

NTSTATUS
MiPerformHotPatch(
  IN PLDR_DATA_TABLE_ENTRY ImageHandle,
  IN PVOID ImageBaseAddress,
  IN DWORD Flags
  )
{
  PHOTPATCH_HEADER SectionData ;
  PRTL_PATCH_HEADER Header;    
  NTSTATUS Status;
  PVOID LockVariable;
  PVOID LockedBuffer;
  BOOLEAN f;
  PLDR_DATA_TABLE_ENTRY LdrEntry;

  SectionData = RtlGetHotpatchHeader( ImageBaseAddress );
  if ( ! SectionData  )
  {
    return STATUS_INVALID_PARAMETER;
  }
  
  //
  // Try to get header from MiHotPatchList
  //
  Header = RtlFindRtlPatchHeader(
    MiHotPatchList,
    ImageHandle );

  if ( ! Header )
  {
    PLIST_ENTRY Entry;

    if ( Flags & FLG_HOTPATCH_ACTIVE )
    {
      return STATUS_NOT_SUPPORTED;
    }

    Status = RtlCreateHotPatch(
      &Header,
      SectionData,
      ImageHandle,
      Flags
      );
    if ( ! NT_SUCCESS( Status ) )
    {
      return Status;
    }

    ExAcquireResourceExclusiveLite(
      PsLoadedModuleResource,
      TRUE
      );

    Entry =  PsLoadedModuleList;
    while ( Entry != PsLoadedModuleList )
    {
      LdrEntry = DataTableEntry = CONTAINING_RECORD( Entry,
                                            KLDR_DATA_TABLE_ENTRY,
                                            InLoadOrderLinks )
      if ( LdrEntry->DllBase DllBase >= MiSessionImageEnd )
      {
        if ( RtlpIsSameImage( Header, LdrEntry ) )
        {
          break;
        }
      }
    }

    ExReleaseResourceLite( PsLoadedModuleResource );

    if ( ! PatchHeader->TargetDllBase )
    {
      Status = STATUS_DLL_NOT_FOUND ;
    }

    Status = ExLockUserBuffer(
      ImageHandle->DllBase,
      ImageHandle->SizeOfImage,
      KernelMode,
      IoWriteAccess,
      LockedBuffer,
      LockVariable
      );
    if ( ! NT_SUCCESS( Status ) )
    {
      FreeHotPatchData( Header );
      return Status;
    }


    Status = RtlInitializeHotPatch(
      ( PRTL_PATCH_HEADER ) Header,
      ( PBYTE ) LockedBuffer - ImageHandle->DllBase
      );

    ExUnlockUserBuffer( LockVariable );

    if ( ! NT_SUCCESS( Status ) )
    {
      FreeHotPatchData( ImageHandle );
      return Status;
    }

    f = 1;
  }
  else
  {
    if ( ( Flags ^ ImageHandle->CodeInfo->Flags ) & FLG_HOTPATCH_ACTIVE )
    {
      return STATUS_NOT_SUPPORTED;
    }

    if ( ! ( ImageHandle->CodeInfo->Flags & FLG_HOTPATCH_ACTIVE ) )
    {
      Status = RtlReadHookInformation( Header );
      if ( ! NT_SUCCESS( Status ) )
      {
        return Status;
      }
    }

    f = 0;
  }
  
  Status = MmLockAndCopyMemory(
    ImageHandle->CodeInfo,
    KernelMode
    );
  if ( NT_SUCCESS( Status ) )
  {
    if ( ! f  )
    {
      return Status;
    }

    LdrEntry->EntryPointActivationContext = Header;  // ???
    InsertTailList( MiHotPatchList, LdrEntry->PatchList );
  }
  else
  {
    if ( f ) 
    {
      RtlFreeHotPatchData( Header );
    }
  }

  return Status;
}

So MiPerformHotPatch inspects the hotpatch information stored in the hotpatch image. This data includes information about which code regions need to be updated. After the neccessary information has been gathered, it applies the code changes.

Two basic problems have to be overcome now: On the one hand, all code sections of drivers are mapped read/execute only. Overwring the instructions thus does not work. On the other hand, the system has to properly synchronize the patching process, i.e. it has to make sure no CPU is currently executing the code that is about to be patched.

To overcome the memory protection problems, Windows facilitates a trick I previously only knew from malware: It creates a memory descriptor list (MDL) for the affected code region, maps the MDL, and updates the code through this mapped region. The memory protection is thus circumvented. As it turns, out, there is even a handy, undocumented helper routine for this purpose: ExLockUserBuffer, which is used by MiPerformHotPatch.

To proceed, MiPerformHotPatch calls MmLockAndCopyMemory to do the actual patching. So how does Windows synchronize the update process? Again, it uses a technique I assumed was a malware trick: It schedules CPU-specific DPCs on all CPUs but the current and keeps those DPCs busy while the current thread is uddating the code. Again, Windows provides a neat routine for that: KeGenericCallDpc. In addition to this, Windows raises the IRQL to clock level in order to mask all interrupts.

Here is the pseudo-code for MmLockAndCopyMemory and its helper, MiDoCopyMemory:

NTSTATUS
MmLockAndCopyMemory (
    IN PSYSTEM_HOTPATCH_CODE_INFORMATION PatchInfo,
    IN KPROCESSOR_MODE ProbeMode
    )
{
  PVOID Buffer;
  NTSTATUS Status;
  UINT Index;

  if ( 0 == PatchInfo->CodeInfo.DescriptorsCount )
  {
    return STATUS_SUCCESS;
  }

  Buffer = ExAllocatePoolWithQuotaTag( 
    9,
    PatchInfo->CodeInfo.DescriptorsCount * 2,
    'PtoH' );
  if ( ! Buffer )
  {
    return STATUS_INSUFFICIENT_RESOURCES;
  }
  RtlZeroMemory( Buffer, PatchInfo->CodeInfo.DescriptorsCount * 2 );

  if ( 0 == PatchInfo->CodeInfo.DescriptorsCount )
  {
    Status = STATUS_INVALID_PARAMETER;
    goto Cleanup;
  }

  for ( Index = 0; Index CodeInfo.DescriptorsCount; Index++ )
  {
    if ( PatchInfo->CodeInfo.CodeDescriptors[ Index ].CodeOffset > PatchInfo->InfoSize ||
       PatchInfo->CodeInfo.CodeDescriptors[ Index ].CodeSize > PatchInfo->InfoSize ||
       PatchInfo->CodeInfo.CodeDescriptors[ Index ].CodeOffset +
       PatchInfo->CodeInfo.CodeDescriptors[ Index ].CodeSize > PatchInfo->InfoSize || 
       /* other checks... */ )
    {
      Status = STATUS_INVALID_PARAMETER;
      goto Cleanup;
    }

    Status = ExLockUserBuffer(
      TargetAddress,
      PatchInfo->CodeInfo.CodeDescriptors[ Index ].CodeSize
      ProbeMode,
      IoWriteAccess,
      &PatchInfo->CodeInfo.CodeDescriptors[ Index ].MappedAddress,
      Buffer[ Index ]
      );
    if ( ! NT_SUCCESS( Status ) )
    {
      goto Cleanup;
    }
  }

  PatchInfo->Flags |= FLG_HOTPATCH_ACTIVE;

  KeGenericCallDpc(
    MiDoCopyMemory,
    PatchInfo );

  if ( PatchInfo->Flags & FLG_HOTPATCH_VERIFICATION_ERROR )
  {
    PatchInfo->Flags &= ~FLG_HOTPATCH_ACTIVE;
    PatchInfo->Flags &= ~FLG_HOTPATCH_VERIFICATION_ERROR;
    Status = STATUS_DATA_ERROR;
  }

Cleanup:
  if ( PatchInfo->CodeInfo.DescriptorsCount > 0 )
  {
    for ( Index = 0; Index CodeInfo.DescriptorsCount; Index++ )
    {
      ExUnlockUserBuffer( Buffer[ Index ] );
    }
  }

  ExFreePoolWithTag( Buffer, 0 );
  return Status;
}

VOID MiDoCopyMemory(
  IN PKDPC Dpc,
  IN PSYSTEM_HOTPATCH_CODE_INFORMATION PatchInfo,
  IN ULONG NumberCpus,
  IN DEFERRED_REVERSE_BARRIER ReverseBarrier
  )
{
  KIRQL OldIrql;
  UNREFERENCED_PARAMETER( Dpc );
  NTSTATUS Status;
  ULONG Index;

  OldIrql = KfRaiseIrql( CLOCK1_LEVEL );

  //
  // Decrement reverse barrier count.
  //
  Status = KeSignalCallDpcSynchronize( ReverseBarrier );
  if ( ! NT_SUCCESS( Status ) )
  {
    goto Cleanup;
  }

  PatchInfo->Flags &= ~FLG_HOTPATCH_VERIFICATION_ERROR;
    
  for ( Index = 0; Index CodeInfo.DescriptorsCount; Index++ )
  {
    if ( PatchInfo->Flags & FLG_HOTPATCH_ACTIVE )
    {
      if ( PatchInfo->CodeInfo.CodeDescriptors[ Index ].ValidationSize != 
        RtlCompareMemory(
          PatchInfo->CodeInfo.CodeDescriptors[ Index ].MappedAddress,
          ( PBYTE ) PatchInfo + PatchInfo->CodeInfo.CodeDescriptors[ Index ].ValidationOffset,
          PatchInfo->CodeInfo.CodeDescriptors[ Index ].ValidationSize ) )
      {

        if ( PatchInfo->CodeInfo.CodeDescriptors[ Index ].CodeSize != 
          RtlCompareMemory(
            PatchInfo->CodeInfo.CodeDescriptors[ Index ].MappedAddress,
            ( PBYTE ) PatchInfo + PatchInfo->CodeInfo.CodeDescriptors[ Index ].OrigCodeOffset,
            PatchInfo->CodeInfo.CodeDescriptors[ Index ].CodeSize ) )
        {
          PatchInfo->Flags &= FLG_HOTPATCH_VERIFICATION_ERROR;
          break;
        }
      }
    }
    else
    {
      if ( PatchInfo->CodeInfo.CodeDescriptors[ Index ].CodeSize !=
        RtlComparememory(
          PatchInfo->CodeInfo.CodeDescriptors[ Index ].MappedAddress,
          ( PBYTE ) PatchInfo + PatchInfo->CodeInfo.CodeDescriptors[ Index ].CodeOffset,
          PatchInfo->CodeInfo.CodeDescriptors[ Index ].CodeSize ) )
      {
        PatchInfo->Flags &= FLG_HOTPATCH_VERIFICATION_ERROR;
        break;
      }
    }
  }

  //loc_479533
  if ( PatchInfo->Flags & FLG_HOTPATCH_VERIFICATION_ERROR ||
     PatchInfo->CodeInfo.DescriptorsCount <= 0 )
  {
    goto Cleanup;
  }

  for ( Index = 0; Index CodeInfo.DescriptorsCount; Index++ )
  {
    PVOID Source;
    if ( PatchInfo->Flags & FLG_HOTPATCH_ACTIVE )
    {
      Source = ( PBYTE ) PatchInfo + PatchInfo->CodeInfo.CodeDescriptors[ Index ].CodeOffset;
    }
    else
    {
      Source = ( PBYTE ) PatchInfo + PatchInfo->CodeInfo.CodeDescriptors[ Index ].OrigCodeOffset;
    }

    RtlCopyMemory(
      PatchInfo->CodeInfo.CodeDescriptors[ Index ].MappedAddress,
      Source,
      PatchInfo->CodeInfo.CodeDescriptors[ Index ].CodeSize
      );
  }


Cleanup:
   KeSignalCallDpcSynchronize( ReverseBarrier );
   KfLowerIrql( OldIrql );
   KeSignalCallDpcDone( NumberCpus );
}

To see the code, in action, we set a breakpoint on nt!MiDoCopyMemory:

kd> k
ChildEBP RetAddr  
f6acbac0 8087622f nt!MiDoCopyMemory
f6acbae8 80990a10 nt!KeGenericCallDpc+0x3d
f6acbb0c 80990bea nt!MmLockAndCopyMemory+0xf1
f6acbb34 80990cba nt!MiPerformHotPatch+0x143
f6acbb68 809b2d67 nt!MmHotPatchRoutine+0x75
f6acbba8 808caeff nt!ExApplyCodePatch+0x191
f6acbd50 8082337b nt!NtSetSystemInformation+0xa1e

Before letting MiDoCopyMemory do its work, let’s see what it is about to do. No modifications have yet been done to mrxsmb:

kd> !chkimg mrxsmb
0 errors : mrxsmb 

kd> !chkimg rdbss
0 errors : rdbss

The second argument is a structure holding the information garthered previously, peeking into it reveals:

kd> dd /c 1 poi(esp+8) l 4
81583008  00000001
8158300c  00000149
81583010  00000008   <-- # of code patches
81583014  f9648b1f   <-- hmm...

As it turns out, address 81583014 refers to a variable length array of size 8. Poking aroud with dd, the following listing suggests that the structure is of size 28 bytes:

kd> dd /c 7 81583014
81583014  f9648b1f fa2afb1f 000000ec 00000005 000000f1 000000f6 00000005
81583030  f9648b24 fa2b2b24 000000fb 00000002 000000fd 000000ff 00000002
8158304c  f96585ef fa2b15ef 00000101 00000005 00000106 0000010b 00000005
81583068  f96585f4 fa2b45f4 00000110 00000002 00000112 00000114 00000002
81583084  f9658569 fa2b3569 00000116 00000005 0000011b 00000120 00000005
815830a0  f965856e fa2b656e 00000125 00000002 00000127 00000129 00000002
815830bc  f9653378 fa2b5378 0000012b 00000005 00000130 00000135 00000005
815830d8  f965337d fa2b837d 0000013a 00000005 0000013f 00000144 00000005

Given that rdbss was loaded to address range f9641000-f9671000, it is obvious that the first 2 columns refer to code addresses. The third, fifth and sixth column looks like an offset, the fourth and seventh like the length of the code change. First, let’s see where the first column points to:

kd> u f9648b1f
rdbss!RxInitiateOrContinueThrottling+0x6b:
f9648b1f 90              nop
f9648b20 90              nop
f9648b21 90              nop
f9648b22 90              nop
f9648b23 90              nop
rdbss!RxpCancelRoutine:
f9648b24 8bff            mov     edi,edi
f9648b26 55              push    ebp
f9648b27 8bec            mov     ebp,esp

Now that looks promising, especially since the fourth column holds the value 5. Let’s look at the second row:

kd> u f9648b24
rdbss!RxpCancelRoutine:
f9648b24 8bff            mov     edi,edi

No doubt, the first and second row define the two patches necessary to redirect RxpCancelRoutine. But what to replace this code with? As it turns out, the offsets in column three are relative to the structure and point to the code that is to be written:

kd> u poi(esp+8)+000000ec
815830f4 e9dcc455fd      jmp     7eadf5d5          mov     edi,edi

kd> u poi(esp+8)+000000fb
81583103 ebf9            jmp     815830fe

That makes perfectly sense — the five nops are to be overwritten by a near jump, the mov edi, edi will be replaced by a short jump.

So let’s run MiDoCopyMemory and have a look at the results. Back in MmLockAndCopyMemory, the code referred to by the first to rows look like this:

kd> u f9648b1f
rdbss!RxInitiateOrContinueThrottling+0x6b:
f9648b1f e9dcc455fd      jmp     hpf3!RxpCancelRoutine (f6ba5000)

kd> u f9648b24
rdbss!RxpCancelRoutine:
f9648b24 ebf9            jmp     rdbss!RxInitiateOrContinueThrottling+0x6b (f9648b1f)
f9648b26 55              push    ebp
f9648b27 8bec            mov     ebp,esp

Voilà, RxpCancelRoutine has been patched and calls are redirected to hpf3!RxpCancelRoutine, the new routine located in the auxiliarry ‘hpf3′ driver. All that remains to be done is cleanup (unlocking the memory etc).

That’s it — that’s how Windows applies patches on the fly using hotpatching. Too bad that the technology is so rarely used in practice.

Trying to applying hotfixes on the fly introduces a variety of problems — the most important being:

• Patching code that is currently in use
• Atomically replacing files on disk that are currently in use and therefore locked
• Making sure that all changes take effect for both, processes currently running and processes which are yet to be started (i.e. before the next reboot)
• Allowing further hotfixes to be applied on system that has not been rebooted since the last hotfix has been applied in an on-the-fly fashion

The Windows Hotpatching infrastructure is capable of handling all these cases — it is, however, not applicable to all kinds of code fixes. Generally speaking, it can only be used for fixes that merely comprise smallish code changes but do not affect layout or semantics of data structures. A fix for a buffer overflow caused by an off-by-one error, however, is a perfect example for a fix that could certainly be applied using the Hotpatching infrastructure.

That all sounds good and nice, but reality is that we still reboot our machines for just about every update Microsoft provides us, right?

Right. The answer for this is threefold. First, as indicated, some hotfixes can be expected to make changes that cannot be safely applied using the Hotpatching system. Secondly, Hotpatching is used on an opt-in basis, so you will not benefit from it automatically: When a hotpatch-enabled hotfix is applied through Windows Update or by launching the corresponding exe file, it is not used and a reboot will be required. The user has to explicitly specify the /hotpatch:enable switch in order to have the hotfix to be applied on the fly.

In the months after the release of SP1, a certain fraction of the hotfixes issued by Microsoft were indeed hotpatch-enabled and could be applied without a reboot. Interestingly, however, I am not aware of a single hotfix issued since Server 2003 SP2 that supported hotpatching!

And thirdly: Whether Microsoft has lost faith in their hotpatching facility, whether the effort to test such hotfixes turned out to be too high or whether there were other reasons speaking against issueing hotpatch-enabled hotfixes — I do not know.

Notwithstanding this observation, Hotpatching is an interesting technology that deserves to be looked at in more detail. Although I will not cover the entire infrastructure, I will spend at least one more blog post on the mechanisms implemented in Windows that allow code modifications to be performed on the fly. That is, I will focus on the hotpatching part of the infrastructure and will ignore coldpatching and other, smaller aspects of the infrastructre.

While the MSDN documentation on CreateProcess does contain a remark saying that current directory information (=C: and friends) should be included in such a custom environment block, it does not mention the importance of SystemRoot.

The SystemRoot environment variable usually contains the path c:\windows — the path that is also accessible using the GetWindowsDirectory function. This environment variable, as it turns out, is not only handy for scripting purposes — it is, in fact, essential for the proper operation of many libraries.

For very simple programs, forgetting to include SystemRoot in a custom environment block usually goes unnoticed — even an empty environment block works just fine. In case of more complex applications, however, the omission of this variable can quickly lead to errors — on Vista, the most common error that can be tracked back to a missing SystemRoot variable is SXS failing to find/load basic system libraries.

Now that we have Windows 7, SystemRoot seems to have become even more important: Now it is not only SXS that requires SystemRoot to be specified properly, but also CryptoAPI.

In my particular case, I was experiencing a 0×80090006 (“Invalid Signature”, NTE_BAD_SIGNATURE) error whenever the child process attempted to call CoGetObject to retrieve a pointer to a DCOM object. While this error occured on Windows 7, the same code worked fine on Windows Vista and XP.

Given this more than general error message, it seemed anything but clear to me what the problem was, so I attached a debugger to the child process (using gflags/Image File Execution Options). Once I did that, I got the following messages in my debug output output:

CryptAcquireContext: CheckSignatureInFile failed at cryptapi.c line 5198
CryptAcquireContext: Failed to read registry signature value at cryptapi.c line 873

I set a breakpoint on CryptAcquireContextW and looked at the stack trace:

0:000> k
ChildEBP RetAddr  
0008f8a4 75760a4f ole32!CRandomNumberGenerator::Initialize+0x2e
0008f8b0 75760769 ole32!CRandomNumberGenerator::GenerateRandomNumber+0xd
0008f8e8 757609cf ole32!CStdMarshal::AddIPIDEntry+0x48
0008f93c 75766aae ole32!CStdMarshal::MarshalServerIPID+0x5a
0008f994 75767519 ole32!CStdMarshal::MarshalObjRef+0xb9
0008f9c8 7576778e ole32!MarshalInternalObjRef+0x8c
0008fa4c 757676ba ole32!CRemoteUnknown::CRemoteUnknown+0x3b
0008fa8c 7576754a ole32!CComApartment::InitRemoting+0x19c
0008fa98 7586d83e ole32!CComApartment::StartServer+0x13
0008faa8 757652b3 ole32!InitChannelIfNecessary+0x1e
0008fb20 757fc046 ole32!CoUnmarshalInterface+0x38
0008fb34 757fd3d5 ole32!CObjrefMoniker::Load+0x26
0008fb70 7573cb7f ole32!CObjrefMonikerFactory::ParseDisplayName+0x16f
0008fbbc 7573caae ole32!FindClassMoniker+0x8b
0008fbf4 75789dc7 ole32!MkParseDisplayName+0xbb
0008fc3c 6954ce84 ole32!CoGetObject+0x82
...

Quite obviously, COM, trying to unmarshal an interface, needed a random number and attempted to use CryptoAPI for this purpose. Looking at the paramters of CryptAcquireContext, I saw that the Microsoft Strong Cryptographic Provider was attempted to be loaded — one of the standard Windows CSPs — so everything seemed normal.

Guided by the message Failed to read registry signature, I switched to Process Monitor to see which registry key was being queried: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Cryptography\Defaults\Provider\Microsoft Strong Cryptographic Provider.

Taking a look at this key in regedit, it did not take long before spotting SystemRoot as the culprit:

Looking at file system activity in Process Monitor proved this:

Interestingly, on Windows Vista, all Image Path values in the CSP registry keys do not use SystemRoot — they just contain the file name and rely on the library path in order to locate the libraries at runtime. This explains why my code worked fine on Vista.

(While making this change, the developer seemed to forget to change the value’s type from REG_SZ to REG_EXPAND_SZ though :) )

Bottom Line 1: Always, always include SystemRoot when passing a custom environment block to CreateProcess.

Bottom Line 2: The case also shows how a seemingly trivial change in Windows (using an absolute path rather then just a file name in the CSP registry key) can lead to an application incompatibility.

There is little doubt that native code, and C and C++ in particular, is here to stay. And still, it is pretty obvious that when it comes to tools and IDEs, it is the managed world that has gotten most attention from tool vendors over the past years.

While there are lots and lots of useful tools for native development, many of them probably even better than their managed counterparts, there are some areas where the managed language fraction is far ahead: One of these areas certainly is IDE support for unit testing.

JUnit for Eclipse, TestDriven.Net for Visual Studio and MS Test make test-driven development so much more convenient and efficient that it is almost ridiculous that using command line tools is still state of the art for C/C++ development.

That said, there is great news: With cfix studio, there finally is a solution filling in this gap! cfix studio, based on the cfix unit testing framework, is a Visual Studio-AddIn that allows you to easily write, manage, run, and debug your unit tests from within Visual Studio. No fiddling with command line tools, complex configuration, or boilerplate code required!

Among lots of other features, cfix studio also has first-class support for multi-architecture development – you can easily switch back and forth between 32-bit and 64-bit and can even mix tests of different architectures in a single test run. Needless to say, cfix studio, like cfix, is also fully compatible to WinUnit.

If that has caught your interest, you are invited to check out the first beta version of cfix studio:

It is free, quick to install and comes with a set of example projects. Give it a try — and please let me know about all your crticism, suggestions and other feedback!

Here are some screenshots of cfix studio in action:

The Test Explorer allows you to start a single or set of tests, the Run Window shows the results

Run Window: Viewing test progress

Run Window: Viewing test results and details of a failed assertion

When running in the debugger, a failed assertion will hit a breakpoint and the Run Window will show additional details

Updated cfix binaries and source code are now available for download

Under certain circumstances, however, this versioning information may be too imprecise: Versions are not necessarily incremented after each build, so it is possible that two copies of a module carry the same versioning information, yet differ significantly in their implementation. In such situations, identifying the actual build of the module might become neccessary.

The most common, but by no means the only situation in which this applies in practice concerns debugging — to identify the PDB file exactly matching a given module, the debugger must be able to recognize the specific build of a module. It thus does not come as a surprise that all images for which debugging information has been generated contain a dedicated identifier for this purpose: The CodeView signature GUID.

Summarizing what Oleg Starodumov has covered in more detail, cl, when directed to generate a PDB file, implicitly creates this GUID and, along with the path to the PDB file, embeds this data into the PE image. For current versions, the relevant structure used to encode this information is CV_INFO_PDB70, which seems to have been documented once, but not any more:

typedef struct _CV_INFO_PDB70
{
  ULONG CvSignature;
  GUID Signature;
  ULONG Age;
  UCHAR PdbFileName[ ANYSIZE_ARRAY ];
} CV_INFO_PDB70, *PCV_INFO_PDB70;

In order to be able to locate the structure within the PE image, a directory entry of type IMAGE_DEBUG_TYPE_CODEVIEW is written to the image’s debug directory. The following code listing demonstrates how to obtain the signature GUID of an image:

#define PtrFromRva( base, rva ) ( ( ( PUCHAR ) base ) + rva )

static PIMAGE_DATA_DIRECTORY GetDebugDataDirectory(
  __in ULONG_PTR LoadAddress
  )
{
  PIMAGE_DOS_HEADER DosHeader = 
    ( PIMAGE_DOS_HEADER ) ( PVOID ) LoadAddress;
  PIMAGE_NT_HEADERS NtHeader = ( PIMAGE_NT_HEADERS ) 
    PtrFromRva( DosHeader, DosHeader->e_lfanew );
  ASSERT ( IMAGE_NT_SIGNATURE == NtHeader->Signature );

  return &NtHeader->OptionalHeader.DataDirectory
      [ IMAGE_DIRECTORY_ENTRY_DEBUG ];
}

NTSTATUS GetDebugGuid(
  __in ULONG_PTR ModuleBaseAddress,
  __out GUID *Guid
  )
{
  PIMAGE_DATA_DIRECTORY DebugDataDirectory;
  PIMAGE_DEBUG_DIRECTORY DebugHeaders;
  ULONG Index;
  ULONG NumberOfDebugDirs;
  ULONG_PTR ModuleBaseAddress;
  NTSTATUS Status;

  DebugDataDirectory  = DebugDataDirectory( ModuleBaseAddress );
  DebugHeaders    = ( PIMAGE_DEBUG_DIRECTORY ) PtrFromRva( 
    ModuleBaseAddress, 
    DebugDataDirectory->VirtualAddress );

  ASSERT( ( DebugDataDirectory->Size % sizeof( IMAGE_DEBUG_DIRECTORY ) ) == 0 );
  NumberOfDebugDirs = DebugDataDirectory->Size / sizeof( IMAGE_DEBUG_DIRECTORY );

  //
  // Lookup CodeView record.
  //
  for ( Index = 0; Index < NumberOfDebugDirs; Index++ )
  {
    PCV_INFO_PDB70 CvInfo;
    if ( DebugHeaders[ Index ].Type != IMAGE_DEBUG_TYPE_CODEVIEW )
    {
      continue;
    }

    CvInfo = ( PCV_INFO_PDB70 ) PtrFromRva( 
      ModuleBaseAddress, 
      DebugHeaders[ Index ].AddressOfRawData );

    if ( CvInfo->CvSignature != 'SDSR' )
    {
      //
      // Weird, old PDB format maybe.
      //
      return STATUS_xxx_UNRECOGNIZED_CV_HEADER;
    }

    *Guid = CvInfo->Signature;
    return STATUS_SUCCESS;  
  }

  return STATUS_xxx_CV_GUID_LOOKUP_FAILED;
}

The new MSI package is now available for download on Sourceforge.

New C++ API

To date, cfix has primarily focussed on C as the programming language to write unit tests in. Although C++ has always been supported, cfix has not made use of the additional capabilities C++ provides. With version 1.2, cfix makes C++ a first class citizen and introduces an additional API that leverages the benefits of C++ and allows writing test cases in a more convenient manner.

Being implemented on top of the existing C API, the C++ API is not a replacement, but rather an addition to the existing API set.

As the following example suggests, fixtures can now be written as classes, with test cases being implemented as methods:

#include <cfixcc.h>

class ExampleTest : public cfixcc::TestFixture
{
public:
  void TestOne() 
  {}
  
  void TestTwo() 
  {}
};

CFIXCC_BEGIN_CLASS( ExampleTest )
  CFIXCC_METHOD( TestOne )
  CFIXCC_METHOD( TestTwo )
CFIXCC_END_CLASS()

To learn more about the definition of fixtures, have a look at the respective TestFixture chapter in the cfix documentation.

Regarding the implementation of test cases, cfix adds a new set of type-safe, template-driven assertions that, for instance, allow convenient equality checks:

void TestOne() 
{
  const wchar_t* testString = L"test";
  
  //
  // Use typesafe assertions...
  //
  CFIXCC_ASSERT_EQUALS( 1, 1 );
  CFIXCC_ASSERT_EQUALS( L"test", testString );
  CFIXCC_ASSERT_EQUALS( wcslen( testString ), ( size_t ) 4 );
  
  //
  // ...log messages...
  //
  CFIX_LOG( L"Test string is %s", testString );
  
  //
  // ...or use the existing "C" assertions.
  //
  CFIX_ASSERT( wcslen( testString ) == 4 );
  CFIX_ASSERT_MESSAGE( testString[ 0 ] == 't', 
    L"Test string should start with a 't'" );
}

Again, have a look at the updated API reference for an overview of the new API additions.

Customizing Test Runs

Another important new feature is the addition of the new switches -fsf (Shortcut Fixture), -fsr (Shortcut Run), and -fss (Shortcut Run On Failing Setup). Using these switches allows you to specify how a test run should resume when a test case fails.

When a test case fails, the default behavior of cfix is to report the failure, and resume at the next test case. By specifying -fsf, however, the remaining test cases of the same fixture will be skipped and execution resumes at the next fixture. With -fsr, cfix can be requirested to abort the entire run as soon as a single test case fails.

What else is new in 1.2?

• CFIX_ASSERT_MESSAGE
• ANSI support for CFIX_LOG, CFIX_INCONCLUSIVE,
  and CFIX_ASSERT_MESSAGE (and the entire C++ API)
• CfixPeGetValue and CfixPeSetValue
• Kernel mode: Drivers do not need to link against aux_klib.lib any more
• Before and After routines
• Support for cl 13.00 and Visual Studio 2003 (in addition to 2005 and 2008)

Download

As always, cfix 1.2 is source and binary compatible to previous versions. The new MSI package and source code can now be downloaded on Sourceforge.

cfix is open source and licensed under the GNU Lesser General Public License.

All threads start off as Non-GUI threads. Once the application makes a call to a system service that does not fall within the default range, however, the NT kernel will suspect this thread to be about to do GUI stuff — and will convert the thread into a GUI thread.

Converting a thread to a GUI thread naturally has to entail two things — swapping the SSDT, and enlarging the stack. While swapping the SSDT is not really interesting, enlarging the stack size poses a challenge — you cannot really enlarge a stack as the nearby pages that would need to be acquired may not be available.

As a consequence, enlarging the stack works by swapping the stack. The old, small stack is exchanged against a newly allocated, larger stack. Now swapping a stack is not really a common thing to do and is pretty easy to get wrong. And well, as it turns out, the Svr03 kernel did in fact get it wrong.

But let’s start at the beginning.

When the number of the requested system service is found to be beyond the range supported by the default SSDT, KiConvertToGuiThread is called to perform the thread conversion. KiConvertToGuiThread itself is pretty dumb and lets PsConvertToGuiThread do the actual work.

The following pseudo code illustrates what PsConvertToGuiThread does:

NTSTATUS PsConvertToGuiThread()
{
  //
  // Create the new stack.
  //
  LargeStack = MmCreateKernelStack( ... )
  
  if ( LargeStack == NULL )
  {
    __try
    {
      //
      // Allocation failed -- set last error value.
      //
      NtCurrentTeb()->LastErrorValue = ERROR_NOT_ENOUGH_MEMORY;
    }
    __except( ... )
    {
    }
    
    //
    // N.B. We are still on the old stack.
    //
    
    //
    // This will copy the old thread's contents to the new stack and 
    // migrate the context of the current thread to the new stack.
    //
    SmallStack = KeSwitchKernelStack( LargeStack, ... );

    //
    // Now we are on the new stack.
    //
    MmDeleteKernelStack( SmallStack, ... );
  }
  ...
  //
  // Notify Win32k.
  //
  
  ( PspW32ProcessCallout )( ... )
  ...
  ( PspW32ThreadCallout ) ( ... )
  
  ...
}

This code looks innocent enough, but infact, it is lying. Too see why, you have to recall how Structured Exception Handling is implemented on i386 and how the C compiler makes use of it (I think I have spent way too much time with SEH over the past months…): The __try/__except-block at the top of the routine will cause to the compiler to emit the typical SEH prolog at the beginning of the function. The purpose of this prolog is to set up an EXCEPTION_REGISTRATION_RECORD and to put this record onto the current thread’s SEH chain, which in turn is rooted in the PCR. In the same way, the compiler will put an appropriate epilog to the end of the routine.

So while the code above suggests that the SEH stuff is scoped to the very beginning of the function, it will not be until the end of the function has been reached that the EXCEPTION_REGISTRATION_RECORD is torn down and removed from the SEH chain.

And at this point, it should become clear why this becomes a problem in the context of stack swapping. At the point where KeSwitchKernelStack is called, the EXCEPTION_REGISTRATION_RECORD will still be listed in the SEH chain, although it does not serve any particular purpose any more. So KeSwitchKernelStack is called, which will, as indicated before, copy the contents of the old stack to the new stack — which, of course, includes the EXCEPTION_REGISTRATION_RECORD.

But…

neither KeSwitchKernelStack, nor PsConvertToGuiThread updates the SEH pointer in the PCR! After the swapping has been conducted and MmDeleteKernelStack has returned, the root of the SEH chain will point to freed memory — memory where the EXCEPTION_REGISTRATION_RECORD once has been.

Now two things are worth noting. First, PsConvertToGuiThread can be expected to occupy the bottommost stack frame of the kernel stack. A situation where the dangling pointer could harm a caller of PsConvertToGuiThread is thus not possible.

Secondly, PsConvertToGuiThread makes callouts to Win32k by invoking the callbacks pointed to by PspW32ProcessCallout and PspW32ThreadCallout. And in fact, it is only PsConvertToGuiThread‘s luck that these routines are so well behaved that they do not cause the system to bugcheck because of the dangling pointer. If one of these routines (or routines called by these) did anything with the SEH chain going beyond adding another record to the chain and removing it later, odds were that this routine would dereference a stray pointer… and would bugcheck the system…

It is worth noting that the implementation of PsConvertToGuiThread has changed in Windows Vista, so that the above discussion does not apply to this and later releases.

One especially helpful tool for leak detection is the debug CRT. Although the leak detection facilities provided by the debug CRT are not as far-reaching as those of, say, UMDH, using the debug CRT is probably the most friction-less way of identifying leaks.

Of course, the debug CRT will only track allocations of the CRT heap. That is, allocations performed using malloc or, in case of C++, the default operator new.

So how to enable allocation tracking? As it turns out, it is already enabled by default for the debug heap — so changing the CRT flags using _CrtSetDbgFlag usually is not even neccessary. All there is to do is to call _CrtDumpMemoryLeaks() at the end of the program.

When exactly is “the end of the program”? That depends on which CRT you use. Each CRT uses a separate heap and thus, must have its resources be tracked separately. If your application EXE and your DLLs all link against the DLL version of the CRT, the right moment to call _CrtDumpMemoryLeaks() is at the end of main(). If you use the static CRT, the right moment is when the respective module is about to unload — for an EXE, this is the end of main() again (atexit is another option). For a DLL, however, this is DllMain (in the DLL_PROCESS_DETACH case).

To illustrate how to make use of this CRT feature, consider the following leaky code:

#include <crtdbg.h>

class Widget
{
private:
  int a;

public:
  Widget() : a( 0 )
  {
  }
};

void UseWidget( Widget* w )
{
}

int __cdecl wmain()
{
  Widget* w = new Widget();
  UseWidget( w );

  _CrtDumpMemoryLeaks();
  return 0;
}

Running the debug build (i.e. a build using the debug CRT) of this program will yield the following output in the debugger:

Detected memory leaks!
Dumping objects ->
{124} normal block at 0x008C2880, 4 bytes long.
 Data:  00 00 00 00 
Object dump complete.

So we have a memory leak — allocation #124 is not freed. The default procedure to locate the leak now is to include a call to _CrtSetBreakAlloc( 124 ) in the program and run it in the debugger — it will break when allocation #124 is performed. While this practice is ok for smaller programs, it will fail as soon as your program is not fully deterministic any more — most likely because it uses multiple threads. So for many programs, this technique is pretty much worthless.

But before continueing on the topic of how to find the culprit, there is another catch to discuss. Let’s include this snippet of code into our program:

class WidgetHolder
{
private:
  Widget* w;

public:
  WidgetHolder() : w( new Widget() )
  {}

  ~WidgetHolder()
  {
    delete w;
  }
};

WidgetHolder widgetHolder;

No leak here — we are properly cleaning up. But let’s see what the debugger window tells:

Detected memory leaks!
Dumping objects ->
{125} normal block at 0x000328C0, 4 bytes long.
 Data:  00 00 00 00 
{124} normal block at 0x00032880, 4 bytes long.
 Data:  00 00 00 00 
Object dump complete.

Urgh. But the reason should be obvious — when main() is about to return, ~WidgetHolder has not run yet. As a consequence, WidgetHolder’s allocation has not been freed yet and _CrtDumpMemoryLeaks will treat this as a leak. Unfortunately, there is no good way to avoid such false positives. Of course, this only holds for C++. For C, this problem does not exist.

Ok, back to the problem of locating the leak. We know that allocation #124 is the problem, but assuming our program does more than the simplistic example, breaking on #124 during the next runs is likely to lead us to ever changing locations. So this information is worthless. That leaves the address of the leaked memory — 0x008C2880.

At this point, we can leverage the fact that the CRT heap is not really a heap but just a wrapper around the RTL heap. Therefore, we can use the incredibly powerful debugging facilities of the RTL heap to help us out.

In order to fix a leak, it is usually extremely helpful to locate the code having conducted the allocation. Once you have this information, it is often trivial to spot the missing free operation. As it turns out, the the RTL heap’s page heap feature offers this capability.

Open Application Verifier and enable Heap Checks for our application. By default, this enables the full page heap, but the normal page heap is enough for our case.

Note that for the following discussion, I assume you are using the Visual Studio debugger.

Set a breakpoint on the statement immediately following the _CrtDumpMemoryLeaks() statement and run the application until it breaks there. This time, the locations 0x02CDFFA0 and 0x02CDFF40 are reported as being leaked. Do not continue execution yet.

Rather, open WinDBG and attach noninvasively to the debugged process. VisualStudio is already attached, so we cannot perform a real attach, but a noninvasive attach does the trick.

In WinDBG, we now use the !heap extension to query page heap information:

0:000> !heap -p -a 0x02CDFF40
    address 02cdff40 found in
    _HEAP @ 2cd0000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        02cdfef8 000c 0000  [00]   02cdff20    00028 - (busy)
        Trace: 02dc
        776380d8 ntdll!RtlDebugAllocateHeap+0x00000030
[...]
        6a2fab29 MSVCR80D!malloc+0x00000019
        6a34908f MSVCR80D!operator new+0x0000000f
        4115c9 Leak!WidgetHolder::WidgetHolder+0x00000049
        415808 Leak!`dynamic initializer for 'widgetHolder''+0x00000028
        6a2e246a MSVCR80D!_initterm+0x0000001a
        411d33 Leak!__tmainCRTStartup+0x00000103
        411c1d Leak!wmainCRTStartup+0x0000000d
        767b19f1 kernel32!BaseThreadInitThunk+0x0000000e
        7764d109 ntdll!_RtlUserThreadStart+0x00000023

0:000> !heap -p -a 0x02CDFFA0
    address 02cdffa0 found in
    _HEAP @ 2cd0000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        02cdff58 000c 0000  [00]   02cdff80    00028 - (busy)
        Trace: 02e0
        776380d8 ntdll!RtlDebugAllocateHeap+0x00000030
[...]
        6a2fab29 MSVCR80D!malloc+0x00000019
        6a34908f MSVCR80D!operator new+0x0000000f
        411464 Leak!wmain+0x00000044
        411dd6 Leak!__tmainCRTStartup+0x000001a6
        411c1d Leak!wmainCRTStartup+0x0000000d
        767b19f1 kernel32!BaseThreadInitThunk+0x0000000e
        7764d109 ntdll!_RtlUserThreadStart+0x00000023
        

Aha, stack traces! The remaining analysis is almost trivial: 0x02CDFF40 has been allocated on behalf of WidgetHolder::WidgetHolder. WidgetHolder::WidgetHolder, however, is not indirectly invoked by wmain, but rather by MSVCR80D!_initterm! That is a strong indication for this being a global object that can be ignored in this analysis.

0x02CDFFA0, in turn, is allocated by wmain, so this is a real leak. But which allocation is it, exactly? lsa will tell us:

0:000> lsa Leak!wmain+0x00000044
    33: }
    34: 
    35: int __cdecl wmain()
    36: {
>   37: 	Widget* w = new Widget();
    38: 	UseWidget( w );
    39: 
    40: 	_CrtDumpMemoryLeaks();
    41: 	return 0;
    42: }

There you go, we have found the culprit.

Although simple, I have found this technique to be very effective in practice, as it enables you to find leaks as you develop your code. As Application Verifier should be enabled anyway for any application you are developing on, the technique also turns out to be a lot less laborious than it may seem. It almost certainly is a lot more convenient than routinely doing UMDH runs. To be fair, however, UMDH is able to catch more leaks (non CRT-leaks), so additionally using UMDH remains being a good idea.

To test that, I recently started a Windows 2003 Server VM, installed VCExpress 2008 and cfix and attempted to run an example project in the VC debugger. As long as no assertions fired, everything seemed fine. I then altered the example’s source code so that one of the assertion would fail, ran it in the debugger — and waited. Nothing happened.

When an assertion fires, cfix 1.1 attempts to capture the stack trace to make debugging easier. For this to work, the respective debugging symbols must be loaded, and — if not yet available — be downloaded from the symbol server. It is thus not uncommon that it takes a couple of seconds before the failure message appears and the debugger breaks in when an assertion fails. But after about a minute, still nothing happened — a deadlock seems to have occured.

To make the situation worse, I then noticed that while cfix was hanging, Internet Explorer would not properly start any more. After creating its main window, it would hang as well.

So I rebooted the machine and hooked up a kernel debugger. Luckily, the scenario was easy to reproduce and I could take a closer look…

Debugging the deadlock

First, let’s find the cfix32 process:

kd> !process 0 0 cfix32.exe
PROCESS 8254e490  SessionId: 0  Cid: 06b0    Peb: 7ffde000  ParentCid: 0658
    DirBase: 06e90000  ObjectTable: e170ee58  HandleCount:  69.
    Image: cfix32.exe

Now that we have the VA, we can switch to the process and load its user mode symbols:

kd> .process /r /p 8254e490  
Implicit process is now 8254e490
.cache forcedecodeuser done
Loading User Symbols
...............................

Looking at this process’ threads, it is easy to see that one stack looks suspicious:

kd> !process 8254e490  
PROCESS 8254e490  SessionId: 0  Cid: 06b0    
	Peb: 7ffde000  ParentCid: 0658
    DirBase: 06e90000  ObjectTable: e170ee58  HandleCount:  69.
    Image: cfix32.exe
    VadRoot 82455628 Vads 76 Clone 0 Private 491. 
    Modified 8. Locked 0.
    DeviceMap e148f3c0

    [...]

THREAD 826276f8  Cid 06b0.06b4  Teb: 7ffdd000 Win32Thread: 
  e1734a38 WAIT: (Unknown) KernelMode Non-Alertable
	f4422b44  SynchronizationEvent
Not impersonating
DeviceMap                 e148f3c0
Owning Process            8254e490       Image:         cfix32.exe
Wait Start TickCount      13474          Ticks: 9138 (0:00:01:31.511)
Context Switch Count      219                 LargeStack
UserTime                  00:00:00.010
KernelTime                00:00:00.010
Win32 Start Address cfix32!wmainCRTStartup (0x010048cf)
Start Address kernel32!BaseProcessStartThunk (0x77e617f8)
Stack Init f4423000 Current f4422aa0 Base f4423000 Limit f441f000 Call 0
Priority 10 BasePriority 8 PriorityDecrement 0

ChildEBP RetAddr  
f4422ab8 808202b6 nt!KiSwapContext+0x25
f4422ad0 8081fb6e nt!KiSwapThread+0x83 
f4422b14 809bae63 nt!KeWaitForSingleObject+0x2e0 
f4422bf4 809bc06d nt!DbgkpQueueMessage+0x178 
f4422c18 8096ba9a nt!DbgkpSendApiMessage+0x45 
f4422cc8 80909942 nt!DbgkMapViewOfSection+0xcf 
f4422d34 8082350b nt!NtMapViewOfSection+0x269 
f4422d34 7c8285ec nt!KiFastCallEntry+0xf8 
00069b94 7c82728b ntdll!KiFastSystemCallRet 
00069b98 7c831e05 ntdll!NtMapViewOfSection+0xc
00069bdc 7c831fd6 ntdll!LdrpMapViewOfDllSection+0x64 
00069ccc 7c833027 ntdll!LdrpMapDll+0x390 
[...]

The thread is blocked — it is waiting on en event. Events do not have an owner so we have to do a little more to find out, what it is waiting for. Unfortunately, !thread crops the trace, so let us first get the full one:

kd> .thread 826276f8  
kd> kn100
 # ChildEBP RetAddr  
00 f4422ab8 808202b6 nt!KiSwapContext+0x25
01 f4422ad0 8081fb6e nt!KiSwapThread+0x83
02 f4422b14 809bae63 nt!KeWaitForSingleObject+0x2e0
03 f4422bf4 809bc06d nt!DbgkpQueueMessage+0x178
04 f4422c18 8096ba9a nt!DbgkpSendApiMessage+0x45
05 f4422cc8 80909942 nt!DbgkMapViewOfSection+0xcf
06 f4422d34 8082350b nt!NtMapViewOfSection+0x269
07 f4422d34 7c8285ec nt!KiFastCallEntry+0xf8
08 00069b94 7c82728b ntdll!KiFastSystemCallRet
09 00069b98 7c831e05 ntdll!NtMapViewOfSection+0xc
0a 00069bdc 7c831fd6 ntdll!LdrpMapViewOfDllSection+0x64
0b 00069ccc 7c833027 ntdll!LdrpMapDll+0x390
0c 00069f30 7c8330f5 ntdll!LdrpLoadImportModule+0x17c
0d 00069f70 7c8330a4 ntdll!LdrpHandleOneNewFormatImportDescriptor+0x4d
0e 00069f8c 7c833248 ntdll!LdrpHandleNewFormatImportDescriptors+0x1d
0f 0006a014 7c833049 ntdll!LdrpWalkImportDescriptor+0x195
10 0006a264 7c8330f5 ntdll!LdrpLoadImportModule+0x1cb
11 0006a2a4 7c8330a4 ntdll!LdrpHandleOneNewFormatImportDescriptor+0x4d
12 0006a2c0 7c833248 ntdll!LdrpHandleNewFormatImportDescriptors+0x1d
13 0006a348 7c83427d ntdll!LdrpWalkImportDescriptor+0x195
14 0006a5e0 7c834065 ntdll!LdrpLoadDll+0x241
15 0006a85c 77e41bf3 ntdll!LdrLoadDll+0x198
16 0006a8c4 77e41dbd kernel32!LoadLibraryExW+0x1b2
17 0006a8d8 77e41df3 kernel32!LoadLibraryExA+0x1f
18 0006a8f8 46a7870c kernel32!LoadLibraryA+0xb5
19 0006a954 46a93b3e WININET!__delayLoadHelper2+0xfc
1a 0006a994 46a93950 WININET!_tailMerge_RASAPI32_dll+0xd
1b 0006a9a8 46a93a4e WININET!DoConnectoidsExist+0x2b
1c 0006a9d4 46a93abc WININET!GetRasConnections+0x34
1d 0006a9f0 46a8c559 WININET!IsDialUpConnection+0xa9
1e 0006aa0c 46a97a44 WININET!FixProxySettingsForCurrentConnection+0x31
1f 0006b5e4 46aa3774 WININET!InternetQueryOptionA+0xa47
20 0006b748 01d12dc6 WININET!InternetQueryOptionW+0x1fa
21 0006b98c 01d12583 symsrv!StoreWinInet::dumpproxyinfo+0x46
22 0006be04 01d1290a symsrv!StoreWinInet::connect+0x273
23 0006c040 01d05ae7 symsrv!StoreWinInet::find+0x3a
24 0006c134 01d06c47 symsrv!cascade+0x87
25 0006c684 01d06a57 symsrv!SymbolServerByIndexW+0x127
26 0006c8b4 0302e30e symsrv!SymbolServerW+0x77
27 0006ccf4 03018eed dbghelp!symsrvGetFile+0x12e
28 0006d9dc 03019f57 dbghelp!diaLocatePdb+0x33d
29 0006dc58 03041ade dbghelp!diaGetPdb+0x207
2a 0006de7c 0303ff15 dbghelp!GetDebugData+0x2be
2b 0006e324 03040516 dbghelp!modload+0x305
2c 0006e7a4 0304068e dbghelp!LoadModule+0x3f6
2d 0006e9e8 03044eaf dbghelp!GetModule+0x4e
2e 0006ea30 03044bda dbghelp!NTGetProcessModules+0x16f
2f 0006eae8 03032e80 dbghelp!GetProcessModules+0x4a
30 0006ed70 60f03f7a dbghelp!SymInitializeW+0x320
31 0006f17c 60f032b0 cfix!CfixpCaptureStackTrace+0x117 
32 0006f598 100419b5 cfix!CfixPeReportFailedAssertion+0xc5 
WARNING: Stack unwind information not available. 
Following frames may be wrong.
33 0006f7c0 10040f40 VsSample!__CfixFixturePeSimpleAdderTest+0x5ab1
34 0006f8b4 10040db6 VsSample!__CfixFixturePeSimpleAdderTest+0x503c
35 0006fa68 10040a8a VsSample!__CfixFixturePeSimpleAdderTest+0x4eb2
36 0006fb48 60f02b64 VsSample!__CfixFixturePeSimpleAdderTest+0x4b86
37 0006fb84 60f02be6 cfix!CfixsRunTestRoutine+0x33 
38 0006fb94 60f038e9 cfix!CfixsRunTestCaseMethod+0x27 
39 0006fbac 60f03a06 cfix!CfixsRunTestCase+0x25 
3a 0006fbcc 60f03ce5 cfix!CfixsRunTsexecActionMethod+0xfb 
3b 0006fbf0 0100e135 cfix!CfixsRunSequenceAction+0x122 
3c 0006fc2c 0100d5c2 cfix32!CfixrunpRunFixtures+0x90 
3d 0006fc40 0100d85c cfix32!CfixrunsMainWorker+0x3f 
3e 0006fe7c 010046b8 cfix32!CfixrunMain+0x1b9 
3f 0006fee0 0100485e cfix32!wmain+0x80 
40 0006ffc0 77e6f23b cfix32!_wmainCRTStartup+0x12b 
41 0006fff0 00000000 kernel32!BaseProcessStart+0x23

Whoa, what a trace! cfix!CfixpCaptureStackTrace tries to assemble a stack trace, for which it has to initialize dbghelp.dll first. dbghelp!SymInitializeW seeks help of symsrv.dll, which in turn tries to connect to the Microsoft symbol server. Before it can so, it obviously attempts to get its proxy settings straight, which in turn leads to some DLL (rasapi.dll, in case you wonder) being loaded. The loader then calls into the debugging subsystem (nt!Dbgk*). It may be assumed that the loader is notifying the debugger about the DLL having been loaded.

Turining our attention to Internet Explorer, we look at iexplore.exe’s threads:

kd> !process 0 0 iexplore.exe
PROCESS 824ec3b0  SessionId: 0  Cid: 07f0    Peb: 7ffdb000  ParentCid: 054c
    DirBase: 0307c000  ObjectTable: e15186b8  HandleCount: 225.
    Image: iexplore.exe
    
kd> .process /r /p 824ec3b0  
Implicit process is now 824ec3b0
.cache forcedecodeuser done
Loading User Symbols
...............................................

Now, iexplore has lots of threads, but skimming over them, one looked interesting:

kd> !process 824ec3b0  
PROCESS 824ec3b0  SessionId: 0  Cid: 07f0    
	Peb: 7ffdb000  ParentCid: 054c
    DirBase: 0307c000  ObjectTable: e15186b8  HandleCount: 225.
    Image: iexplore.exe
    VadRoot 824991c8 Vads 168 Clone 0 Private 643. 
    Modified 44. Locked 0.
    DeviceMap e148f3c0

    [...]
    
THREAD 82431980  Cid 07f0.00b8  Teb: 7ffd5000 Win32Thread: 00000000 
  WAIT: (Unknown) UserMode Non-Alertable
	82631eb0  Mutant - owning thread 826276f8
Not impersonating
DeviceMap                 e148f3c0
Owning Process            824ec3b0       Image:         iexplore.exe
Wait Start TickCount      21924          Ticks: 688 (0:00:00:06.889)
Context Switch Count      1             
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address ntdll!RtlpWorkerThread (0x7c839efb)
Start Address kernel32!BaseThreadStartThunk (0x77e617ec)
Stack Init f40a3000 Current f40a2c78 Base f40a3000 Limit f40a0000 Call 0
Priority 8 BasePriority 8 PriorityDecrement 0

ChildEBP RetAddr  
f40a2c90 808202b6 nt!KiSwapContext+0x25 
f40a2ca8 8081fb6e nt!KiSwapThread+0x83 
f40a2cec 8090e64e nt!KeWaitForSingleObject+0x2e0 
f40a2d50 8082350b nt!NtWaitForSingleObject+0x9a 
f40a2d50 7c8285ec nt!KiFastCallEntry+0xf8 
01e5fdd0 7c827d0b ntdll!KiFastSystemCallRet 
01e5fdd4 77e61d1e ntdll!NtWaitForSingleObject+0xc 
01e5fe44 77e61c8d kernel32!WaitForSingleObjectEx+0xac 
01e5fe58 46a8c54d kernel32!WaitForSingleObject+0x12 
01e5fe74 46a7eeca WININET!FixProxySettingsForCurrentConnection+0x25 
01e5fe8c 46a7ee3f WININET!CFsm_HttpSendRequest::RunSM+0x61 
01e5fea4 46a7efa3 WININET!CFsm::Run+0x39 
01e5fed4 77da5938 WININET!CFsm::RunWorkItem+0x79 
01e5feec 7c83a827 SHLWAPI!ExecuteWorkItem+0x1d 
01e5ff44 7c83aa0b ntdll!RtlpWorkerCallout+0x71 
01e5ff64 7c83aa82 ntdll!RtlpExecuteWorkerRequest+0x4f 
01e5ff78 7c839f60 ntdll!RtlpApcCallout+0x11 
01e5ffb8 77e64829 ntdll!RtlpWorkerThread+0x61 
01e5ffec 00000000 kernel32!BaseThreadStart+0x34 

Now we are getting somewhere. We have seen FixProxySettingsForCurrentConnection in cfix’s trace already — but in this case, it is waiting on something. Let’s see…

kd> !object 82631eb0  
Object: 82631eb0  Type: (827a5550) Mutant
    ObjectHeader: 82631e98 (old version)
    HandleCount: 3  PointerCount: 6
    Directory Object: e1496420  Name: WininetProxyRegistryMutex

And 826276f8, that’s the cfix32 thread we have already assessed. Obviously, iexplore waits for cfix to release the WininetProxyRegistryMutex, and cfix waits on someone else.

Turning over to VC, we can find a stack that also contains a call to FixProxySettingsForCurrentConnection on its stack. Again, blocking on WininetProxyRegistryMutex.

kd> k100
ChildEBP RetAddr  
f4492c90 808202b6 nt!KiSwapContext+0x25
f4492ca8 8081fb6e nt!KiSwapThread+0x83
f4492cec 8090e64e nt!KeWaitForSingleObject+0x2e0
f4492d50 8082350b nt!NtWaitForSingleObject+0x9a
f4492d50 7c8285ec nt!KiFastCallEntry+0xf8
065b95c8 7c827d0b ntdll!KiFastSystemCallRet
065b95cc 77e61d1e ntdll!NtWaitForSingleObject+0xc
065b963c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
065b9650 46a8c54d kernel32!WaitForSingleObject+0x12
065b966c 46a7eeca WININET!FixProxySettingsForCurrentConnection+0x25
065b9684 46a7ee3f WININET!CFsm_HttpSendRequest::RunSM+0x61
065b969c 46a7fefa WININET!CFsm::Run+0x39
065b96b4 46ab0a67 WININET!DoFsm+0x25
065b96dc 46aa1092 WININET!HttpWrapSendRequest+0x148
065b9714 06b231da WININET!HttpSendRequestW+0x5e
065b973c 06b22ea8 SYMSRV!StoreWinInet::request+0x2a
065b9770 06b226cc SYMSRV!StoreWinInet::fileinfo+0x18
065b9780 06b22741 SYMSRV!StoreWinInet::get+0x7c
065b9fc4 06b229a3 SYMSRV!StoreWinInet::open+0x41
065ba204 06b15ae7 SYMSRV!StoreWinInet::find+0xd3
065ba2f8 06b16c47 SYMSRV!cascade+0x87
065ba848 06b16a57 SYMSRV!SymbolServerByIndexW+0x127
065baa78 51412896 SYMSRV!SymbolServerW+0x77
065bb8cc 51413383 mspdb80!LOCATOR::SYMSRV::SymbolServer+0x190
065bbf10 514136f8 mspdb80!LOCATOR::FLocatePdbSymsrv+0x75
065bbf38 514139ce mspdb80!LOCATOR::FLocatePdbPathHelper+0x179
065bc96c 51413cbe mspdb80!LOCATOR::FLocatePdbPath+0x105
065bccb4 51414371 mspdb80!LOCATOR::FLocatePdb+0x1ad
065bd9a8 458cc1e8 mspdb80!PDBCommon::OpenValidate5+0xab
065bd9ec 45959d4c msenc90!enc::EncImageEdit::
                  `scalar deleting destructor'+0x4d
065bda34 45958e62 NatDbgDE!OLPDBOpen+0x93
065be6c0 45958f0a NatDbgDE!OLStart+0x107
065be6fc 45958fae NatDbgDE!LoadOmfForReal+0x23
065be714 45959019 NatDbgDE!LoadSymbols+0x43
065be72c 459590d9 NatDbgDE!OLLoadOmf+0x55
065be75c 45959154 NatDbgDE!SHLoadDll+0xd5
065be7ac 45959247 NatDbgDE!CSymbolHandlerX::SHLoadDll+0x5a
065be844 4595937c NatDbgDE!CModule::Load+0x1a1
065be8ac 4594d002 NatDbgDE!CNativeProcess::NotifyModLoad+0xc8
065be9ec 4594cf6d NatDbgDE!EngineCallback+0xb3
065bea18 45958d3f NatDbgDE!EMCallBackDB+0x4c
065bf050 4594d0dc NatDbgDE!LoadFixups+0x218
065bf0ac 4594d289 NatDbgDE!DebugPacket+0x213
065bfdb4 4594cf39 NatDbgDE!EMFunc+0x40f
065bfddc 4594d73d NatDbgDE!TLCallBack+0x1e
065bfdf4 4594d711 NatDbgDE!TLClientLib::Local_TLFunc+0xc8
065bfe3c 4594d85c NatDbgDE!DMSendPacket+0x121
065bfee8 45959b1d NatDbgDE!NotifyEM+0x3ae
065bff0c 4594d663 NatDbgDE!ProcessLoadDLLEvent+0x47
065bff44 4594d686 NatDbgDE!ProcessDebugEvent+0x30d
065bffb8 77e64829 NatDbgDE!DmPollLoop+0x3c
065bffec 00000000 kernel32!BaseThreadStart+0x34

But — looking closely, it becomes obvious that this thread must be the one handling debug events, and in fact, the call to ProcessLoadDLLEvent is a strong indication for that this thread is currently handling a DLL load event. And now we have closed the loop — this thread must be handling the DLL load event for rasapi.dll, the DLL which cfix was about to load. And to do this, VC attempts to acquire the WininetProxyRegistryMutex, which is owned by the original cfix thread. Deadlock.

What is interesting about this situation is that neither party — cfix, iexplore or VCExpress, and also none of the modules clearly is the culprit and behaving wrong. It is more like a combination of special circumstances that bring up the deadlock as discussed.

It is also notable that I am not using any particular proxy settings on this machine and automatic proxy configuration has been turned off.

So far, I have not experienced the same problem with VS 2003 and VS 2005 — I thus assume that only VS 2008 is affected by this.

Although I am pretty sure that cfix is not really at fault here, I have to adapt it to avoid this deadlock in the future. Until an updated version is available, you can use this workaround.

Until a new version of cfix is available, you can work around this problem as follows: Go to the cfix installation directory and rename or delete symsrv.dll. This will stop cfix from contacting the symbol server. While cfix now will not be able to capture proper stack traces any more, this will prevent the mentioned deadlock situation to arise.

The problem only applies to cfix 1.1 and VS 2008. The problem does not occur when debugging with VS 2003, VS 2005, or cdb/ntsd/WinDBG.

On a uniprocessor system, there is only one Idle thread and this idle thread will have the thread ID 0 (in process 0). On a multiprocessor system, however, Windows creates one Idle thread per CPU. That makes sense — however, what may be surprising at first is that although all Idle threads have their own KTHREAD structure, all share the same thread ID 0 (CID 0:0). That is, each multiprocessor system will have multiple threads with ID 0, which in turn means that CID 0:0 does not uniquely identify a single thread.

This is easily verified using the kernel debugger on a multiprocessor system (a quad core in this case):

0: kd> !running

System Processors f (affinity mask)
  Idle Processors f
All processors idle.

0: kd> !pcr 0
KPCR for Processor 0 at ffdff000:
[...]

	      CurrentThread: 8089d8c0
	         NextThread: 00000000
	         IdleThread: 8089d8c0


0: kd> !thread 8089d8c0
THREAD 8089d8c0  Cid 0000.0000  Teb: 00000000 Win32Thread: 00000000 RUNNING on processor 0
[...]

0: kd> !pcr 1
KPCR for Processor 1 at f7727000:
[...]

	      CurrentThread: f772a090
	         NextThread: 00000000
	         IdleThread: f772a090
	         
0: kd> !thread f772a090
THREAD f772a090  Cid 0000.0000  Teb: 00000000 Win32Thread: 00000000 RUNNING on processor 1
[...]

...and so on.

Now, the idle threads are usually not of major intest. Still, it is possible that these threads can become relevant — such as when doing certain kinds of analysis such as tracing interrupts. If such analysis groups events by the CID of the thread they were captured on (rather than the VA of the KTHREAD structure), the results for CID 0:0 will be wrong.

Not a big thing. Still, it took me a while to figure out that this was indeed the reason for some of my traces containing timestamps in strange order: The traces had been created on a quadcode machine and I did make the mistake to correlate the events by their CID during later analsis. As a result, the traces of the four idle threads were all mixed up…

When programming in assembly, it is equally easy to use memory below and above the stack pointer. Reading from or writing to addresses beyond the top of the stack is unusual and under normal circumstances, there is little reason to do so. There are, however, situations — rare situations — where it may tempting to temporarily use memory beyond the top of the stack.

That said, the question is whether it is really just a convention and good style not to grab beyond the stack of the stack or whether there are actually reasons why doing so could lead to problems.

When trying to answer this question, one first has to make a distinction between user mode and kernel mode. In user mode Windows, I am unable to come up with a single reason of why usage of memory beyond the top of the stack could lead to problems. So in this case, it is probably merely bad style.

However, things are different in kernel mode.

In one particular routine I recently wrote, I encountered a situation where temporarily violating the rule of not reaching beyond the top of the stack came in handy. The routine worked fine for quite a while. In certain situations, however, it suddenly started to fail due to memory corruption. Interestingly enough, the routine did not fail always, but still rather frequently.

Having identified the specific routine as being the cuplrit, I started single stepping the code. Everything was fine until I reached the point where the memory above the stack pointer was used. The window span only a single instruction. Yet, as soon as I had stepped over the two instructions, the system crashed. I tried it multiple times, and it was prefectly reproduceable when being single-stepped.

So I took a look at the stack contents after every single step I took. To my surprise, as soon as I reached the critical window, the contents of the memory location just beyond the current stack pointer suddenly became messed. Very weird.

After having been scratching my head for a while, that suddenly started to made sense: I was not the only one using the stack — in between the two instructions, an interrupt must have occured and been dispatched. As my thread happened to be the one currently running, it was my stack that has been used for dispatching it. This also explains why it did not happened always unless I was single-stepping the respective code.

When an interrupt occurs and no privilege-level change has to be performed, the CPU will push the EFLAGS, CS and EIP registers on the stack. That is, the stack of whatever kernel thread happens to be the one currently running on this CPU is reused and the memory locations beyond the stack pointer will be overwritten by these three values. So what I initially interpreted as garbage, actually were the contents of EFLAGS, CS and EIP.

On Windows NT, unlike some other operating systems (FreeBSD, IIRC), handling the interrupt, which involves runing the interrupt service routine (ISR) occurs on the same stack as well. The following stack trace, taken elsewhere, shows an ISR being executed on the stack of the interrupted thread:

f6bdab4c f99bf153 i8042prt!I8xQueueCurrentMouseInput+0x67
f6bdab78 80884289 i8042prt!I8042MouseInterruptService+0xa58
f6bdab78 f6dd501a nt!KiInterruptDispatch+0x49
f6bdac44 f6dd435f driver!Quux+0x11a 
f6bdac58 f6dd61db driver!Foobar+0x6f 
...

Morale of the story: Using memory beyond the current stack pointer is not only bad practice, it is actually illegal when done in kernel mode.

Things are not quite as easy when analyzing symbols for a process (or kernel) that is not running any more or executes on a different machine. In this case, manual intervention is necessary — without your help, dbghelp will not be able to decide which PDB to load.

An example for a case where this applies is writing a binary tracefile. To keep the trace file small and to make writing entries as efficient as possible, it makes sense to write raw instruction pointer and function pointer addresses to the file. This way, resolving symbols and finding the names of the affected routines is delayed until the file is actually read.

However, when the trace file is read, the traced process (or kernel) may not be alive any more, so dbghelp’s automatisms cannot be used. Moreover, the version of the OS and the individual modules involved may not be the same on the machine where the trace is obtained and the machine where the trace file is read. It is thus crucial that the trace file contains all neccessary information required for the reading application to find and load the matching symbols.

Capturing the debug information

That sounds easy enough. But once again, the documentation on this topic is flaky enough to make this a non-trivial task. This thread on the OSR WinDBG list revealed some important insights, yet the summary is not entirely correct. So I will try to explain it here. Note that the discussion equally applies to user and kernel mode.

Two basic pieces of information need to be captured and stored in the trace file: The load address and the debug data of each module involved. The load address is required in order to later be able to convert from virtual addresses (VAs) to relative virtual addresses (RVAs). The debug data is neccessary for finding the PDB exactly matching the respective module.

The debug data is stored in the PE image and can be obtained via the debug directory. The following diagram illustrates the basic layout of the affected parts of a PE image:

Via the _IMAGE_DATA_DIRECTORY structure, which is at index IMAGE_DIRECTORY_ENTRY_DEBUG of the DataDirectory, we can obtain the RVA of the debug directory — as always, RVAs are relative to the load address. The debug directory is a sequence of IMAGE_DEBUG_DIRECTORY structures. Although the CodeView entry should be the one of primary interest, expect more than one entry in this sequence. The entry count can be calculated by dividing _IMAGE_DATA_DIRECTORY::Size by sizeof( IMAGE_DEBUG_DIRECTORY ).

The IMAGE_DEBUG_DIRECTORY structures do not contain all necessary information by themselves — rather, they refer to a blob of debug data located elsewhere in the image. IMAGE_DEBUG_DIRECTORY::AddressOfRawData contains the RVA (again, relative to the load address) to this blob. Note that the blobs are not necessarily laid out in the same order as the corresponding IMAGE_DEBUG_DIRECTORY structures are. (The role of the PointerToRawData member in this context remains unclear.)

Summing up, we need to write to the trace file:

• The module load address
• all IMAGE_DEBUG_DIRECTORY entries
• all debug data blobs referred to

In order to maintain the relation between the IMAGE_DEBUG_DIRECTORY entries and their corresponding blobs, the AddressOfRawData and PointerToRawData members have to be updated before being written to the file. In order to make loading symbols a bit easier, it makes sense to save the entire sequence of IMAGE_DEBUG_DIRECTORY structs (with updated RVAs) contigously to the file, immediately followed by the debug data blobs.

Loading symbols

Having all neccessary information in the trace file, we can now load the symbols. The key is to use SymLoadModuleEx rather than SymLoadModule64 and passing it a MODLOAD_DATA struct:

• MODLOAD_DATA::data receives a pointer to the sequence of IMAGE_DEBUG_DIRECTORY structs. For each struct, the AddressOfRawData member should be set to 0 and PointerToRawData should receive the RVA of the corresponding debug data blob relative to the beginning of the respective IMAGE_DEBUG_DIRECTORY struct.

  As indicated before, it therefore makes sense to write the IMAGE_DEBUG_DIRECTORY structs and the blobs consecutievly to the file and — at the time of writing — adjust the AddressOfRawData and PointerToRawData members. This way, you can just assign the entire memory chunk to MODLOAD_DATA::data.

• MODLOAD_DATA::size receives the size of the entire memory chunk, including both the sequence of IMAGE_DEBUG_DIRECTORY structs and all blobs.

There you go, dbghelp should now be able to identify the proper PDB and load the symbols.

Although more than 10 years old, the best article on this topic still seems to be Matt Pirtrek’s A Crash Course on the Depths of Win32™ Structured Exception Handling, which I assume you have read. However, note that this article as well as this post refer to i386 only, albeit both to user and kernel mode.

Exception Registration Record Validation

On the i386, SEH uses a linked list of exception registration records. The first record is pointed to by the first member of the TIB. In user mode, the TIB is part of the TEB, in kernel mode it is part of the KPCR — in any case, it is at fs:[0]. Each record, besides containing a pointer to the next lower record, stores a pointer to an exception handler routine.

Installing an exception registration record is thus straightforward and merely requires adjusting the TIB pointer and having the new record point to next lower record. So I set up my custom exception registration record, registered it properly, verified that all pointers are correct and tried using it. However, I was unpleasently surprised that exeption handling totally failed as soon as my exception registration got involved. !exchain reported an “Invalid exception stack”, although checking the pointers manually again seemed to show that the chain of exception registration records was fine and my record seemed ok.

Digging a little deeper I found the reason for that — and in fact I cannot remember ever having heard or read about this requirement before: Windows requires all EXCEPTION_REGISTRATION_RECORDs to be located on the stack. Both RtlDispatchException and RtlUnwind check the location of each EXCEPTION_REGISTRATION_RECORD against the stack limits and abort exception handling as soon as a record is found to be not stack-located. Aborting exception handling in this case means that RaiseException/ExRaiseStatus will just return and execution will be resumed at the caller site as if nothing happened.

This requirement is fair enough, actually, but in my case it totally wrecked my design. I did not have the 8 spare bytes to store this record and thus therefore put the record on some dedicated place on the heap. Urgh. Anyway…

As an interesting side note, Windows Server 2003 performs this stack check against both limits — minimum and maximum address of the stack. Vista, however, only checks against the maximum address (i.e. bottom of the stack) and does not care whether the minimum address (i.e. top of stack) has been exceeded.

Moreover, there is another restriction on exception records that only applies to user mode: The handler routine pointed to by the exception record is verified to not point into the stack. This is obviously another security measure to avoid SEH records to point into some overflown buffers.

SafeSEH

It is worth pointing out that all these checks are unrelated to SafeSEH and are performed regardless of whether your module is SafeSEH compatible or not. Not before these checks have all passed, the exception handler has to undergo the SafeSEH validation: The image base is calculated, the table listing the trusted SEH handlers is looked up and it is checked whether the handler routine pointed to by the current exception record is located in this table.

SafeSEH Handler Registration

Using SafeSEH is a good thing and I link all my modules with /SafeSEH. So when you use low level SEH, i.e. without using the __try/__except compiler support, the obvious question is how to get your SEH handler to be recognized as a trusted handler and be included in the SafeSEH table. After all, the compiler will not be able to recognize that the routine you have just written will in fact be used as an exception handler. The C compiler does not seem to offer support for that — luckily however, ml does by providing the .SAFESEH directive.

If you like writing your exception handler in assmbler, this is all you need. If, however, you prefer C, this is somewhat unsatisfying. The documentation of .SAFESEH states that it can be used with an extrn proc, but that does not seem to work. My solution was thus to write the actual routine in C and write a little thunk in assembler, which I was then able to register using the .SAFESEH directive:

.586               
.model flat, stdcall
option casemap :none

extrn RealExceptionHandlerWrittenInC@16

...

ExceptionHandlerThunk proto
.SAFESEH ExceptionHandlerThunk

...

.code

ExceptionHandlerThunk proc
	jmp RealExceptionHandlerWrittenInC@16
ExceptionHandlerThunk endp

Stupid things you should not do

Finally, there is another little quirk that bit me: Do not use EXCEPTION_CONTINUE_SEARCH where ExceptionContinueSearch would have been appropriate. The EXCEPTION_* constants are for use by exception filters as used for __except statements, whereas the Exception* values have to be used for low level exception handlers. Should be obvious, right? :)

Having chosen the wrong group of constants, I returned EXCEPTION_CONTINUE_SEARCH from my exception handler to indicate that the handler is unable to handle certain exceptions. However, as it turns out, EXCEPTION_CONTINUE_SEARCH has the value 0 and is thus interpreted as ExceptionContinueExecution. Now, returning ExceptionContinueExecution when being requested to handle an exception raised by ExRaiseStatus is obviously a bad idea and in this case led to a STATUS_NONCONTINUABLE_EXCEPTION. After a few of those had stacked up (in kernel mode), VirtualPC crashed with an unrecoverable CPU error. Nice :)

With little success:

#
# An unexpected error has been detected by Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00000000772b219b, pid=4188, tid=2272
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (10.0-b22 mixed mode windows-amd64)
# Problematic frame:
# C  [ntdll.dll+0x5219b]
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

The project uses a native library via JNI, so of course I immediately suspected this to be the problem. So I placed a Java breakpoint on the respective System.loadLibrary call with the intent of attaching WinDBG as soon as this breakpoint is hit. In WinDBG, I could then break on the exception and see what the problem is.

But to my surprise, the Java breakpoint was not hit — the VM crashed immediately and I received the same output about an unexpected error having occured. That seemed strange to me — maybe it was not the fault of the JNI library after all? So I created a simple Hello World application and ran it — that worked. Then I created this innocent JUnit test:

public class JTest
{
  @org.junit.Test
  public void testname() throws Exception
  {    
  }
}

This one failed again, yielding the same error message as above. Well, at least that gave me evidence that not my JNI library but the JVM was the culprit of the crash — but still, the situation seemed weird. Running the test again under WinDBG, I could see that the AV occured during a heap free operation (As a side node, it is annoying that Sun does not supply symbols for its binaries):

00000000`0404ca88 00000000`7727e7e2 ntdll!RtlCaptureContext+0x8c
00000000`0404ca98 00000000`7727e72b ntdll!RtlpWalkFrameChain+0x52
00000000`0404d018 00000000`773352f2 ntdll!RtlCaptureStackBackTrace+0x4b
00000000`0404d048 00000000`772e1d35 ntdll!RtlpStackTraceDatabaseLogPrefix+0x42
00000000`0404d178 00000000`7715d9fa ntdll! ?? ::FNODOBFM::`string'+0xa93f
00000000`0404d1f8 000007fe`fef0175c kernel32!HeapFree+0xa
*** WARNING: Unable to verify checksum for C:\Program Files\Java\jdk1.6.0_06\jre\bin\server\jvm.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for C:\Program Files\Java\jdk1.6.0_06\jre\bin\server\jvm.dll - 
00000000`0404d228 00000000`08101c09 msvcrt!free+0x1c
00000000`0404d258 00000000`081026cc jvm!JVM_EnqueueOperation+0x8c139
00000000`0404d288 00000000`040b4937 jvm!JVM_EnqueueOperation+0x8cbfc
00000000`0404d318 00000000`0404d338 0x40b4937

Well, could be a heap corruption — but it is interesting that crash did not occur during block coalescence or similar operations but during stack trace capturing. As a matter of fact, I always run my machine with user mode stack trace database creation enabled for debugging purposes. So I disabled the stack trace database in gflags, rebooted the machine and — voilà, the crash disappeared!

Wow. I think this is worth being filed as a bug.

void foo()
{
}

void bar()
{
}

int main()
{
  // Some random code...
  int a = 1, b = 2;
  
  // Call a child function.
  foo();
  
  // More useless code...
  a+=b;
  if ( a == b) a = b;
  
  // Call another child function.
  bar();  
  
  return 0;
}

wt will produce the following output:

0:000> wt
Tracing test!main to return address 00401291
    6     0 [  0] test!main
    1     0 [  1]   test!ILT+5(_foo)
    4     0 [  1]   test!foo
   13     5 [  0] test!main
    1     0 [  1]   test!ILT+0(_bar)
    4     0 [  1]   test!bar
   17    10 [  0] test!main

27 instructions were executed in 26 events 
                                  (0 from other threads)

Function Name         Invocations MinInst MaxInst AvgInst
test!ILT+0(_bar)                1       1       1       1
test!ILT+5(_foo)                1       1       1       1
test!bar                        1       4       4       4
test!foo                        1       4       4       4
test!main                       1      17      17      17

0 system calls were executed

Although helpful, tracing a larger function calling a multitude of other functions slows down the debuggee significantly. An interesting question is thus how wt is implemented. Three possible implementation strategies come to mind:

1. Use single-stepping. After each instruction executed, a debug trap is raised and the debugger is delivered a single-step debugging event. Though all non-branching instructions are probably irrelevant to wt, by intercepting each call and ret instruction, the debugger is able to trace function entry and exit.
2. Explicitly set breakpoints. The debugger disassembles the function to be traced and places an ordinary breakpoint on each call instruction as well on as the return address of the function. Whenever one of the call-breakpoints fires, the debugger instruments the target function in the same way (i.e. place breakpoints on each call instruction as well as the return address) and continues execution (without single-stepping). By intercepting all function calls and returns, the debugger is able to deduce the call tree. This approach would be similar to UMSS.
3. Use Last Branch Recording. This is a rather new additon to the IA-32 instruction set that allows setting breakpoints on taken branches, interrupts, and exceptions, and to single-step from one branch to the next.

In order to find out, we have to debug the debugger to observe how it debugs the target. We thus start WinDBG, choose our test application as target and let it break on main. We then start another WinDBG instance and attach it to the first WinDBG instance. In order to find out which debugging events are consumed by the first instance, we use the second debugger to trace function calls made by the first debugger.

All usermode debuggers eventually end up calling ntdll!NtWaitForDebugEvent in a loop — so to find out which debugging events are consumed, all we need to do is trace all calls to this function. While being an undocumented native function, there is an excellent summary on the inner workings of user mode debugging which also covers ntdll!NtWaitForDebugEvent. Given this information, all we need to do to check whether strategy #1 or strategy #2 has been implemented (I assume #3 may safely be neglected) is to put together a little breakpoint command like the following (line breaks added for clarity):

bp ntdll!NtWaitForDebugEvent "
   r @$t1=poi(esp+10); 
   g @$ra; 
   .if (poi(@$t1)==8) {.echo \"SingleStep\n\" } 
   .else {.printf \"Excp %p\\n\", poi(@$t1+c)};
   g "

When entering ntdll!NtWaitForDebugEvent, we store the address of the fourth parameter (which receives a PDBGUI_WAIT_STATE_CHANGE structure) in $t1 and step out of the function. Then we reach into the structure whose address is stored in $t1 and check if the first field marks the event of being of type DbgSingleStepStateChange (0×8) and output an appropriate message. If we receive about 30 single-step events, strategy #1 has probably been chosen. For #2 we would expect to receive 5 breakpoint events.

Back to the first debugger, we now opt to trace the main function by running wt. This yields the output shown above. Switching to the second debugger again, we now see the following output:

SingleStep
SingleStep
SingleStep

[...about 20 more...]

SingleStep
SingleStep
SingleStep
SingleStep
SingleStep

Quite obviously, wt implements strategy #1 — it does single stepping. Although this does not really come as a surprise, it is still unfortunate as it is most likely the slowest approach of tracing calls. And as anybody who has ever used wt can probably confirm, wt is really slow.

As an interesting side note, as of Linux kernel 2.6.25, ptrace on x86 has been enhanced to facilitate Last Branch Recording on CPUs that support it.

Although you can walk the stack of any thread using StackWalk64, this post only deals with walking the current thread’s stack. In order to walk the stack of the current thread, you first have to obtain the CONTEXT of the current thread. The naive way to obtain this context would be to call GetThreadContext( GetCurrentThread() ) — however, as the documentation for GetThreadContext clearly states, the result of this function is undefined if used on the current thread.

Obtaining a CONTEXT

64 Bit windows provides the function RtlCaptureContext for this purpose, so obtaining the context is trivial. 32 Bit Windows, however, does not implement this function, yet it is not hard to come up with a surregate.

One way to obtain the context is to set up a SEH frame, deliberately raise an exception and obtain the context in the exception filter using GetExceptionInformation. Although that works, it is probably the slowest way to do it. An easier and quicker solution is to use inline assembly (we only have to support 32 bit as we have RtlCaptureContext for x64/IA64, so using assembly is ok). Using appropriate instructions, we can manually populate the x86 CONTEXT structure. However, as eip cannot be accessed using a mov instruction, we have to use a little trick to obtain the value of the instruction pointer — use an (otherwise useless) label:

CONTEXT Context;
ZeroMemory( &Context, sizeof( CONTEXT ) );
Context.ContextFlags = CONTEXT_CONTROL;

__asm
{
Label:
  mov [Context.Ebp], ebp;
  mov [Context.Esp], esp;
  mov eax, [Label];
  mov [Context.Eip], eax;
}

Preparing to call StackWalk64

Having obtained the context, we have to initialize the STACKFRAME64 structure for the first call. This is trivial, but platform-specific.

In order to enable StackWalk64 to use symbol information, dbghelp has to be initialized by calling SymInitialize, passing the current process handle as parameter. Unless you plan to manually call SymLoadModule64 for all affected modules, it is crucial to pass TRUE for the fInvadeProcess argument. If you pass FALSE or forget to call SymInitialize altogether (After all, the documentation of StackWalk64 does not state that calling SymInitialize is required), you will notice that StackWalk64 works on x86 builds, but fails (with last error set to 0x1E7) on x64. Also, as dbghelp is not threadsafe, it is usually a good idea to protect any calls to dbghelp by a critical section

Calling StackWalk64

After these provisions have been made, calling StackWalk64 is a snap. The only unfortnate thing is that you cannot really distinguish between StackWalk64 returning FALSE because of a failure or because of the bottom of the stack having been reached as StackWalk64 does not reliably set the last Win32 error.

The following code puts it all together. Note that only a the minimal amount of stack trace information is gathered, i.e. only the return addresses are saved in the array.

typedef struct _STACKTRACE
{
  //
  // Number of frames in Frames array.
  //
  UINT FrameCount;

  //
  // PC-Addresses of frames. Index 0 contains the topmost frame.
  //
  ULONGLONG Frames[ ANYSIZE_ARRAY ];
} STACKTRACE, *PSTACKTRACE;

/*++
  Routine Description:
    Capture the stack trace based on the given CONTEXT.

  Parameters:
    Context    Thread context. If NULL, the current context is
               used.
    Event      Event structure to populate.
    MaxFrames  Max number of frames the structure can hold.
--*/
#ifdef _M_IX86
  //
  // Disable global optimization and ignore /GS waning caused by 
  // inline assembly.
  //
  #pragma optimize( "g", off )
  #pragma warning( push )
  #pragma warning( disable : 4748 )
#endif
HRESULT CaptureStackTrace(
  __in_opt CONST PCONTEXT InitialContext,
  __in PSTACKTRACE StackTrace,
  __in UINT MaxFrames 
  )
{
  DWORD MachineType;
  CONTEXT Context;
  HRESULT Hr;
  STACKFRAME64 StackFrame;

  ASSERT( StackTrace );
  ASSERT( MaxFrames > 0 );

  if ( InitialContext == NULL )
  {
    //
    // Use current context.
    //
    // N.B. GetThreadContext cannot be used on the current thread.
    // Capture own context - on i386, there is no API for that.
    //
#ifdef _M_IX86
    ZeroMemory( &Context, sizeof( CONTEXT ) );

    Context.ContextFlags = CONTEXT_CONTROL;
    
    //
    // Those three registers are enough.
    //
    __asm
    {
    Label:
      mov [Context.Ebp], ebp;
      mov [Context.Esp], esp;
      mov eax, [Label];
      mov [Context.Eip], eax;
    }
#else
    RtlCaptureContext( &Context );
#endif  
  }
  else
  {
    CopyMemory( &Context, InitialContext, sizeof( CONTEXT ) ); 
  }

  //
  // Set up stack frame.
  //
  ZeroMemory( &StackFrame, sizeof( STACKFRAME64 ) );
#ifdef _M_IX86
  MachineType                 = IMAGE_FILE_MACHINE_I386;
  StackFrame.AddrPC.Offset    = Context.Eip;
  StackFrame.AddrPC.Mode      = AddrModeFlat;
  StackFrame.AddrFrame.Offset = Context.Ebp;
  StackFrame.AddrFrame.Mode   = AddrModeFlat;
  StackFrame.AddrStack.Offset = Context.Esp;
  StackFrame.AddrStack.Mode   = AddrModeFlat;
#elif _M_X64
  MachineType                 = IMAGE_FILE_MACHINE_AMD64;
  StackFrame.AddrPC.Offset    = Context.Rip;
  StackFrame.AddrPC.Mode      = AddrModeFlat;
  StackFrame.AddrFrame.Offset = Context.Rsp;
  StackFrame.AddrFrame.Mode   = AddrModeFlat;
  StackFrame.AddrStack.Offset = Context.Rsp;
  StackFrame.AddrStack.Mode   = AddrModeFlat;
#elif _M_IA64
  MachineType                 = IMAGE_FILE_MACHINE_IA64;
  StackFrame.AddrPC.Offset    = Context.StIIP;
  StackFrame.AddrPC.Mode      = AddrModeFlat;
  StackFrame.AddrFrame.Offset = Context.IntSp;
  StackFrame.AddrFrame.Mode   = AddrModeFlat;
  StackFrame.AddrBStore.Offset= Context.RsBSP;
  StackFrame.AddrBStore.Mode  = AddrModeFlat;
  StackFrame.AddrStack.Offset = Context.IntSp;
  StackFrame.AddrStack.Mode   = AddrModeFlat;
#else
  #error "Unsupported platform"
#endif

  StackTrace->FrameCount = 0;

  //
  // Dbghelp is is singlethreaded, so acquire a lock.
  //
  // Note that the code assumes that 
  // SymInitialize( GetCurrentProcess(), NULL, TRUE ) has 
  // already been called.
  //
  EnterCriticalSection( &DbgHelpLock );

  while ( StackTrace->FrameCount < MaxFrames )
  {
    if ( ! StackWalk64(
      MachineType,
      GetCurrentProcess(),
      GetCurrentThread(),
      &StackFrame,
      MachineType == IMAGE_FILE_MACHINE_I386 
        ? NULL
        : &Context,
      NULL,
      SymFunctionTableAccess64,
      SymGetModuleBase64,
      NULL ) )
    {
      //
      // Maybe it failed, maybe we have finished walking the stack.
      //
      break;
    }

    if ( StackFrame.AddrPC.Offset != 0 )
    {
      //
      // Valid frame.
      //
      StackTrace->Frames[ StackTrace->FrameCount++ ] = 
        StackFrame.AddrPC.Offset;
    }
    else
    {
      //
      // Base reached.
      //
      break;
    }
  }
  
  LeaveCriticalSection( &DbgHelpLock );

  return S_OK;
}
#ifdef _M_IX86
  #pragma warning( pop )
  #pragma optimize( "g", on )
#endif

As it turns out, things change when Application Verifier heap checks are enabled for this process. To check heap operations, Application Verifier intercepts certain operations. Presumably to protect its own internal data structures, at some points, Avrf enters a critical section, as shown in this stack trace:

0:009> k
ChildEBP RetAddr  
0840f540 77d9cfad ntdll!ZwWaitForSingleObject+0x15
0840f5a4 77d9cf78 ntdll!RtlpWaitOnCriticalSection+0x154
0840f5cc 6db25f92 ntdll!RtlEnterCriticalSection+0x152
0840f5f0 6db29d99 vfbasics!AVrfpFreeMemLockChecks+0x42 
0840f60c 6db2bae8 vfbasics!AVrfpFreeMemNotify+0x39 
0840f668 77731d27 vfbasics!AVrfpRtlFreeHeap+0x148 
0840f67c 6d242757 kernel32!HeapFree+0x14
...

Entering a critical section is no big thing — nontheless, it effectvely means that heap operations, whether they are conducted with HEAP_NO_SERIALIZE flag set or not, must be considered to be at least partially serialized by Application Verifier.

Although this somewhat contradicts the idea of the HEAP_NO_SERIALIZE flag, this should not be a problem in most cases — especially when HEAP_NO_SERIALIZE has been passed for performance reasons only. However, if HEAP_NO_SERIALIZE has been used for correctness reasons, this might indeed bite you. In my specific case, I used HEAP_NO_SERIALIZE and did the synchronization myself in order to break a deadlock situation. And while the application ran fine (and deadlock-free) without Avrf enabled, it began deadlocking randomly as soon as I enabled Avrf heap checks — for the very reason just described. This is especially annoying as I am now left with two less-than-optimal choices: Leave Avrf enabled for my test suite and experience random deadlocks or forgo Avrf’s helpful heap checks in order not to deadlock. Grrr, I need more choices!