Johannes Passing's Blog

Last time I showed that as a result of its implementation strategy, Detours is unable to properly instrument certain kinds of functions. Unlike last time, where this merely led to unexpected debugging output, we will see how this can easily lead to a crash.

Consider this code (error checking omitted):

#include <stdio.h>
#include <tchar.h>
#include <windows.h>
#include <detours.h>

static PDETOUR_TRAMPOLINE Trampoline;

__declspec(noinline)
static void DoSomething()
{
  wprintf( L"DoSomething\n" );
  Sleep( 100 );
}

//
// This is the function we want to instrument.
//
__declspec(noinline)
static void OriginalFunction( PCWSTR Arg1, LONG Arg2 )
{
  _asm xor eax, eax;
  for (;;)
  {
    DoSomething();
  }
}

//
// The generic hook we will install using Detours.
//
__declspec(naked)
static void GenericHook()
{
  //
  // Do something -- not important what...
  //
  __asm pushad;
  wprintf( L"GenericHook\n" );
  __asm popad;

  //
  // Done with instrumentation code, continue with
  // the real function. The trampoline (created
  // by Detours) contains the replaced instructions
  // and a jump to the body of the real function.
  //
  __asm jmp [Trampoline];
}

int wmain()
{
  ... Install hook -- same as last time ...

  //
  // Call the (now instrumented) function.
  //
  OriginalFunction( L"Hello", 42 );

  return 0;
}

The only difference to the code last time is the

_asm xor eax, eax;

inserted at the beginning of OriginalFunction. Note that any other instruction occupying less than 5 bytes should work just as well.

Try running this code and you will notice that the process immediately crashes with an Illegal Instruction exception. If you replace

_asm xor eax, eax;

by

_asm nop;

you get a Privileged instruction exception, which is no better. In any case, there is something going completely wrong here, so let us have a look at the assembly code again.

Unlike last time, where OriginalFunction consisted of a single basic block, we now have two blocks:

The important thing to notice is that the first block is shorter than 5 bytes. This forces Detours to copy this block as well as the beginning of the following block to the trampoline in order to make place for the jump to the hook. After the hooking, the code looks like this:

OriginalFunction2:
00401031  jmp         GenericHook (40101Dh)
00401036  int         3    		; Padding bytes inserted
00401037  int         3    		; by Detours
00401038  jmp         OriginalFunction2+2 (401033h)

Like last time, Detours fails to notice the short jump pointing back into the first 5 bytes. However, this time the consequences are worse — whereas last time the jump pointed to a jump instruction (the jump to the hook), this time the jump points to the third byte of the jump instruction! So after following the jump, the processor begins decoding at the wrong position in code and suddenly sees this:

OriginalFunction2:
00401031  db          e9h
00401032  db          e7h
00401033  db          ffh  ; <== EIP after following the jump
00401034  db          ffh
00401035  dec         esp
00401037  int         3
00401038  jmp         OriginalFunction2+2 (401033h)

Or, when using nop rather that xor:

OriginalFunction2:
00401031  db          e9h
00401032  out         0FFh,eax ; <== EIP after following the jump
00401034  db          ffh
00401035  dec         esp
00401037  jmp         OriginalFunction2+1 (401032h)

…which explains the exceptions.

This concludes the discussion of detouring functions with unusual prologs. However, there is another interesting catch regarding function un-hooking, which I will describe next time.