Archive for the 'Misc' Category

Runtime Code Modification Explained, Part 1: Dealing With Memory

Runtime code modification, of self modifying code as it is often referred to, has been used for decades — to implement JITters, writing highly optimized algorithms, or to do all kinds of interesting stuff. Using runtime code modification code has never been really easy — it requires a solid understanding of machine code and it is straightforward to screw up. What’s not so well known, however, is that writing such code has actually become harder over the last years, at least on the IA-32 platform: Comparing the 486 and current Core architectures, it becomes obvious that Intel, in order to allow more advanced CPU-interal optimizations, has actually lessened certain gauarantees made by the CPU, which in turn requires the programmer to pay more attection to certain details.

Looking around on the web, there are plenty of code snippets and example projects that make use of self-modifying code. Without finger-pointing specific resources, it is, however, safe to assume that a significant (and I mean significant!) fraction of these examples fail to address all potential problems related to runtime code modification. As I have shown a while ago, even Detours, which is a well-done and widely recognized and used library relying on runtime code modification has its issues:

Adopting the nomenclature suggested by the Intel processor manuals, code writing data to memory with the intent of having the same processor execute this data as code is referred to as self-modifying code. On SMP machines, it is possible for one processor to write data to memory with the intent of having a different processor execute this data as code. This process if referred to as cross-modifying code. I will jointly refer to both practices as runtime code modification.

Memory Model

The easiest part of runtime code modification is dealing with the memory model. In order to implement self-modifying or cross-modifying code, a program must be able to address the regions of memory containing the code to be modified. Moreover, due to memory protection mechanisms, overwriting code may not be trivially possible.

The IA-32 architecture offers three memory models — the flat, segmented and real mode memory model. Current OS like Windows and Linux rely on the flat memory model, so I will ignore the other two.

Whenever the CPU fetches code, it addresses memory relative to the segment mapped by the CS segment register. In the flat memory model, the CS segment register, which refers to the current code segment, is always set up to map to linear address 0. In the same manner, the data and stack segment registers (DS, SS) are set up to refer to linear address 0.

It is worth mentioning that AMD64 has retired the use of segmentation and the segment bases for code and data segment are therefore always treated as 0.

Given this setup, code can be accessed and modified on IA-32 as well as on AMD64 in the same manner as data. Easy-peasy.

Memory Protection

One of the features enabled by the use of paging is the ability to enforce memory protection. Each page can specify restrictions to which operations are allowed to be performed on memory of the respective page.

In the context of runtime code modification, memory protection is of special importance as memory containing code usually does not permit write access, but rather read and execute access only. A prospective solution thus has to provide a means to either circumvent such write protection or to temporarily grant write access to the required memory areas.

As other parts of the image are write-protected as well, memory protection equally applies to approaches that modify non-code parts of the image such as the Import Address Table. That’s why the call to VirtualProtect is neccessary when Patching the IAT. Programs using runtime code modification often do not restrict themselves to changing existing code but rather generate additional code. Assuming Data Execution Prevention has been enabled, it is thus vital for such approaches to work properly that any code generated is placed into memory regions that grant execute access. While user mode implementations can rely on a feature of the RTL heap (i.e. using the HEAP_CREATE_ENABLE_EXECUTE when calling RtlCreateHeap) for allocating executable memory, no comparable facility for kernel mode exist — a potential instrumentation solution thus has to come up with a custom allocation strategy.

Jump distances

Whenever code is being generated, odds are that there are branching instructions involved. Depending on where memory for the new code has been allocated and where the branch targets falls, the offset between the branching instruction itself and the jump target may be of significant size. In such cases, the software has to make sure that the branch instruction chosen does in fact support offsets at least as large as required for the individual purpose. This sounds trivial, but it is not: Software that overwrites existing code with a branch may face severe limitation w.r.t. how many bytes the branch instruction may occupy — if, for example, there is less than 5 bytes of space (assuming IA-32), a far jump cannot be used. To use a near jump, however, the newly allocated code better be near.

Further safety concerns will be discussed in Part 2 of this series of posts.

cfix finished 2nd in ATI’s Automation Honors Awards, surpassed only by JUnit

Along with JUnit, JWebUnit, NUnit, and SimpleTest, cfix was one of the nominees for the Automated Testing Institute’s Automation Honors Award 2009 in the category Best Open Source Unit Automated Test Tool. A few days ago, the results were published and cfix finished second — surpassed only by JUnit, which finished 1st (No real surprise here). If you are interested, there is a video in which the results are presented.

Overview on Designing High-Performance Windows Applications

Back in 2008, the Windows Server Performance Team Blog, which I came across recently, ran a series of posts on Designing Applications for High Performance:

If you are interested in developing server side applications for Windows, these articles are definitely worth reading.

Vote for cfix

The Automated Testing Institute has elected cfix to be one of the finalists for the Autmation Honors award. The winners of the award will be highlighted in a Special December Edition of the Automated Software Testing Magazine.

If you are a cfix user, be sure to vote for cfix here.

And by the way, I think The Grinder, which is a really neat web performance testing framework, also deserves being voted for…

More Context Menu Handlers for Everyday Use

Although Windows Explorer may actually not be the brightest spot of Windows, it is still, for most users, among the most often used pograms. Customizing it to speed up certain tasks is thus a natural desire.

A while ago, I wrote about how to extend the context menu by new commands that allow MSI packages to be installed/uninstalled with logfiles being created. The registry entries I used were:

Windows Registry Editor Version 5.00

@="&Logged Install"

@="msiexec.exe /l* \"%1-install.log\" /i \"%1\" %*"

@="L&ogged Uninstall"

@="msiexec.exe /l* \"%1-uninstall.log\" /x \"%1\" %*"

@="msiexec.exe /l* \"%1.log\" /i \"%1\" %*"

While I do not use Windows Installer every day, I am a heavy user of cmd.exe command prompts. Another set of custom verbs I use on my machines therefore deal with opening command line windows. Getting Windows Explorer to open a “normal” command prompt using the context menu is not hard and it has been demonstrated on various places. The idea becomes truly powerful, though, when the commands are specialized to open special kinds of command windows:

  • A plain command prompt
  • An elevated command prompt (using elevate.exe)
  • A WDK command prompt (WLH-chk)
  • A WDK command prompt (WLHA64-chk)
  • A Visual Studio 2005 command prompt
  • etc …

To distinguish the different types of consoles, I like to use different colors — The Visual Studio command prompt is white/green, the elevated prompt is green/blue, and so on. The following script puts it all together (mind the static paths):

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Directory\shell\Open DDK Console here (WLH-chk)]
@="Open DD&K Console here (WLH-chk)"

[HKEY_CLASSES_ROOT\Directory\shell\Open DDK Console here (WLH-chk)\command]
@="C:\\Windows\\System32\\cmd.exe /k C:\\WinDDK\\6000\\bin\\setenv.bat C:\\WinDDK\\6000\\ chk WLH && cd /D %1 && color 1f"

[HKEY_CLASSES_ROOT\Directory\shell\Open DDK Console here (WLHA64-chk)]
@="Open DD&K Console here (WLHA64-chk)"

[HKEY_CLASSES_ROOT\Directory\shell\Open DDK Console here (WLHA64-chk)\command]
@="C:\\Windows\\System32\\cmd.exe /k C:\\WinDDK\\6000\\bin\\setenv.bat C:\\WinDDK\\6000\\ chk AMD64 && cd /D %1 && color 1f"

[HKEY_CLASSES_ROOT\Directory\shell\Open Default Console here]
@="Open Default Conso&le here"

[HKEY_CLASSES_ROOT\Directory\shell\Open Default Console here\command]
@="cmd.exe /K \"title %1 && cd /D %1\""

[HKEY_CLASSES_ROOT\Directory\shell\Open Elevated Console here]
@="Open Ele&vated Console here"

[HKEY_CLASSES_ROOT\Directory\shell\Open Elevated Console here\command]
@="d:\\bin\\elevate.exe /K \"title %1 && color 1a && cd /D %1\""

[HKEY_CLASSES_ROOT\Directory\shell\Open VS.Net 2005 Console here]
@="Open VS.Net 200&5 Console here"

[HKEY_CLASSES_ROOT\Directory\shell\Open VS.Net 2005 Console here\command]
@="cmd.exe /K \"cd /D %1 && \"C:\\Program Files (x86)\\Microsoft Visual Studio 8\\VC\\vcvarsall.bat\" && color 2f\""

Finally, if you perform backups to the cloud from time to time and do want to upload unencrypted data or for other reasons encrypt specific files occasionaly, it may also be practical to have two GPG commands in your context menu — one to (symetrically) encrypt, and one to decrypt:

Windows Registry Editor Version 5.00





@="\"c:\\Program Files (x86)\\GNU\\GnuPG\\gpg.exe\" -d -i -o %1.plain %1"

@="Encrypt with GPG (symmetric)"

@="\"c:\\Program Files (x86)\\GNU\\GnuPG\\gpg.exe\" -c %1"

Bryan Cantrill on Real-World Concurrency

Browsing through ACM Queue’s archives I came across the article Real-World Concurrency by Bryan Cantrill (who, by the way, is the inventor of DTrace) and Jeff Bonwick (Issue 5/2008). The article provides a nice summary of actual challenges and best practices for systems programming in a multithreaded/shared memory environment. Worth reading.

Google Calendar, WTF?

Quite obviously, Google does not always get it right either. Ever when I try to see my Google Calendar (using Opera), I am requested to login. So I enter my credentials, am redirected a couple of times and — are broght to the login page again. Logging in again does not help, I have by then entered an infinite loop. Thankfully, I can escape this loop by jumping to the original calendar URL again — now Google recognizes that I have already logged in and shows me my calendar. Great.

But this one is even better. Having received an invitation, I was presented with a page offering me to accept or reject the event.

Look at the screenshot — you can select all three options Yes, No and Maybe at the same time! Very convenient indeed. Having submitted the form, this answer seems to have been recognized as a Yes. Amazing.


About me

Johannes Passing, M.Sc., living in Berlin, Germany.

Besides his consulting work, Johannes mainly focusses on Win32, COM, and NT kernel mode development, along with Java and .Net. He also is the author of cfix, a C/C++ unit testing framework for Win32 and NT kernel mode, Visual Assert, a Visual Studio Unit Testing-AddIn, and NTrace, a dynamic function boundary tracing toolkit for Windows NT/x86 kernel/user mode code.

Contact Johannes: jpassing (at) acm org

Johannes' GPG fingerprint is BBB1 1769 B82D CD07 D90A 57E8 9FE1 D441 F7A0 1BB1.

LinkedIn Profile
Xing Profile
Github Profile