When HEAP_NO_SERIALIZE heap operations are not quite ‘not serialized’

While the RTL Heap performs all heap operation in an interlocked manner by default, it can be requested not to serialize operations by passing the HEAP_NO_SERIALIZE flag to HeapCreate or HeapAlloc. In this case, the caller is in charge to provide proper synchronization.

As it turns out, things change when Application Verifier heap checks are enabled for this process. To check heap operations, Application Verifier intercepts certain operations. Presumably to protect its own internal data structures, at some points, Avrf enters a critical section, as shown in this stack trace:

0:009> k
ChildEBP RetAddr  
0840f540 77d9cfad ntdll!ZwWaitForSingleObject+0x15
0840f5a4 77d9cf78 ntdll!RtlpWaitOnCriticalSection+0x154
0840f5cc 6db25f92 ntdll!RtlEnterCriticalSection+0x152
0840f5f0 6db29d99 vfbasics!AVrfpFreeMemLockChecks+0x42 
0840f60c 6db2bae8 vfbasics!AVrfpFreeMemNotify+0x39 
0840f668 77731d27 vfbasics!AVrfpRtlFreeHeap+0x148 
0840f67c 6d242757 kernel32!HeapFree+0x14

Entering a critical section is no big thing — nontheless, it effectvely means that heap operations, whether they are conducted with HEAP_NO_SERIALIZE flag set or not, must be considered to be at least partially serialized by Application Verifier.

Although this somewhat contradicts the idea of the HEAP_NO_SERIALIZE flag, this should not be a problem in most cases — especially when HEAP_NO_SERIALIZE has been passed for performance reasons only. However, if HEAP_NO_SERIALIZE has been used for correctness reasons, this might indeed bite you. In my specific case, I used HEAP_NO_SERIALIZE and did the synchronization myself in order to break a deadlock situation. And while the application ran fine (and deadlock-free) without Avrf enabled, it began deadlocking randomly as soon as I enabled Avrf heap checks — for the very reason just described. This is especially annoying as I am now left with two less-than-optimal choices: Leave Avrf enabled for my test suite and experience random deadlocks or forgo Avrf’s helpful heap checks in order not to deadlock. Grrr, I need more choices!



About me

Johannes Passing, M.Sc., living in Berlin, Germany.

Besides his consulting work, Johannes mainly focusses on Win32, COM, and NT kernel mode development, along with Java and .Net. He also is the author of cfix, a C/C++ unit testing framework for Win32 and NT kernel mode, Visual Assert, a Visual Studio Unit Testing-AddIn, and NTrace, a dynamic function boundary tracing toolkit for Windows NT/x86 kernel/user mode code.

Contact Johannes: jpassing (at) acm org

Johannes' GPG fingerprint is BBB1 1769 B82D CD07 D90A 57E8 9FE1 D441 F7A0 1BB1.

LinkedIn Profile
Xing Profile
Github Profile

%d bloggers like this: