When HEAP_NO_SERIALIZE heap operations are not quite 'not serialized'
While the RTL Heap performs all heap operation in an interlocked manner by default, it can be requested not to serialize operations by passing the HEAP_NO_SERIALIZE flag to HeapCreate or HeapAlloc. In this case, the caller is in charge to provide proper synchronization.
As it turns out, things change when Application Verifier heap checks are enabled for this process. To check heap operations, Application Verifier intercepts certain operations. Presumably to protect its own internal data structures, at some points, Avrf enters a critical section, as shown in this stack trace:
0:009k
ChildEBP RetAddr
0840f540 77d9cfad ntdll!ZwWaitForSingleObject+0x15
0840f5a4 77d9cf78 ntdll!RtlpWaitOnCriticalSection+0x154
0840f5cc 6db25f92 ntdll!RtlEnterCriticalSection+0x152
0840f5f0 6db29d99 vfbasics!AVrfpFreeMemLockChecks+0x42
0840f60c 6db2bae8 vfbasics!AVrfpFreeMemNotify+0x39
0840f668 77731d27 vfbasics!AVrfpRtlFreeHeap+0x148
0840f67c 6d242757 kernel32!HeapFree+0x14
...
Entering a critical section is no big thing – nontheless, it effectvely means that heap operations, whether they are conducted with HEAP_NO_SERIALIZE flag set or not, must be considered to be at least partially serialized by Application Verifier.
Although this somewhat contradicts the idea of the HEAP_NO_SERIALIZE flag, this should not be a problem in most cases – especially when HEAP_NO_SERIALIZE has been passed for performance reasons only. However, if HEAP_NO_SERIALIZE has been used for correctness reasons, this might indeed bite you. In my specific case, I used HEAP_NO_SERIALIZE and did the synchronization myself in order to break a deadlock situation. And while the application ran fine (and deadlock-free) without Avrf enabled, it began deadlocking randomly as soon as I enabled Avrf heap checks – for the very reason just described. This is especially annoying as I am now left with two less-than-optimal choices: Leave Avrf enabled for my test suite and experience random deadlocks or forgo Avrf’s helpful heap checks in order not to deadlock. Grrr, I need more choices!
