Archive Page 2

How to test MFC applications using Visual Assert or cfix

Automated testing of GUI applications is tricky. It is not only tricky because testing the GUI itself is hard (despite there being good tools around), it is also tricky because GUI applications often tend to be a bit hostile towards unit testing.

One class of GUI applications for which this kind of hostility often applies is MFC applications. Although MFC allows the use of DLLs, components, etc, the framework still encourages the use of relatively monolithic architectures.

Regardless of whether this is a good or bad thing, the question is how to make unit testing MFC applications feasible and less painful.

In one of the last releases, both cfix and Visual Assert introduced the abilify to embed unit tests into executable modules. That is, the framework allows unit tests to be compiled and linked into the main application .exe file. Such an executable can then be run in two modes: On the one hand, it can be launched regularly and the existance of unit tests will largely go unnnoticed. On the other hand, the executable can be launched indirectly via cfix of Visual Assert — and in this case, the application will not have its WinMain routine run, but rather have its unit tests be executed.

Although this feature enables us to do unit testing without having to rethink or tweak the application’s architecture more than necessary, the question still in how far MFC likes this kind of testing.

At this point, a distinction has to be made on whether the unit tests make use of MFC APIs or not. Tests that only exercise internal methods will run smoothless, without further setup or precautions needed. Of course, this also holds for tests that make of of basic MFC functionality such as using classes like CString or CArray.

For test that make use of more “interesting” MFC functions, however, it is quite possible that you will observe slightly strange behavior: API calls failing, fields not having the proper value, or maybe even crashes.

An example for this is the field AFX_MODULE_STATE::m_hCurrentInstanceHandle. During a normal run, this field will hold the address of the current module. In a unit tests, however, the following assertion will fail:

CFIX_ASSERT( AfxGetModuleState()->m_hCurrentInstanceHandle != 0 );

If you think about this for a minute, this should not really come as a surprise. As described previously, Visual Assert/cfix, when attempting to run tests embedded into an executable module, will not call the applocation’s WinMain/main routine. They will ensure that all initializers (in particular: constructors of global C++ objects) are run, but calling the actual main routine is effectively skipped. Many of the MFC APIs, however, rely on quite a bit of initialization work that is performed during startup. Some of this work is conducted as part of constructors of global objects executed, a significant part of this work, however, is done in AfxWinInit.

Quoting MSDN:

This function is called by the MFC-supplied WinMain function, as part of the CWinApp initialization of a GUI-based application, to initialize MFC.

And indeed, if you look at the default MFC WinMain routine, AfxWinMain, you will see that it calls AfxWinInit.

The key to using “interesting” MFC APIs in your unit tests therefore is to make sure that AfxWinMain has been called before.

Unfortunately, there is no counterpart to AfxWinInit, so initializing MFC in a fixture’s Before or Setup routine and shutting MFC down in a After or Teardown routine is not feasible. Rather, you have to call it once per process, something that is a bit unusual for unit testing and requires a tiny bit of manual work. The easiest way to accomplish this is to a lazy initialization helper routine:

static void InitalizeMfc()
{
  static BOOL Initialized = FALSE;
  if ( ! Initialized )
  {
    CFIX_ASSERT( AfxWinInit(
      ::GetModuleHandle(NULL),
      NULL,
      ::GetCommandLine(),
      0 ) );
    Initialized = TRUE;
  }
}

…and call it from each fixture’s Setup routine:

static void SetUp()
{
  InitalizeMfc();
}

With this in place, MFC APIs will now work properly and AfxGetModuleState()->m_hCurrentInstanceHandle will contain a proper address.

With this bit of MFC background in mind, writing unit tests for MFC applications should be not much different than writing tests for any other kind of application.

One final tip: Adding unit tests to your project introduces a dependeny to cfix.dll. In your final build, you’ll want to remove (#ifdef out, for example) all your tests so this dependency will disappear. During development, however, this dependency might be a bit of a drag because there might be machines on which cfix is not available. To alleviate this situation, consider making cfix.dll a delay-load: Unless you want to run unit tests, it is then ok to miss cfix.dll on other machines.

Visual Assert 1.1 beta and cfix 1.7 released

Slightly delayed, Visual Assert 1.1 beta is now available for download. As announced in a previous post, the most important change in the new version is added suport for the latest version of Visual Studio, Visual Studio 2010.

However, the new version also brings a couple of new features that apply to all versions of Visual Studio. Most importantly, cfix and Visual Assert now expose an API that allows developers to plug in custom event sinks. A custom event sink is implemented as a DLL and receives all events the runtime generates during the execution of a test suite. As such, the API is perfectly suited for implementing custom loggers.

To make this new feature easily usable, the Options dialog (Menu: Tools > Options) has been enhanced appropriately. Moreover, the dialog now includes some further options concerning stack size, current directory adjustment, and VC++ directory registration that have not been exposed previously.

Coming along with the new Visual Assert release, a new cfix release, version 1.7, is now available on Sourceforge.

Coming soon: Visual Assert for Visual Studio 2010

Now that Visual Studio 2010 has oficially been released, I keep getting questions about a Visual Studio 2010-enabled version of Visual Assert.

The fact that Visual Studio 2010 is already out, yet there is no Visual Assert version for it is unfortunate. It would have been nice to have Visual Studio 2010 support ready on Visual Studio’s release date, however, that was not possible due to lack of time. Having solved most compatibilty issues though (of which there were many, Visual Studio 2010 is a truly painful release for AddIn developers), I am now confident to be able to offer a first beta by begin of May.

This version will not only add support for Visual Studio 2010, but will also contain a set of other improvements and new features.

What a weirdo: How the /analyze switch changes its behavior depending on its environment

In Visual Studio 2005 Team System (VSTS), the “ultimate” SKU of Visual Studio 2005, Microsoft introduced the /analyze compiler switch. When the /analyze switch is used, the cl compiler not only does its regular checks, but performs a much more thorough static code analysis.

While /analyze is very useful indeed, it was only available in the top SKU — the Standard and Professional versions of Visual Studio lacked support for this compiler switch (this has changed by now, Professional now also supports this feature). As some smart people quickly figured out though, the compilers shipped as part of the Windows SDK did support /analyze, too.

So given that some compilers do support /analyze while other do not, you may well expect that there are two slightly different types of binaries, one that the SDK and VSTS uses, and one that is shipped with other Visual Studio SKUs.

At least this was what I expected. As it turns out though, this is not quite the case.

Where’s /analyze?

For the past two years, I have been developing using Visual Studio 2005 Team System along with Windows SDK 6.0 and WDK 6000 on a Vista x64 machine. Using this setup, I was able to use the /analyze switch in both, “regular” Visual Studio projects and WDK (build.exe-driven) projects. That led me to the conclusion that the WDK 6000 compilers, like the SDK compilers were in fact /analyze-enabled binaries as well.

Switching to a Windows 7 machine with VSTS 2005 and 2008, SDK 7.0, and WDK 6000 did not change this — /analyze kept working fine in all environments.

Then I set up a build server, installed WDK 6000 and Windows SDK 7.0 and attempted to perform a build — to my surprise, though, I got plenty of complaints about the /anayze switch not being supported.

I verified that the right compilers (WDK 6000) were used and compared cl versions between the build machine and my development machine — both were 14.00.50727.220, so everything seemed right. Running cl.exe /? on both machines, however, I noticed that despite versions being the same, this Code Analsis section was missing in the output on the build machine:

                         -CODE ANALYSIS-

/analyze[:WX-] enable code analysis
    WX- - code analysis warnings should not be treated as errors even if /WX is invoked

So obviously, Code Analysis support is enabled or disabled depending on external factors — not the binary itself, but the environment somehow determines whether the /analyze switch is supported or not.

Observing cl.exe /? with Process Monitor on my development machine resulted in the following output:

Process Monitor tracing the search for c1xxast

This trace leaves little room for interpretation: The code analysis features must (mainly) be implemented in c1xxast.dll. c1xxast.dll, however, is not shipped with the WDK itself, nor is it shipped with the non-VSTS SKUs of Visual Studio. So by default, the WDK’s cl will fail to locate the DLL and will revert to “/analyze-disabled mode”.

If, however, you have VSTS or the Windows SDK installed on your machine and your %PATH% happens to include the right directories, cl’s search for c1xxast.dll will succeeded and — tada — /analyze suddenly works. On my development machine, this obviously was the case, whilst on the build machine, it was not.

Compiler version mish-mash

I added the Windows SDK’s bin directory to the build machine’s %PATH% and rerun the build. As I expected, /analyze now worked fine — what I did not quite expect though was that I was now getting dozens of compilation warnings like:

warning C6309: Argument '1' is null: this does not adhere to function 
specification of 'CfixCreateThread'

The reason for this was simple: The WDK cl.exe (remember, version 14.00.50727.220), thanks to a proper %PATH%, now used c1xxast.dll from SDK 7 to perform code analysis — despite the fact that c1xxast.dll actually “belonged” to cl version 15.00.30729.01. So the c1xxast.dll was one generation ahead of the WDK I was using.

The really, really cool thing about cl being able to work with a newer c1xxast.dll is that you can continue using WDK 6000 or 6001 (with W2K support!) and still benefit from the latest-and-greatest static code analysis features.

The reason for getting several warnings on the build machine while not getting similar warnings on my development machine was simply that on my development machine, the VS 2005 directory preceded the SDK directory in my %PATH%. Once I switched the order, I got the same wanings on both machines. This leads me to:

The ugly thing about this, however, is that a tiny change in the order of directories in %PATH% can suddenly make a huge difference w.r.t. code analysis. This is not quite what you’d normally expect.

(The additional compiler warnings, by the way, were a result of the improved analysis checks in cl 15: cl 14 routinely failed to verify the usage of __in vs. __in_opt parameters; cl 15 has become much more precise here and found several mis-attributed function signatures.)

Visual Assert hits RTM, now available for free

Visual Assert, the unit testing Add-In for Visual Studio/Visual C++ has finally left its beta status and — better yet — is now available for free, both for commercial and non-commercial use.

Visual Assert, based on the cfix 1.6 unit testing framework, allows you to easily write, manage, run, and debug your C/C++ unit tests -– without ever leaving the Visual Studio® IDE. No fiddling with command line tools, no complex configuration, and no boilerplate code required.

Sounds good? Then go straight to the Visual Assert homepage and download the installer!

cfix 1.6 released, simplifies authoring of multi-threaded tests

A new release of cfix, the unit testing framework for C and C++, is now available for download. Besides some minor enhancements like extending the maximum permitted fixture name, cfix 1.6 introduces a major new feature, Anonymous Thread Auto-Registration.

Since its very first release, cfix has supported multi-threaded test cases, i.e. test cases that spawn child threads, each of which potentially making use of the various assertion statements like CFIX_ASSERT. To make this work and ensure that failing assertions are handled properly, however, usage of CfixCreateThread (rather than the native Win32 CreateThread) was mandatory when spawning such threads.

Although using CfixCreateThread (or CfixCreateSystemThread in case of kernel mode code) remains the preferred way to create child threads, there are situations where usage of this API is not possible — for example, when the child threads are created by libraries such as boost.

To support such scenarios, cfix now allows you to annotate a fixture to enable Anonymous Thread Auto-Registration, which means that newly spawned threads are automatically registered with cfix — no usage of CfixCreateThread required. Thanks to this feature, writing multi-threaded tests becomes straightforward — and integrating with libraries such as boost does not pose a problem any more.

For more details about this feature, please refer to the respective section in the cfix documentation.

As always, updated binaries and source code are available on Sourceforge.

How to customize test run execution to make your cfix test runs more effective

One of the features introduced back in cfix 1.2 was the ability to customize test execution with the command line parameters -fsr and -fsf. Using these switches can make your test runs more effective and can help simplify debugging — so it is worth spending a minute on this topic.

Assume our test run comprises two fixtures, Fixture A and Fixture B. As fixtures are always run in alphabetic order, and tests run in the order they are defined, the first test to be executed is Test 1 of Fixture A, followed by Test 2 of Fixture A, and so on. Assuming all tests of Fixture A succeed, execution then proceeds with the first test of Fixture B. The following figure illustrates this:

Successful execution of a unit testing suite

Things get interesting when one of the tests fails: Let us assume Test 2 of Fixture A leads to a failed assertion. The default behavior of cfix, which equals the behavior of JUnit and NUnit, is to report the failure and proceed with Test 3 of the same fixture:

Execution resumes after a failed unit test

In many cases, this is the appropriate thing to do — letting the test run to completion and figure out the reasons for the failure afterwards. But in certain scenarios, resuming with Test 3 might not be the smartest thing to do — rather, as soon as a test fails, execution should proceed with the next fixture. This is what -fsf (short-circuit fixture”) does:

Short-circuiting the fixture

Once a test case has failed, all remaining tests of the same fixture are skipped.

When debugging, even resuming at the next fixture is often not desired. Rather, as soon as one test fails, you may immediately want to take a closer a look at the failure, so resuming the run is mute. -fsr (short-circuit run”) does right that: It aborts the run immediately.

Short-circuiting the run

By the way — in Visual Assert, you can quickly access these options in the Options menu in the Test Explorer window:

Visual Assert Test Explorer


Categories




About me

Johannes Passing, M.Sc., living in Berlin, Germany.

Besides his consulting work, Johannes mainly focusses on Win32, COM, and NT kernel mode development, along with Java and .Net. He also is the author of cfix, a C/C++ unit testing framework for Win32 and NT kernel mode, Visual Assert, a Visual Studio Unit Testing-AddIn, and NTrace, a dynamic function boundary tracing toolkit for Windows NT/x86 kernel/user mode code.

Contact Johannes: jpassing (at) acm org

Johannes' GPG fingerprint is BBB1 1769 B82D CD07 D90A 57E8 9FE1 D441 F7A0 1BB1.

LinkedIn Profile
Xing Profile
Github Profile