Posts Tagged 'Testing'

How to test MFC applications using Visual Assert or cfix

Automated testing of GUI applications is tricky. It is not only tricky because testing the GUI itself is hard (despite there being good tools around), it is also tricky because GUI applications often tend to be a bit hostile towards unit testing.

One class of GUI applications for which this kind of hostility often applies is MFC applications. Although MFC allows the use of DLLs, components, etc, the framework still encourages the use of relatively monolithic architectures.

Regardless of whether this is a good or bad thing, the question is how to make unit testing MFC applications feasible and less painful.

In one of the last releases, both cfix and Visual Assert introduced the abilify to embed unit tests into executable modules. That is, the framework allows unit tests to be compiled and linked into the main application .exe file. Such an executable can then be run in two modes: On the one hand, it can be launched regularly and the existance of unit tests will largely go unnnoticed. On the other hand, the executable can be launched indirectly via cfix of Visual Assert — and in this case, the application will not have its WinMain routine run, but rather have its unit tests be executed.

Although this feature enables us to do unit testing without having to rethink or tweak the application’s architecture more than necessary, the question still in how far MFC likes this kind of testing.

At this point, a distinction has to be made on whether the unit tests make use of MFC APIs or not. Tests that only exercise internal methods will run smoothless, without further setup or precautions needed. Of course, this also holds for tests that make of of basic MFC functionality such as using classes like CString or CArray.

For test that make use of more “interesting” MFC functions, however, it is quite possible that you will observe slightly strange behavior: API calls failing, fields not having the proper value, or maybe even crashes.

An example for this is the field AFX_MODULE_STATE::m_hCurrentInstanceHandle. During a normal run, this field will hold the address of the current module. In a unit tests, however, the following assertion will fail:

CFIX_ASSERT( AfxGetModuleState()->m_hCurrentInstanceHandle != 0 );

If you think about this for a minute, this should not really come as a surprise. As described previously, Visual Assert/cfix, when attempting to run tests embedded into an executable module, will not call the applocation’s WinMain/main routine. They will ensure that all initializers (in particular: constructors of global C++ objects) are run, but calling the actual main routine is effectively skipped. Many of the MFC APIs, however, rely on quite a bit of initialization work that is performed during startup. Some of this work is conducted as part of constructors of global objects executed, a significant part of this work, however, is done in AfxWinInit.

Quoting MSDN:

This function is called by the MFC-supplied WinMain function, as part of the CWinApp initialization of a GUI-based application, to initialize MFC.

And indeed, if you look at the default MFC WinMain routine, AfxWinMain, you will see that it calls AfxWinInit.

The key to using “interesting” MFC APIs in your unit tests therefore is to make sure that AfxWinMain has been called before.

Unfortunately, there is no counterpart to AfxWinInit, so initializing MFC in a fixture’s Before or Setup routine and shutting MFC down in a After or Teardown routine is not feasible. Rather, you have to call it once per process, something that is a bit unusual for unit testing and requires a tiny bit of manual work. The easiest way to accomplish this is to a lazy initialization helper routine:

static void InitalizeMfc()
{
  static BOOL Initialized = FALSE;
  if ( ! Initialized )
  {
    CFIX_ASSERT( AfxWinInit(
      ::GetModuleHandle(NULL),
      NULL,
      ::GetCommandLine(),
      0 ) );
    Initialized = TRUE;
  }
}

…and call it from each fixture’s Setup routine:

static void SetUp()
{
  InitalizeMfc();
}

With this in place, MFC APIs will now work properly and AfxGetModuleState()->m_hCurrentInstanceHandle will contain a proper address.

With this bit of MFC background in mind, writing unit tests for MFC applications should be not much different than writing tests for any other kind of application.

One final tip: Adding unit tests to your project introduces a dependeny to cfix.dll. In your final build, you’ll want to remove (#ifdef out, for example) all your tests so this dependency will disappear. During development, however, this dependency might be a bit of a drag because there might be machines on which cfix is not available. To alleviate this situation, consider making cfix.dll a delay-load: Unless you want to run unit tests, it is then ok to miss cfix.dll on other machines.

Advertisements

Coming soon: Visual Assert for Visual Studio 2010

Now that Visual Studio 2010 has oficially been released, I keep getting questions about a Visual Studio 2010-enabled version of Visual Assert.

The fact that Visual Studio 2010 is already out, yet there is no Visual Assert version for it is unfortunate. It would have been nice to have Visual Studio 2010 support ready on Visual Studio’s release date, however, that was not possible due to lack of time. Having solved most compatibilty issues though (of which there were many, Visual Studio 2010 is a truly painful release for AddIn developers), I am now confident to be able to offer a first beta by begin of May.

This version will not only add support for Visual Studio 2010, but will also contain a set of other improvements and new features.

cfix 1.6 released, simplifies authoring of multi-threaded tests

A new release of cfix, the unit testing framework for C and C++, is now available for download. Besides some minor enhancements like extending the maximum permitted fixture name, cfix 1.6 introduces a major new feature, Anonymous Thread Auto-Registration.

Since its very first release, cfix has supported multi-threaded test cases, i.e. test cases that spawn child threads, each of which potentially making use of the various assertion statements like CFIX_ASSERT. To make this work and ensure that failing assertions are handled properly, however, usage of CfixCreateThread (rather than the native Win32 CreateThread) was mandatory when spawning such threads.

Although using CfixCreateThread (or CfixCreateSystemThread in case of kernel mode code) remains the preferred way to create child threads, there are situations where usage of this API is not possible — for example, when the child threads are created by libraries such as boost.

To support such scenarios, cfix now allows you to annotate a fixture to enable Anonymous Thread Auto-Registration, which means that newly spawned threads are automatically registered with cfix — no usage of CfixCreateThread required. Thanks to this feature, writing multi-threaded tests becomes straightforward — and integrating with libraries such as boost does not pose a problem any more.

For more details about this feature, please refer to the respective section in the cfix documentation.

As always, updated binaries and source code are available on Sourceforge.

Visual Assert Beta 3 released

A third beta release of Visual Assert is now available for download on www.visualassert.com.

Visual Assert, in case you have not tried it yet, is an Add-In for Visual Studio that adds unit testing capabilities to the Visual C++ IDE: Based on the cfix unit testing framework, Visual Assert allows unit tests to be written, run, and debugged from within the IDE. Pretty much like Junit/Eclipse, TestDriven.Net or MSTest, but for real, native code — code written in C or C++.

Bugs, bugs, bugs, bugs

Alas, there were still a few of them in the previous two beta releases. Luckily though, almost all I received from users or found by internal testing were relatively minor in nature. Still, I want Visual Assert to be as high quality as possible and decided to add this third beta release into the schedule and take the time to focus on — you guessed it — bugfixing, bugfixing, bugfixing, and bugfixing.

Speaking of bug reports, I have to thank all users of Visual Assert and cfix who reported bugs, suggested new features or provided general feedback. Your input has been, and still is highly appreciated. Although I had to postpone any feature suggestions to a later release, I tried hard to resolve all bugs and have them fixed in this new release.

Download, Try it, Share Your Opinion

Of course, using the new Beta version is free. So whether you have already used the previous beta or not, whether you are a unit testing newbie or write unit tests on a daily basis, be sure to give the new version a try. And of course, do not forget to let me know about your feedback, suggestions, found bugs, etc.!

cfix studio Beta 2 to add support for EXE-based unit tests

N.B. cfix studio was the code name of what has become Visual Assert

The biggest shortcoming of the current cfix studio version certainly is that it requires all tests be implemented in a DLL. Conceptually, keeping test cases separated from the remaining code certainly is a good idea — and implementing tests in a DLL is a way to accomplish this. However, there are many projects in which such separation is either not feasible or just too much effort.

The good news is that with Beta 2, this will finally change: EXEs become first class-citizens in cfix studio and it will not matter any more whether your tests are part of a DLL or EXE project — you can just put them where you think is appropriate.

Take a classic MFC/GUI application project as an example: It is pretty common for these kinds of projects that most, if not all, application logic is part of a single Visual Studio project that compiles into a single EXE. There may be some additional DLLs or LIBs, but by and large, the EXE itself is where most of the interesting things happen.

The upcoming Beta 2 release now allows you to implement all your unit tests as part of the same EXE project. This means that your tests have access to all classes, functions and resources that are part of the project — all of which you would not easily have access to if you implemented the tests in a separate DLL.

Of course, embedding unit tests into an executable raises two questions:

  1. How to strip the tests from the final release?
  2. How on earth will you be able to run these tests without having main() create windows, load files, play sounds, etc each time?

Thankfully, C/C++ has a preprocessor and Visual C++ has the “exclude from build” feature which allows you to exclude certain files whenever the project is built using a specific configuration. Using any of these two features, the first question is easily answered.

The second problem is more tricky — but thankfully, it has already been solved for you: When cfix studio runs unit tests, it is well aware of that running main() might have, let’s say interesting effects — so what it does is simple: It just makes sure that main() is never run! Not only does this ensure that the tests run silently, i.e. without windows popping up etc, it also has the benefit that all unit tests “see” the application in a pristine state: Rather than having to worry about which state main() has brought the application into, you can initialize and clean up any state you need in your Setup and Teardown functions1.

To make a long story short: You can write unit tests in EXE projects in exactly the same manner as you would in a DLL project. No special considerations needed, no project settings that need to be changed, no additional boilerplate code to write. And when you run the EXE outside cfix studio, i.e. hit F5 in Visual Studio or launch the EXE directly, you will not even notice that the EXE houses some unit tests — everything works as normal.

Sounds good? Then wait a few more days and see it in action!

Remarks
1: Needless to say, all global variables are initialized, constructors are run, etc. All CRT initialization happens as normal; only main()/WinMain() is not run. And yes, it also works for apps that link statically to MFC and therefore do not have a “regular” WinMain().

cfix 1.0.1 adds support for Windows 2000

Despite the fact that mainstream support for Windows 2000 has ended in 2005 and the system is well on its way to retirement, Windows 2000 is still in wide use today. As such, it remains being an important target platform for many software packages.

The fact that cfix has not provided support for Windows 2000 was thus unfortunate — after all, if Windows 2000 is among the target platforms of your software, you should be able to run your tests on this platform.

cfix 1.0.1 remedies this shortcoming and adds Windows 2000 i386 SP4 as an supported platform.

The updated packages are available for download on Sourceforge.


Categories




About me

Johannes Passing, M.Sc., living in Berlin, Germany.

Besides his consulting work, Johannes mainly focusses on Win32, COM, and NT kernel mode development, along with Java and .Net. He also is the author of cfix, a C/C++ unit testing framework for Win32 and NT kernel mode, Visual Assert, a Visual Studio Unit Testing-AddIn, and NTrace, a dynamic function boundary tracing toolkit for Windows NT/x86 kernel/user mode code.

Contact Johannes: jpassing (at) acm org

Johannes' GPG fingerprint is BBB1 1769 B82D CD07 D90A 57E8 9FE1 D441 F7A0 1BB1.

LinkedIn Profile
Xing Profile
Github Profile
Advertisements