Posts Tagged 'qa'

Coming soon: Visual Assert for Visual Studio 2010

Now that Visual Studio 2010 has oficially been released, I keep getting questions about a Visual Studio 2010-enabled version of Visual Assert.

The fact that Visual Studio 2010 is already out, yet there is no Visual Assert version for it is unfortunate. It would have been nice to have Visual Studio 2010 support ready on Visual Studio’s release date, however, that was not possible due to lack of time. Having solved most compatibilty issues though (of which there were many, Visual Studio 2010 is a truly painful release for AddIn developers), I am now confident to be able to offer a first beta by begin of May.

This version will not only add support for Visual Studio 2010, but will also contain a set of other improvements and new features.


cfix 1.6 released, simplifies authoring of multi-threaded tests

A new release of cfix, the unit testing framework for C and C++, is now available for download. Besides some minor enhancements like extending the maximum permitted fixture name, cfix 1.6 introduces a major new feature, Anonymous Thread Auto-Registration.

Since its very first release, cfix has supported multi-threaded test cases, i.e. test cases that spawn child threads, each of which potentially making use of the various assertion statements like CFIX_ASSERT. To make this work and ensure that failing assertions are handled properly, however, usage of CfixCreateThread (rather than the native Win32 CreateThread) was mandatory when spawning such threads.

Although using CfixCreateThread (or CfixCreateSystemThread in case of kernel mode code) remains the preferred way to create child threads, there are situations where usage of this API is not possible — for example, when the child threads are created by libraries such as boost.

To support such scenarios, cfix now allows you to annotate a fixture to enable Anonymous Thread Auto-Registration, which means that newly spawned threads are automatically registered with cfix — no usage of CfixCreateThread required. Thanks to this feature, writing multi-threaded tests becomes straightforward — and integrating with libraries such as boost does not pose a problem any more.

For more details about this feature, please refer to the respective section in the cfix documentation.

As always, updated binaries and source code are available on Sourceforge.

How to customize test run execution to make your cfix test runs more effective

One of the features introduced back in cfix 1.2 was the ability to customize test execution with the command line parameters -fsr and -fsf. Using these switches can make your test runs more effective and can help simplify debugging — so it is worth spending a minute on this topic.

Assume our test run comprises two fixtures, Fixture A and Fixture B. As fixtures are always run in alphabetic order, and tests run in the order they are defined, the first test to be executed is Test 1 of Fixture A, followed by Test 2 of Fixture A, and so on. Assuming all tests of Fixture A succeed, execution then proceeds with the first test of Fixture B. The following figure illustrates this:

Successful execution of a unit testing suite

Things get interesting when one of the tests fails: Let us assume Test 2 of Fixture A leads to a failed assertion. The default behavior of cfix, which equals the behavior of JUnit and NUnit, is to report the failure and proceed with Test 3 of the same fixture:

Execution resumes after a failed unit test

In many cases, this is the appropriate thing to do — letting the test run to completion and figure out the reasons for the failure afterwards. But in certain scenarios, resuming with Test 3 might not be the smartest thing to do — rather, as soon as a test fails, execution should proceed with the next fixture. This is what -fsf (short-circuit fixture”) does:

Short-circuiting the fixture

Once a test case has failed, all remaining tests of the same fixture are skipped.

When debugging, even resuming at the next fixture is often not desired. Rather, as soon as one test fails, you may immediately want to take a closer a look at the failure, so resuming the run is mute. -fsr (short-circuit run”) does right that: It aborts the run immediately.

Short-circuiting the run

By the way — in Visual Assert, you can quickly access these options in the Options menu in the Test Explorer window:

Visual Assert Test Explorer

cfix 1.5.1 released

A new version of cfix, the unit testing framework for C and C++ on Windows, is now available on Sourceforge. Despite fixing several minor issues, the new version resolves the following two issues that were reported by users:

  • Definiting multiple WinUnit fixtures with setup/teardown routines in a single .cpp file leads to a compilation error
  • A thread handle is leaked during execution of a test (#2889511)

Updated binaries and source code are available for download on Sourceforge.

Btw, in case you use cfix for kernel mode testing and are using WDK 7600, please have a look at my previous post: LTCG issues with the WIN7/amd64 environment of WDK 7600

Visual Assert Beta 2 Released

The Beta 2 release of Visual Assert (formerly named cfix studio) is now available for download. The release marks a major step in the development of Visual Assert for that it not only comprises a number of bugfixes but also introduces major new features. The two most important certainly are support for EXE targets and Wizard assistance.

Support for EXE Targets

As announced in a previous post and also discussed in the post about the cfix 1.5 release, Visual Assert now fully supports unit tets emedded in EXE modules.

Previous releases required all unit tests to be compiled and linked into DLLs. In fact, the usage of DLLs has many advantages and therefore remains the recommended practice. However, for certain projects, such a requirement can turn out to be a true obstacle: Whenever the code to be tested is not exported from a DLL or part of a static library (LIB), accessing this code from within such a test DLL could become quite a challenge.

The fact that Visual Assert Beta 2 now fully supports EXE modules means the following: You can now place your unit tests wherever you think they fit best. Whether they are part of a DLL or an EXE, whether you create separate “unit test” projects or intermingle your test code with other code — it is now all up to you. Wherever you placed your tests, Visual Assert will find them and will provide a consistent user experience.

And the best part of the support for EXE modules is that it is totally non-intrusive: You do not have to change your main/WinMain function, much less any other code or build settings. And when run “outside” Visual Assert, i.e. launched directly or in the Visual Studio Debugger, the application will behave as normal.

Needless to say, the cfix 1.5 command line test runners, cfix32.exe and cfix64.exe also have been updated to properly deal with EXE modules.

The Wizard

Although neither the WinUnit API nor the cfix C and C++ API require much boilerplate code to be written, there still is some amount of code that more or less all unit tests share. Thanks to the new Wizard, you can now have Visual Assert generate this code for you. This really helps creating new fixtures more quickly!

Download, Try it, Share Your Opinion

Of course, using the new Beta version is free. So whether you are a full time tester or a unit testing sceptic, download the new release and try it by yourself. And of course, your feedback, both positive and negative, is always welcome and can be posted here.

Download Visual Assert Beta 2

cfix 1.5 released; adds support for EXE-embedded tests and kernel mode multi-threading

A new version of the cfix unit testing framework is now ready for download.

Unlike the previous release, which was mainly a maintenance release, cfix 1.5 adds major new features: kernel mode multithreading and EXE-based unit tests.

EXE based unit tests

As I discussed in a previous post in the context of Visual Assert/cfix studio, cfix’ restriction to DLL based unit tests has turned out to be quite a limitation for certain kinds of projects.

In cfix 1.5, this restriction has been removed: cfix now supports both, unit tests compiled and linked into DLL modules and unit tests embedded into EXE modules.

Now, when I say embeded into an EXE module, I do not mean that you may merely leave out the /LD compiler switch — it means that you may compile and link unit tests into the actual application EXE module without impacting this application’s behavior or having to change the application’s main() routine: When you run the application direcly or through the debugger, it will behave as normal (e.g. launch GUI). Once you launch it via cfix32 -exe app.exe (or cfix64 -exe app.exe, respectively), however, the application’s main() routine will not execute and instead, your embedded unit tests will run.

This may seem awkward at first — but it offers a tremendous advantage: All of the application’s code now immediately becomes testable (i.e. accessible by unit test code) without having to do complex reorganization of your build process or source tree layout (although it is still a very good idea to enhance the build process s.t. the unit tests are stripped in the RTM builds and to make cfix.dll a delayload DLL). For many projects, relying on this feature will therefore make adopting cfix and maybe even unit testing in general much easier.

In almost all regards, EXE-embedded unit tests behave the same as their DLL counterparts. They are, however, slightly less handy when it comes to debugging (unless you use Visual Assert, which will shield you from this). For this reason, and their greater flexibility in general, it should be noted though that DLL-based unit tests will remain the preferred choice.

Kernel mode multithreading

Since its first release, cfix has featured support for multi-threaded test cases. Multi-threaded test cases are tests which spawn child threads and — and this is the important point — both, the “main” thread and the child threads may trigger assertions (i.e. use CFIX_ASSERT and friends). Regardless of the thread an assertion occurs on, it will be recognized by the framework and will lead to the test case being marked as having failed. In case this sounds trivial to you, be informed that not even JUnit properly supports this :)

And while this feature has been supported for user mode tests ever since, the infrastructure for kernel mode unit tests, which was added in version 1.1, has lacked support for this feature: There was no kernel mode counterpart of CfixCreateThread and thus, only single-threaded kernel mode tests were supported.

cfix 1.5 now finally introduces CfixCreateSystemThread: CfixCreateSystemThread is basically a wrapper for PsCreateSystemThread with the added functionality of registering the child thread with cfix. Thus, all threads spawned using CfixCreateSystemThread (rather than using PsCreateSystemThread directly) are allowed to make use of assertions.

Kudos to Matt Oh for beta-testing this feature.

Minor enhacements

Another enhancement related to multi threaded tests is Auto-joining of child threads: After a test routine completes, the framework will now automatically check whether any child threads were created. If this is the case and any of these threads are still running, they will be waited on — not before all threads have terminated will the test run resume. This feature both makes writing multi-threaded tests more convenient (you do not have to wait by yourself) and safer (No more runaway child threads).

Finally, another feature worth noting is that by specifying the -td command line switch, cfix can be directed to not capture a stack backtrace when an assertion fails. Although stack backtraces are usually very helpful, the symbol loading can make their creation quite expensive. Using this switch can therefore speed up the usage of cfix.

Download/Upgrade Now

You see, there are many good reasons to upgrade your cfix installation or — in case you are not using cfix yet — to give cfix a try.

Better yet, check out the Visual Assert AddIn for Visual C++ — it is based on the new cfix 1.5 release and seamlessly integrates with Visual Studio.

Download cfix 1.5.0 Installer
Download cfix 1.5.0 source code

cfix studio Beta 2 to add support for EXE-based unit tests

N.B. cfix studio was the code name of what has become Visual Assert

The biggest shortcoming of the current cfix studio version certainly is that it requires all tests be implemented in a DLL. Conceptually, keeping test cases separated from the remaining code certainly is a good idea — and implementing tests in a DLL is a way to accomplish this. However, there are many projects in which such separation is either not feasible or just too much effort.

The good news is that with Beta 2, this will finally change: EXEs become first class-citizens in cfix studio and it will not matter any more whether your tests are part of a DLL or EXE project — you can just put them where you think is appropriate.

Take a classic MFC/GUI application project as an example: It is pretty common for these kinds of projects that most, if not all, application logic is part of a single Visual Studio project that compiles into a single EXE. There may be some additional DLLs or LIBs, but by and large, the EXE itself is where most of the interesting things happen.

The upcoming Beta 2 release now allows you to implement all your unit tests as part of the same EXE project. This means that your tests have access to all classes, functions and resources that are part of the project — all of which you would not easily have access to if you implemented the tests in a separate DLL.

Of course, embedding unit tests into an executable raises two questions:

  1. How to strip the tests from the final release?
  2. How on earth will you be able to run these tests without having main() create windows, load files, play sounds, etc each time?

Thankfully, C/C++ has a preprocessor and Visual C++ has the “exclude from build” feature which allows you to exclude certain files whenever the project is built using a specific configuration. Using any of these two features, the first question is easily answered.

The second problem is more tricky — but thankfully, it has already been solved for you: When cfix studio runs unit tests, it is well aware of that running main() might have, let’s say interesting effects — so what it does is simple: It just makes sure that main() is never run! Not only does this ensure that the tests run silently, i.e. without windows popping up etc, it also has the benefit that all unit tests “see” the application in a pristine state: Rather than having to worry about which state main() has brought the application into, you can initialize and clean up any state you need in your Setup and Teardown functions1.

To make a long story short: You can write unit tests in EXE projects in exactly the same manner as you would in a DLL project. No special considerations needed, no project settings that need to be changed, no additional boilerplate code to write. And when you run the EXE outside cfix studio, i.e. hit F5 in Visual Studio or launch the EXE directly, you will not even notice that the EXE houses some unit tests — everything works as normal.

Sounds good? Then wait a few more days and see it in action!

1: Needless to say, all global variables are initialized, constructors are run, etc. All CRT initialization happens as normal; only main()/WinMain() is not run. And yes, it also works for apps that link statically to MFC and therefore do not have a “regular” WinMain().


About me

Johannes Passing, M.Sc., living in Berlin, Germany.

Besides his consulting work, Johannes mainly focusses on Win32, COM, and NT kernel mode development, along with Java and .Net. He also is the author of cfix, a C/C++ unit testing framework for Win32 and NT kernel mode, Visual Assert, a Visual Studio Unit Testing-AddIn, and NTrace, a dynamic function boundary tracing toolkit for Windows NT/x86 kernel/user mode code.

Contact Johannes: jpassing (at) acm org

Johannes' GPG fingerprint is BBB1 1769 B82D CD07 D90A 57E8 9FE1 D441 F7A0 1BB1.

LinkedIn Profile
Xing Profile
Github Profile