Posts Tagged 'unit testing'

cfix 1.6 released, simplifies authoring of multi-threaded tests

A new release of cfix, the unit testing framework for C and C++, is now available for download. Besides some minor enhancements like extending the maximum permitted fixture name, cfix 1.6 introduces a major new feature, Anonymous Thread Auto-Registration.

Since its very first release, cfix has supported multi-threaded test cases, i.e. test cases that spawn child threads, each of which potentially making use of the various assertion statements like CFIX_ASSERT. To make this work and ensure that failing assertions are handled properly, however, usage of CfixCreateThread (rather than the native Win32 CreateThread) was mandatory when spawning such threads.

Although using CfixCreateThread (or CfixCreateSystemThread in case of kernel mode code) remains the preferred way to create child threads, there are situations where usage of this API is not possible — for example, when the child threads are created by libraries such as boost.

To support such scenarios, cfix now allows you to annotate a fixture to enable Anonymous Thread Auto-Registration, which means that newly spawned threads are automatically registered with cfix — no usage of CfixCreateThread required. Thanks to this feature, writing multi-threaded tests becomes straightforward — and integrating with libraries such as boost does not pose a problem any more.

For more details about this feature, please refer to the respective section in the cfix documentation.

As always, updated binaries and source code are available on Sourceforge.

Advertisements

How to customize test run execution to make your cfix test runs more effective

One of the features introduced back in cfix 1.2 was the ability to customize test execution with the command line parameters -fsr and -fsf. Using these switches can make your test runs more effective and can help simplify debugging — so it is worth spending a minute on this topic.

Assume our test run comprises two fixtures, Fixture A and Fixture B. As fixtures are always run in alphabetic order, and tests run in the order they are defined, the first test to be executed is Test 1 of Fixture A, followed by Test 2 of Fixture A, and so on. Assuming all tests of Fixture A succeed, execution then proceeds with the first test of Fixture B. The following figure illustrates this:

Successful execution of a unit testing suite

Things get interesting when one of the tests fails: Let us assume Test 2 of Fixture A leads to a failed assertion. The default behavior of cfix, which equals the behavior of JUnit and NUnit, is to report the failure and proceed with Test 3 of the same fixture:

Execution resumes after a failed unit test

In many cases, this is the appropriate thing to do — letting the test run to completion and figure out the reasons for the failure afterwards. But in certain scenarios, resuming with Test 3 might not be the smartest thing to do — rather, as soon as a test fails, execution should proceed with the next fixture. This is what -fsf (short-circuit fixture”) does:

Short-circuiting the fixture

Once a test case has failed, all remaining tests of the same fixture are skipped.

When debugging, even resuming at the next fixture is often not desired. Rather, as soon as one test fails, you may immediately want to take a closer a look at the failure, so resuming the run is mute. -fsr (short-circuit run”) does right that: It aborts the run immediately.

Short-circuiting the run

By the way — in Visual Assert, you can quickly access these options in the Options menu in the Test Explorer window:

Visual Assert Test Explorer

Visual Assert Beta 3 released

A third beta release of Visual Assert is now available for download on www.visualassert.com.

Visual Assert, in case you have not tried it yet, is an Add-In for Visual Studio that adds unit testing capabilities to the Visual C++ IDE: Based on the cfix unit testing framework, Visual Assert allows unit tests to be written, run, and debugged from within the IDE. Pretty much like Junit/Eclipse, TestDriven.Net or MSTest, but for real, native code — code written in C or C++.

Bugs, bugs, bugs, bugs

Alas, there were still a few of them in the previous two beta releases. Luckily though, almost all I received from users or found by internal testing were relatively minor in nature. Still, I want Visual Assert to be as high quality as possible and decided to add this third beta release into the schedule and take the time to focus on — you guessed it — bugfixing, bugfixing, bugfixing, and bugfixing.

Speaking of bug reports, I have to thank all users of Visual Assert and cfix who reported bugs, suggested new features or provided general feedback. Your input has been, and still is highly appreciated. Although I had to postpone any feature suggestions to a later release, I tried hard to resolve all bugs and have them fixed in this new release.

Download, Try it, Share Your Opinion

Of course, using the new Beta version is free. So whether you have already used the previous beta or not, whether you are a unit testing newbie or write unit tests on a daily basis, be sure to give the new version a try. And of course, do not forget to let me know about your feedback, suggestions, found bugs, etc.!

Visual Assert Beta 2 Released

The Beta 2 release of Visual Assert (formerly named cfix studio) is now available for download. The release marks a major step in the development of Visual Assert for that it not only comprises a number of bugfixes but also introduces major new features. The two most important certainly are support for EXE targets and Wizard assistance.

Support for EXE Targets

As announced in a previous post and also discussed in the post about the cfix 1.5 release, Visual Assert now fully supports unit tets emedded in EXE modules.

Previous releases required all unit tests to be compiled and linked into DLLs. In fact, the usage of DLLs has many advantages and therefore remains the recommended practice. However, for certain projects, such a requirement can turn out to be a true obstacle: Whenever the code to be tested is not exported from a DLL or part of a static library (LIB), accessing this code from within such a test DLL could become quite a challenge.

The fact that Visual Assert Beta 2 now fully supports EXE modules means the following: You can now place your unit tests wherever you think they fit best. Whether they are part of a DLL or an EXE, whether you create separate “unit test” projects or intermingle your test code with other code — it is now all up to you. Wherever you placed your tests, Visual Assert will find them and will provide a consistent user experience.

And the best part of the support for EXE modules is that it is totally non-intrusive: You do not have to change your main/WinMain function, much less any other code or build settings. And when run “outside” Visual Assert, i.e. launched directly or in the Visual Studio Debugger, the application will behave as normal.

Needless to say, the cfix 1.5 command line test runners, cfix32.exe and cfix64.exe also have been updated to properly deal with EXE modules.

The Wizard

Although neither the WinUnit API nor the cfix C and C++ API require much boilerplate code to be written, there still is some amount of code that more or less all unit tests share. Thanks to the new Wizard, you can now have Visual Assert generate this code for you. This really helps creating new fixtures more quickly!

Download, Try it, Share Your Opinion

Of course, using the new Beta version is free. So whether you are a full time tester or a unit testing sceptic, download the new release and try it by yourself. And of course, your feedback, both positive and negative, is always welcome and can be posted here.

Download Visual Assert Beta 2

cfix 1.5 released; adds support for EXE-embedded tests and kernel mode multi-threading

A new version of the cfix unit testing framework is now ready for download.

Unlike the previous release, which was mainly a maintenance release, cfix 1.5 adds major new features: kernel mode multithreading and EXE-based unit tests.

EXE based unit tests

As I discussed in a previous post in the context of Visual Assert/cfix studio, cfix’ restriction to DLL based unit tests has turned out to be quite a limitation for certain kinds of projects.

In cfix 1.5, this restriction has been removed: cfix now supports both, unit tests compiled and linked into DLL modules and unit tests embedded into EXE modules.

Now, when I say embeded into an EXE module, I do not mean that you may merely leave out the /LD compiler switch — it means that you may compile and link unit tests into the actual application EXE module without impacting this application’s behavior or having to change the application’s main() routine: When you run the application direcly or through the debugger, it will behave as normal (e.g. launch GUI). Once you launch it via cfix32 -exe app.exe (or cfix64 -exe app.exe, respectively), however, the application’s main() routine will not execute and instead, your embedded unit tests will run.

This may seem awkward at first — but it offers a tremendous advantage: All of the application’s code now immediately becomes testable (i.e. accessible by unit test code) without having to do complex reorganization of your build process or source tree layout (although it is still a very good idea to enhance the build process s.t. the unit tests are stripped in the RTM builds and to make cfix.dll a delayload DLL). For many projects, relying on this feature will therefore make adopting cfix and maybe even unit testing in general much easier.

In almost all regards, EXE-embedded unit tests behave the same as their DLL counterparts. They are, however, slightly less handy when it comes to debugging (unless you use Visual Assert, which will shield you from this). For this reason, and their greater flexibility in general, it should be noted though that DLL-based unit tests will remain the preferred choice.

Kernel mode multithreading

Since its first release, cfix has featured support for multi-threaded test cases. Multi-threaded test cases are tests which spawn child threads and — and this is the important point — both, the “main” thread and the child threads may trigger assertions (i.e. use CFIX_ASSERT and friends). Regardless of the thread an assertion occurs on, it will be recognized by the framework and will lead to the test case being marked as having failed. In case this sounds trivial to you, be informed that not even JUnit properly supports this :)

And while this feature has been supported for user mode tests ever since, the infrastructure for kernel mode unit tests, which was added in version 1.1, has lacked support for this feature: There was no kernel mode counterpart of CfixCreateThread and thus, only single-threaded kernel mode tests were supported.

cfix 1.5 now finally introduces CfixCreateSystemThread: CfixCreateSystemThread is basically a wrapper for PsCreateSystemThread with the added functionality of registering the child thread with cfix. Thus, all threads spawned using CfixCreateSystemThread (rather than using PsCreateSystemThread directly) are allowed to make use of assertions.

Kudos to Matt Oh for beta-testing this feature.

Minor enhacements

Another enhancement related to multi threaded tests is Auto-joining of child threads: After a test routine completes, the framework will now automatically check whether any child threads were created. If this is the case and any of these threads are still running, they will be waited on — not before all threads have terminated will the test run resume. This feature both makes writing multi-threaded tests more convenient (you do not have to wait by yourself) and safer (No more runaway child threads).

Finally, another feature worth noting is that by specifying the -td command line switch, cfix can be directed to not capture a stack backtrace when an assertion fails. Although stack backtraces are usually very helpful, the symbol loading can make their creation quite expensive. Using this switch can therefore speed up the usage of cfix.

Download/Upgrade Now

You see, there are many good reasons to upgrade your cfix installation or — in case you are not using cfix yet — to give cfix a try.

Better yet, check out the Visual Assert AddIn for Visual C++ — it is based on the new cfix 1.5 release and seamlessly integrates with Visual Studio.

Download cfix 1.5.0 Installer
Download cfix 1.5.0 source code

Vote for cfix

The Automated Testing Institute has elected cfix to be one of the finalists for the Autmation Honors award. The winners of the award will be highlighted in a Special December Edition of the Automated Software Testing Magazine.

If you are a cfix user, be sure to vote for cfix here.

And by the way, I think The Grinder, which is a really neat web performance testing framework, also deserves being voted for…

cfix 1.3.0 Released, Introducing WinUnit Compatibility

cfix 1.3, the latest version of the unit testing framework for C/C++ on Windows, has just been released. As announced in the last blog post, the major new feature of this release is WinUnit compatibility, i.e. the ability to recompile existing WinUnit test suites into cfix test suites without having to change a single line of code.

To demonstrate that this compatibility indeed works, consider the following simple example:

#include "WinUnit.h"

BEGIN_TEST(DummyTest)
{
  WIN_ASSERT_STRING_EQUAL( "foo", "bar" "Descriptive message");
}
END_TEST

Compile it:

cl /I %CFIX_HOME%\include /LD /EHa /Zi winunittest.cpp /link /LIBPATH:%CFIX_HOME%\lib\i386

Or, in case %INCLUDE% and %LIB% already happen to be set properly:

cl /LD /EHa winunittest.cpp

Note that the only difference to compiling the test for WinUnit ist that a different include path is used — rather than the original winunit.h, cfix’ own winunit.h is used, which in turn implements the WinUnit API on top of the existing API.

The resulting DLL is a valid cfix DLL and its tests can be run in the usual manner. As the example contains a failing test, cfix will print the stack trace and error description to the console:

D:\sample>cfix32 -ts -z winunittest.dll
cfix version 1.3.0.3340 (fre)
(C) 2008-2009 - Johannes Passing - http://www.cfix-testing.org/
[Failure]      winunittest.DummyTest.DummyTest
      winunittest.cpp(5): DummyTest

      Expression: Descriptive message: [foo] == [bar] (Expression: "foo" == "bar")
      Last Error: 0 (The operation completed successfully. )

      cfix!CfixpCaptureStackTrace +0x40
      cfix!CfixPeReportFailedAssertion +0xd2
      winunittest!cfixcc::Assertion::Fail<std::...
      winunittest!cfixcc::Assertion::Relate<std...
      winunittest!cfixcc::Assertion::Relate ...
      winunittest!cfixcc::Assertion::RelateStri...
      winunittest!DummyTest +0x9c
      cfix!CfixsRunTestRoutine +0x33
      cfix!CfixsRunTestCaseMethod +0x27
      cfix!CfixsRunTestCase +0x25
      ...

Of course, cfix also supports WinUnit fixtures, as the following example, taken from the original WinUnit article on MSDN demonstrates:

#include "WinUnit.h"
#include <windows.h>

// Fixture must be declared.
FIXTURE(DeleteFileFixture);

namespace
{
  TCHAR s_tempFileName[MAX_PATH] = _T("");
  bool IsFileValid(TCHAR* fileName);
}

// Both SETUP and TEARDOWN must be present. 
SETUP(DeleteFileFixture)
{
  // This is the maximum size of the directory passed to GetTempFileName.
  const unsigned int maxTempPath = MAX_PATH - 14; 
  TCHAR tempPath[maxTempPath + 1] = _T("");
  DWORD charsWritten = GetTempPath(maxTempPath + 1, tempPath);
  // (charsWritten does not include null character)
  WIN_ASSERT_TRUE(charsWritten  0, 
    _T("GetTempPath failed."));

  // Create a temporary file
  UINT tempFileNumber = GetTempFileName(tempPath, _T("WUT"), 
    0, // This means the file will get created and closed.
    s_tempFileName);

  // Make sure that the file actually exists
  WIN_ASSERT_WINAPI_SUCCESS(IsFileValid(s_tempFileName), 
    _T("File %s is invalid or does not exist."), s_tempFileName);
}

// TEARDOWN does the inverse of SETUP, as well as undoing 
// any side effects the tests could have caused.
TEARDOWN(DeleteFileFixture)
{
  // Delete the temp file if it still exists.
  if (IsFileValid(s_tempFileName))
  {
    // Ensure file is not read-only
    DWORD fileAttributes = GetFileAttributes(s_tempFileName);
    if (fileAttributes & FILE_ATTRIBUTE_READONLY)
    {
      WIN_ASSERT_WINAPI_SUCCESS(
        SetFileAttributes(s_tempFileName, 
          fileAttributes ^ FILE_ATTRIBUTE_READONLY),
          _T("Unable to undo read-only attribute of file %s."),
          s_tempFileName);
    }

    // Since I'm testing DeleteFile, I use the alternative CRT file
    // deletion function in my cleanup.
    WIN_ASSERT_ZERO(_tremove(s_tempFileName), 
      _T("Unable to delete file %s."), s_tempFileName);
  }

  // Clear the temp file name.
  ZeroMemory(s_tempFileName, 
    ARRAYSIZE(s_tempFileName) * sizeof(s_tempFileName[0]));
}

BEGIN_TESTF(DeleteFileShouldDeleteFileIfNotReadOnly, DeleteFileFixture)
{
  WIN_ASSERT_WINAPI_SUCCESS(DeleteFile(s_tempFileName));
  WIN_ASSERT_FALSE(IsFileValid(s_tempFileName), 
    _T("DeleteFile did not delete %s correctly."),
    s_tempFileName);
}
END_TESTF

BEGIN_TESTF(DeleteFileShouldFailIfFileIsReadOnly, DeleteFileFixture)
{
  // Set file to read-only
  DWORD fileAttributes = GetFileAttributes(s_tempFileName);
  WIN_ASSERT_WINAPI_SUCCESS(
    SetFileAttributes(s_tempFileName, 
    fileAttributes | FILE_ATTRIBUTE_READONLY));

  // Verify that DeleteFile fails with ERROR_ACCESS_DENIED
  // (according to spec)
  WIN_ASSERT_FALSE(DeleteFile(s_tempFileName));
  WIN_ASSERT_EQUAL(ERROR_ACCESS_DENIED, GetLastError());
}
END_TESTF

namespace
{
  bool IsFileValid(TCHAR* fileName)
  {
    return (GetFileAttributes(fileName) != INVALID_FILE_ATTRIBUTES);
  }
}

Compiling and running this test yields the expected output:

d:\sample>cl /I %CFIX_HOME%\include /LD /EHa /Zi fixture.cpp /link /LIBPATH:%CFIX_HOME%\lib\i386
d:\sample>cfix32 -ts -z fixture.dll
cfix version 1.3.0.3340 (fre)
(C) 2008-2009 - Johannes Passing - http://www.cfix-testing.org/
[Success]      fixture.DeleteFileFixture.DeleteFileShouldDeleteFileIfNotReadOnly
[Success]      fixture.DeleteFileFixture.DeleteFileShouldFailIfFileIsReadOnly


       1 Fixtures
       2 Test cases
           2 succeeded
           0 failed
           0 inconclusive

Limitations

All compatibility has its limitations — although cfix supports all major WinUnit constructs and assertions, there are a small number of known limitations, which are listed in the documentation. And although I am confident that most WinUnit code should compile and run just fine, it is, of course, possible, that further limitations pop up. In such cases, I would welcome an appropriate bug report and will try to fix cfix accordingly.

Documentation

In order to have cfix be a fully adequate replacement for cfix, the cfix documentation has additionally been augmented to include a documentation of the entire WinUnit API.

Technical background

Technically, implementing the compatibility layer went rather smoothly. On the one hand, WinUnit and cfix have similar architectures, which makes many things easier. On the other hand, WinUnit has a clean, single public header file (contrast that to CppUnit!), which also simplified things. And as WinUnit is limited to C++, I was able to use C++ templates (in combination with some preprocessor macros) to implement the entire WinUnit compatibility layer without having to change a single line of cfix itself. Rather, the WinUnit macros/classes are all mapped onto the existing cfix C++ API, which already includes most of what was neccessary to implement the WinUnit functionality.

Conclusion

In case have been using WinUnit the past and have a set of existing WinUnit-based test suites, give cfix a try — Not only should it be a full-featured replacement for WinUnit, you can also expect to see, and benefit from new features in upcoming releases!

Last but not least, the release contains a number of minor bugfixes. So upgrading is recommended even if you do not intend to use the new WinUnit compatibility feature.

cfix can be downloaded here.


Categories




About me

Johannes Passing, M.Sc., living in Berlin, Germany.

Besides his consulting work, Johannes mainly focusses on Win32, COM, and NT kernel mode development, along with Java and .Net. He also is the author of cfix, a C/C++ unit testing framework for Win32 and NT kernel mode, Visual Assert, a Visual Studio Unit Testing-AddIn, and NTrace, a dynamic function boundary tracing toolkit for Windows NT/x86 kernel/user mode code.

Contact Johannes: jpassing (at) acm org

Johannes' GPG fingerprint is BBB1 1769 B82D CD07 D90A 57E8 9FE1 D441 F7A0 1BB1.

LinkedIn Profile
Xing Profile
Github Profile
Advertisements