Archive for the 'Testing' Category



Visual Assert Beta 2 Released

The Beta 2 release of Visual Assert (formerly named cfix studio) is now available for download. The release marks a major step in the development of Visual Assert for that it not only comprises a number of bugfixes but also introduces major new features. The two most important certainly are support for EXE targets and Wizard assistance.

Support for EXE Targets

As announced in a previous post and also discussed in the post about the cfix 1.5 release, Visual Assert now fully supports unit tets emedded in EXE modules.

Previous releases required all unit tests to be compiled and linked into DLLs. In fact, the usage of DLLs has many advantages and therefore remains the recommended practice. However, for certain projects, such a requirement can turn out to be a true obstacle: Whenever the code to be tested is not exported from a DLL or part of a static library (LIB), accessing this code from within such a test DLL could become quite a challenge.

The fact that Visual Assert Beta 2 now fully supports EXE modules means the following: You can now place your unit tests wherever you think they fit best. Whether they are part of a DLL or an EXE, whether you create separate “unit test” projects or intermingle your test code with other code — it is now all up to you. Wherever you placed your tests, Visual Assert will find them and will provide a consistent user experience.

And the best part of the support for EXE modules is that it is totally non-intrusive: You do not have to change your main/WinMain function, much less any other code or build settings. And when run “outside” Visual Assert, i.e. launched directly or in the Visual Studio Debugger, the application will behave as normal.

Needless to say, the cfix 1.5 command line test runners, cfix32.exe and cfix64.exe also have been updated to properly deal with EXE modules.

The Wizard

Although neither the WinUnit API nor the cfix C and C++ API require much boilerplate code to be written, there still is some amount of code that more or less all unit tests share. Thanks to the new Wizard, you can now have Visual Assert generate this code for you. This really helps creating new fixtures more quickly!

Download, Try it, Share Your Opinion

Of course, using the new Beta version is free. So whether you are a full time tester or a unit testing sceptic, download the new release and try it by yourself. And of course, your feedback, both positive and negative, is always welcome and can be posted here.

Download Visual Assert Beta 2

Advertisements

cfix 1.5 released; adds support for EXE-embedded tests and kernel mode multi-threading

A new version of the cfix unit testing framework is now ready for download.

Unlike the previous release, which was mainly a maintenance release, cfix 1.5 adds major new features: kernel mode multithreading and EXE-based unit tests.

EXE based unit tests

As I discussed in a previous post in the context of Visual Assert/cfix studio, cfix’ restriction to DLL based unit tests has turned out to be quite a limitation for certain kinds of projects.

In cfix 1.5, this restriction has been removed: cfix now supports both, unit tests compiled and linked into DLL modules and unit tests embedded into EXE modules.

Now, when I say embeded into an EXE module, I do not mean that you may merely leave out the /LD compiler switch — it means that you may compile and link unit tests into the actual application EXE module without impacting this application’s behavior or having to change the application’s main() routine: When you run the application direcly or through the debugger, it will behave as normal (e.g. launch GUI). Once you launch it via cfix32 -exe app.exe (or cfix64 -exe app.exe, respectively), however, the application’s main() routine will not execute and instead, your embedded unit tests will run.

This may seem awkward at first — but it offers a tremendous advantage: All of the application’s code now immediately becomes testable (i.e. accessible by unit test code) without having to do complex reorganization of your build process or source tree layout (although it is still a very good idea to enhance the build process s.t. the unit tests are stripped in the RTM builds and to make cfix.dll a delayload DLL). For many projects, relying on this feature will therefore make adopting cfix and maybe even unit testing in general much easier.

In almost all regards, EXE-embedded unit tests behave the same as their DLL counterparts. They are, however, slightly less handy when it comes to debugging (unless you use Visual Assert, which will shield you from this). For this reason, and their greater flexibility in general, it should be noted though that DLL-based unit tests will remain the preferred choice.

Kernel mode multithreading

Since its first release, cfix has featured support for multi-threaded test cases. Multi-threaded test cases are tests which spawn child threads and — and this is the important point — both, the “main” thread and the child threads may trigger assertions (i.e. use CFIX_ASSERT and friends). Regardless of the thread an assertion occurs on, it will be recognized by the framework and will lead to the test case being marked as having failed. In case this sounds trivial to you, be informed that not even JUnit properly supports this :)

And while this feature has been supported for user mode tests ever since, the infrastructure for kernel mode unit tests, which was added in version 1.1, has lacked support for this feature: There was no kernel mode counterpart of CfixCreateThread and thus, only single-threaded kernel mode tests were supported.

cfix 1.5 now finally introduces CfixCreateSystemThread: CfixCreateSystemThread is basically a wrapper for PsCreateSystemThread with the added functionality of registering the child thread with cfix. Thus, all threads spawned using CfixCreateSystemThread (rather than using PsCreateSystemThread directly) are allowed to make use of assertions.

Kudos to Matt Oh for beta-testing this feature.

Minor enhacements

Another enhancement related to multi threaded tests is Auto-joining of child threads: After a test routine completes, the framework will now automatically check whether any child threads were created. If this is the case and any of these threads are still running, they will be waited on — not before all threads have terminated will the test run resume. This feature both makes writing multi-threaded tests more convenient (you do not have to wait by yourself) and safer (No more runaway child threads).

Finally, another feature worth noting is that by specifying the -td command line switch, cfix can be directed to not capture a stack backtrace when an assertion fails. Although stack backtraces are usually very helpful, the symbol loading can make their creation quite expensive. Using this switch can therefore speed up the usage of cfix.

Download/Upgrade Now

You see, there are many good reasons to upgrade your cfix installation or — in case you are not using cfix yet — to give cfix a try.

Better yet, check out the Visual Assert AddIn for Visual C++ — it is based on the new cfix 1.5 release and seamlessly integrates with Visual Studio.

Download cfix 1.5.0 Installer
Download cfix 1.5.0 source code

cfix studio renamed to Visual Assert

Back when I began thinking about creating a Visual Studio Add-In for cfix, I needed a code name for the project. After tentatively using the name cfix+ for a while, I quickly settled on cfix studio — given that the project revolved around cfix and Visual Studio, this name pretty much suggested itself.

Soon after going into Beta, however, I had to realize that this name was not without its problems. Most importantly, it makes it hard for users to properly differentiate between cfix and cfix studio. This obviously led to situations where people were not quite sure whether cfix studio is a supplement to, replacement of, or just new version of cfix.

I would not care too much about this ambiguity if the two projects were not very different in terms of licensing: While using the Add-In will require a license to be purchased once it leaves the beta status While the Add-In is freeware (but not open source), the underlying cfix framework is, and will always remain open source and be licensed under the quite permissive LGPL.

As all APIs and libraries unit tests link against are part of the cfix framework, and the cfix framework itself is self-contained, this means that despite the Add-In being closed source, you still get the benefits of open source: Most importantly, there is no lock-in — you can stop using the Add-In and switch back to the command line tools at any time and be all-open source again. You are even free to create a fork of cfix at any time — there really is nothing other than convenience that binds you to using the Add-In.

Given that a growing number of people indeed tends to object to using closed source APIs and fears such vendor lock-in, I consider it important to stress this open source nature of the cfix framework underlying the Add-In.

However, at this point it should also become clear that a name that blurs the distinction between the two projects is counterproductive.

Based on this insight, I opted for dropping the cfix studio name and replacing it by something different: Visual Assert. This name should emphasise that the Add-In may be based on, but really is separate from the cfix framework.

Visual Assert

The new name will be used beginning with the upcoming Beta 2 release.

cfix studio Beta 2 to add support for EXE-based unit tests

N.B. cfix studio was the code name of what has become Visual Assert

The biggest shortcoming of the current cfix studio version certainly is that it requires all tests be implemented in a DLL. Conceptually, keeping test cases separated from the remaining code certainly is a good idea — and implementing tests in a DLL is a way to accomplish this. However, there are many projects in which such separation is either not feasible or just too much effort.

The good news is that with Beta 2, this will finally change: EXEs become first class-citizens in cfix studio and it will not matter any more whether your tests are part of a DLL or EXE project — you can just put them where you think is appropriate.

Take a classic MFC/GUI application project as an example: It is pretty common for these kinds of projects that most, if not all, application logic is part of a single Visual Studio project that compiles into a single EXE. There may be some additional DLLs or LIBs, but by and large, the EXE itself is where most of the interesting things happen.

The upcoming Beta 2 release now allows you to implement all your unit tests as part of the same EXE project. This means that your tests have access to all classes, functions and resources that are part of the project — all of which you would not easily have access to if you implemented the tests in a separate DLL.

Of course, embedding unit tests into an executable raises two questions:

  1. How to strip the tests from the final release?
  2. How on earth will you be able to run these tests without having main() create windows, load files, play sounds, etc each time?

Thankfully, C/C++ has a preprocessor and Visual C++ has the “exclude from build” feature which allows you to exclude certain files whenever the project is built using a specific configuration. Using any of these two features, the first question is easily answered.

The second problem is more tricky — but thankfully, it has already been solved for you: When cfix studio runs unit tests, it is well aware of that running main() might have, let’s say interesting effects — so what it does is simple: It just makes sure that main() is never run! Not only does this ensure that the tests run silently, i.e. without windows popping up etc, it also has the benefit that all unit tests “see” the application in a pristine state: Rather than having to worry about which state main() has brought the application into, you can initialize and clean up any state you need in your Setup and Teardown functions1.

To make a long story short: You can write unit tests in EXE projects in exactly the same manner as you would in a DLL project. No special considerations needed, no project settings that need to be changed, no additional boilerplate code to write. And when you run the EXE outside cfix studio, i.e. hit F5 in Visual Studio or launch the EXE directly, you will not even notice that the EXE houses some unit tests — everything works as normal.

Sounds good? Then wait a few more days and see it in action!

Remarks
1: Needless to say, all global variables are initialized, constructors are run, etc. All CRT initialization happens as normal; only main()/WinMain() is not run. And yes, it also works for apps that link statically to MFC and therefore do not have a “regular” WinMain().

Introducing cfix studio, the Visual Studio AddIn for C/C++ Unit Testing

N.B. cfix studio was the code name of what has become Visual Assert

There is little doubt that native code, and C and C++ in particular, is here to stay. And still, it is pretty obvious that when it comes to tools and IDEs, it is the managed world that has gotten most attention from tool vendors over the past years.

While there are lots and lots of useful tools for native development, many of them probably even better than their managed counterparts, there are some areas where the managed language fraction is far ahead: One of these areas certainly is IDE support for unit testing.

JUnit for Eclipse, TestDriven.Net for Visual Studio and MS Test make test-driven development so much more convenient and efficient that it is almost ridiculous that using command line tools is still state of the art for C/C++ development.

That said, there is great news: With cfix studio, there finally is a solution filling in this gap! cfix studio, based on the cfix unit testing framework, is a Visual Studio-AddIn that allows you to easily write, manage, run, and debug your unit tests from within Visual Studio. No fiddling with command line tools, complex configuration, or boilerplate code required!

Among lots of other features, cfix studio also has first-class support for multi-architecture development – you can easily switch back and forth between 32-bit and 64-bit and can even mix tests of different architectures in a single test run. Needless to say, cfix studio, like cfix, is also fully compatible to WinUnit.

If that has caught your interest, you are invited to check out the first beta version of cfix studio:

Download cfix studio Beta 1
Version 1.0.0.3458

It is free, quick to install and comes with a set of example projects. Give it a try — and please let me know about all your crticism, suggestions and other feedback!

Here are some screenshots of cfix studio in action:

Test Explorer
The Test Explorer allows you to start a single or set of tests, the Run Window shows the results

Run Window
Run Window: Viewing test progress

Failed Assertion
Run Window: Viewing test results and details of a failed assertion

Debgging a failed assertion
When running in the debugger, a failed assertion will hit a breakpoint and the Run Window will show additional details

cfix 1.4 released

Today, a new version of cfix, the open source unit testing framework for user and kernel mode C and C++, has been released. cfix 1.4, in addition to the existing feature of allowing test runs to be restricted to specific fixtures, now also allows single testcases to be run in isolation, which can be a great aid in debugging. Besides several minor fixes, the cfix API has been slightly enhanced and cfix now degrades more gracefully in case of dbghelp-issues.

Updated cfix binaries and source code are now available for download

cfix 1.3.0 Released, Introducing WinUnit Compatibility

cfix 1.3, the latest version of the unit testing framework for C/C++ on Windows, has just been released. As announced in the last blog post, the major new feature of this release is WinUnit compatibility, i.e. the ability to recompile existing WinUnit test suites into cfix test suites without having to change a single line of code.

To demonstrate that this compatibility indeed works, consider the following simple example:

#include "WinUnit.h"

BEGIN_TEST(DummyTest)
{
  WIN_ASSERT_STRING_EQUAL( "foo", "bar" "Descriptive message");
}
END_TEST

Compile it:

cl /I %CFIX_HOME%\include /LD /EHa /Zi winunittest.cpp /link /LIBPATH:%CFIX_HOME%\lib\i386

Or, in case %INCLUDE% and %LIB% already happen to be set properly:

cl /LD /EHa winunittest.cpp

Note that the only difference to compiling the test for WinUnit ist that a different include path is used — rather than the original winunit.h, cfix’ own winunit.h is used, which in turn implements the WinUnit API on top of the existing API.

The resulting DLL is a valid cfix DLL and its tests can be run in the usual manner. As the example contains a failing test, cfix will print the stack trace and error description to the console:

D:\sample>cfix32 -ts -z winunittest.dll
cfix version 1.3.0.3340 (fre)
(C) 2008-2009 - Johannes Passing - http://www.cfix-testing.org/
[Failure]      winunittest.DummyTest.DummyTest
      winunittest.cpp(5): DummyTest

      Expression: Descriptive message: [foo] == [bar] (Expression: "foo" == "bar")
      Last Error: 0 (The operation completed successfully. )

      cfix!CfixpCaptureStackTrace +0x40
      cfix!CfixPeReportFailedAssertion +0xd2
      winunittest!cfixcc::Assertion::Fail<std::...
      winunittest!cfixcc::Assertion::Relate<std...
      winunittest!cfixcc::Assertion::Relate ...
      winunittest!cfixcc::Assertion::RelateStri...
      winunittest!DummyTest +0x9c
      cfix!CfixsRunTestRoutine +0x33
      cfix!CfixsRunTestCaseMethod +0x27
      cfix!CfixsRunTestCase +0x25
      ...

Of course, cfix also supports WinUnit fixtures, as the following example, taken from the original WinUnit article on MSDN demonstrates:

#include "WinUnit.h"
#include <windows.h>

// Fixture must be declared.
FIXTURE(DeleteFileFixture);

namespace
{
  TCHAR s_tempFileName[MAX_PATH] = _T("");
  bool IsFileValid(TCHAR* fileName);
}

// Both SETUP and TEARDOWN must be present. 
SETUP(DeleteFileFixture)
{
  // This is the maximum size of the directory passed to GetTempFileName.
  const unsigned int maxTempPath = MAX_PATH - 14; 
  TCHAR tempPath[maxTempPath + 1] = _T("");
  DWORD charsWritten = GetTempPath(maxTempPath + 1, tempPath);
  // (charsWritten does not include null character)
  WIN_ASSERT_TRUE(charsWritten  0, 
    _T("GetTempPath failed."));

  // Create a temporary file
  UINT tempFileNumber = GetTempFileName(tempPath, _T("WUT"), 
    0, // This means the file will get created and closed.
    s_tempFileName);

  // Make sure that the file actually exists
  WIN_ASSERT_WINAPI_SUCCESS(IsFileValid(s_tempFileName), 
    _T("File %s is invalid or does not exist."), s_tempFileName);
}

// TEARDOWN does the inverse of SETUP, as well as undoing 
// any side effects the tests could have caused.
TEARDOWN(DeleteFileFixture)
{
  // Delete the temp file if it still exists.
  if (IsFileValid(s_tempFileName))
  {
    // Ensure file is not read-only
    DWORD fileAttributes = GetFileAttributes(s_tempFileName);
    if (fileAttributes & FILE_ATTRIBUTE_READONLY)
    {
      WIN_ASSERT_WINAPI_SUCCESS(
        SetFileAttributes(s_tempFileName, 
          fileAttributes ^ FILE_ATTRIBUTE_READONLY),
          _T("Unable to undo read-only attribute of file %s."),
          s_tempFileName);
    }

    // Since I'm testing DeleteFile, I use the alternative CRT file
    // deletion function in my cleanup.
    WIN_ASSERT_ZERO(_tremove(s_tempFileName), 
      _T("Unable to delete file %s."), s_tempFileName);
  }

  // Clear the temp file name.
  ZeroMemory(s_tempFileName, 
    ARRAYSIZE(s_tempFileName) * sizeof(s_tempFileName[0]));
}

BEGIN_TESTF(DeleteFileShouldDeleteFileIfNotReadOnly, DeleteFileFixture)
{
  WIN_ASSERT_WINAPI_SUCCESS(DeleteFile(s_tempFileName));
  WIN_ASSERT_FALSE(IsFileValid(s_tempFileName), 
    _T("DeleteFile did not delete %s correctly."),
    s_tempFileName);
}
END_TESTF

BEGIN_TESTF(DeleteFileShouldFailIfFileIsReadOnly, DeleteFileFixture)
{
  // Set file to read-only
  DWORD fileAttributes = GetFileAttributes(s_tempFileName);
  WIN_ASSERT_WINAPI_SUCCESS(
    SetFileAttributes(s_tempFileName, 
    fileAttributes | FILE_ATTRIBUTE_READONLY));

  // Verify that DeleteFile fails with ERROR_ACCESS_DENIED
  // (according to spec)
  WIN_ASSERT_FALSE(DeleteFile(s_tempFileName));
  WIN_ASSERT_EQUAL(ERROR_ACCESS_DENIED, GetLastError());
}
END_TESTF

namespace
{
  bool IsFileValid(TCHAR* fileName)
  {
    return (GetFileAttributes(fileName) != INVALID_FILE_ATTRIBUTES);
  }
}

Compiling and running this test yields the expected output:

d:\sample>cl /I %CFIX_HOME%\include /LD /EHa /Zi fixture.cpp /link /LIBPATH:%CFIX_HOME%\lib\i386
d:\sample>cfix32 -ts -z fixture.dll
cfix version 1.3.0.3340 (fre)
(C) 2008-2009 - Johannes Passing - http://www.cfix-testing.org/
[Success]      fixture.DeleteFileFixture.DeleteFileShouldDeleteFileIfNotReadOnly
[Success]      fixture.DeleteFileFixture.DeleteFileShouldFailIfFileIsReadOnly


       1 Fixtures
       2 Test cases
           2 succeeded
           0 failed
           0 inconclusive

Limitations

All compatibility has its limitations — although cfix supports all major WinUnit constructs and assertions, there are a small number of known limitations, which are listed in the documentation. And although I am confident that most WinUnit code should compile and run just fine, it is, of course, possible, that further limitations pop up. In such cases, I would welcome an appropriate bug report and will try to fix cfix accordingly.

Documentation

In order to have cfix be a fully adequate replacement for cfix, the cfix documentation has additionally been augmented to include a documentation of the entire WinUnit API.

Technical background

Technically, implementing the compatibility layer went rather smoothly. On the one hand, WinUnit and cfix have similar architectures, which makes many things easier. On the other hand, WinUnit has a clean, single public header file (contrast that to CppUnit!), which also simplified things. And as WinUnit is limited to C++, I was able to use C++ templates (in combination with some preprocessor macros) to implement the entire WinUnit compatibility layer without having to change a single line of cfix itself. Rather, the WinUnit macros/classes are all mapped onto the existing cfix C++ API, which already includes most of what was neccessary to implement the WinUnit functionality.

Conclusion

In case have been using WinUnit the past and have a set of existing WinUnit-based test suites, give cfix a try — Not only should it be a full-featured replacement for WinUnit, you can also expect to see, and benefit from new features in upcoming releases!

Last but not least, the release contains a number of minor bugfixes. So upgrading is recommended even if you do not intend to use the new WinUnit compatibility feature.

cfix can be downloaded here.


Categories




About me

Johannes Passing, M.Sc., living in Berlin, Germany.

Besides his consulting work, Johannes mainly focusses on Win32, COM, and NT kernel mode development, along with Java and .Net. He also is the author of cfix, a C/C++ unit testing framework for Win32 and NT kernel mode, Visual Assert, a Visual Studio Unit Testing-AddIn, and NTrace, a dynamic function boundary tracing toolkit for Windows NT/x86 kernel/user mode code.

Contact Johannes: jpassing (at) acm org

Johannes' GPG fingerprint is BBB1 1769 B82D CD07 D90A 57E8 9FE1 D441 F7A0 1BB1.

LinkedIn Profile
Xing Profile
Github Profile
Advertisements