Archive for the 'cfix' Category



Visual Assert Beta 3 released

A third beta release of Visual Assert is now available for download on www.visualassert.com.

Visual Assert, in case you have not tried it yet, is an Add-In for Visual Studio that adds unit testing capabilities to the Visual C++ IDE: Based on the cfix unit testing framework, Visual Assert allows unit tests to be written, run, and debugged from within the IDE. Pretty much like Junit/Eclipse, TestDriven.Net or MSTest, but for real, native code — code written in C or C++.

Bugs, bugs, bugs, bugs

Alas, there were still a few of them in the previous two beta releases. Luckily though, almost all I received from users or found by internal testing were relatively minor in nature. Still, I want Visual Assert to be as high quality as possible and decided to add this third beta release into the schedule and take the time to focus on — you guessed it — bugfixing, bugfixing, bugfixing, and bugfixing.

Speaking of bug reports, I have to thank all users of Visual Assert and cfix who reported bugs, suggested new features or provided general feedback. Your input has been, and still is highly appreciated. Although I had to postpone any feature suggestions to a later release, I tried hard to resolve all bugs and have them fixed in this new release.

Download, Try it, Share Your Opinion

Of course, using the new Beta version is free. So whether you have already used the previous beta or not, whether you are a unit testing newbie or write unit tests on a daily basis, be sure to give the new version a try. And of course, do not forget to let me know about your feedback, suggestions, found bugs, etc.!

Advertisements

cfix 1.5.1 released

A new version of cfix, the unit testing framework for C and C++ on Windows, is now available on Sourceforge. Despite fixing several minor issues, the new version resolves the following two issues that were reported by users:

  • Definiting multiple WinUnit fixtures with setup/teardown routines in a single .cpp file leads to a compilation error
  • A thread handle is leaked during execution of a test (#2889511)

Updated binaries and source code are available for download on Sourceforge.

Btw, in case you use cfix for kernel mode testing and are using WDK 7600, please have a look at my previous post: LTCG issues with the WIN7/amd64 environment of WDK 7600

LTCG issues with the WIN7/amd64 environment of WDK 7600

Now that Windows 7 is out, we all sooner or later have to upgrade to WDK 7600. I am still reluctant to move away from WDK 6000/6001 because of the dropped W2K support, but this is a different issue.

However, as one cfix user who has obviously already adopted WDK 7600 kindly pointed out to me, linking a kernel mode unit test against cfix using WDK 7600 and the WIN7/amd64 environment fails reproducibly with the following error message:

error fatal error C1047: The object or library file ‘…\lib\amd64\cfixkdrv.lib’ was created with an older compiler than other objects; rebuild old objects and libraries

In contrast, building the same driver for WIN7/x86 works fine.

As the documentation for C1047 indicates, this error is usually related to inconsistent usage of Link Time Code Generation (LTCG): As soon as you use LTCG, all objects and libraries must be compiled with /GL — this normally is not a big deal, but as this WDK page rightfully explains, it means that libraries built this way are not suitable for redistribution because of their dependency on a specific compiler/linker version. But of couse, it also means that a library not built using /GL cannot be used easily when you build your program using LTCG.

Before Windows 7, all WDK build environment configurations I am aware of did not use LTCG. Neither did cfix, so everything worked fine even if your compiler/linker versions did not match the ones used for building cfix.

With WDK 7600, this situation changes: While WLH and other downlevel build environments still do not seem to use LTCG, the WIN7 environment, at least for amd64, enables LTCG by default.

What this means is that as soon as you link against a library which is not part of the WDK and therefore likely to be built using a different compiler version, you’ll get C1047. It is thus no surprise that attempting to link against cfixkdrv.lib, which is the library all kernel mode unit tests have to link against and which itself has been built using WDK 6000, leads to the error quoted above.

However, once you have figured this out, the workaround for this issue is trivial: Disable LTCG for your test driver by adding the following line to your SOURCES file:


USER_C_FLAGS=/GL-

Link time code generation is a very powerful optimization technique and I encourage everybody to make use of it if possible. However, for the compatibility reasons outlined above, I consider it a rather stupid idea to have the WDK enable LTCG by default. Rather, I had much preferred to see an opt-in switch for LTCG.

Anyway, for the next version, I will consider shipping both, a 7600-compatible LTCG-enabled and a non-LTCG-enabled version of cfixkdrv.lib. Until then, the workaround described above will do the trick.

Visual Assert Beta 2 Released

The Beta 2 release of Visual Assert (formerly named cfix studio) is now available for download. The release marks a major step in the development of Visual Assert for that it not only comprises a number of bugfixes but also introduces major new features. The two most important certainly are support for EXE targets and Wizard assistance.

Support for EXE Targets

As announced in a previous post and also discussed in the post about the cfix 1.5 release, Visual Assert now fully supports unit tets emedded in EXE modules.

Previous releases required all unit tests to be compiled and linked into DLLs. In fact, the usage of DLLs has many advantages and therefore remains the recommended practice. However, for certain projects, such a requirement can turn out to be a true obstacle: Whenever the code to be tested is not exported from a DLL or part of a static library (LIB), accessing this code from within such a test DLL could become quite a challenge.

The fact that Visual Assert Beta 2 now fully supports EXE modules means the following: You can now place your unit tests wherever you think they fit best. Whether they are part of a DLL or an EXE, whether you create separate “unit test” projects or intermingle your test code with other code — it is now all up to you. Wherever you placed your tests, Visual Assert will find them and will provide a consistent user experience.

And the best part of the support for EXE modules is that it is totally non-intrusive: You do not have to change your main/WinMain function, much less any other code or build settings. And when run “outside” Visual Assert, i.e. launched directly or in the Visual Studio Debugger, the application will behave as normal.

Needless to say, the cfix 1.5 command line test runners, cfix32.exe and cfix64.exe also have been updated to properly deal with EXE modules.

The Wizard

Although neither the WinUnit API nor the cfix C and C++ API require much boilerplate code to be written, there still is some amount of code that more or less all unit tests share. Thanks to the new Wizard, you can now have Visual Assert generate this code for you. This really helps creating new fixtures more quickly!

Download, Try it, Share Your Opinion

Of course, using the new Beta version is free. So whether you are a full time tester or a unit testing sceptic, download the new release and try it by yourself. And of course, your feedback, both positive and negative, is always welcome and can be posted here.

Download Visual Assert Beta 2

cfix 1.5 released; adds support for EXE-embedded tests and kernel mode multi-threading

A new version of the cfix unit testing framework is now ready for download.

Unlike the previous release, which was mainly a maintenance release, cfix 1.5 adds major new features: kernel mode multithreading and EXE-based unit tests.

EXE based unit tests

As I discussed in a previous post in the context of Visual Assert/cfix studio, cfix’ restriction to DLL based unit tests has turned out to be quite a limitation for certain kinds of projects.

In cfix 1.5, this restriction has been removed: cfix now supports both, unit tests compiled and linked into DLL modules and unit tests embedded into EXE modules.

Now, when I say embeded into an EXE module, I do not mean that you may merely leave out the /LD compiler switch — it means that you may compile and link unit tests into the actual application EXE module without impacting this application’s behavior or having to change the application’s main() routine: When you run the application direcly or through the debugger, it will behave as normal (e.g. launch GUI). Once you launch it via cfix32 -exe app.exe (or cfix64 -exe app.exe, respectively), however, the application’s main() routine will not execute and instead, your embedded unit tests will run.

This may seem awkward at first — but it offers a tremendous advantage: All of the application’s code now immediately becomes testable (i.e. accessible by unit test code) without having to do complex reorganization of your build process or source tree layout (although it is still a very good idea to enhance the build process s.t. the unit tests are stripped in the RTM builds and to make cfix.dll a delayload DLL). For many projects, relying on this feature will therefore make adopting cfix and maybe even unit testing in general much easier.

In almost all regards, EXE-embedded unit tests behave the same as their DLL counterparts. They are, however, slightly less handy when it comes to debugging (unless you use Visual Assert, which will shield you from this). For this reason, and their greater flexibility in general, it should be noted though that DLL-based unit tests will remain the preferred choice.

Kernel mode multithreading

Since its first release, cfix has featured support for multi-threaded test cases. Multi-threaded test cases are tests which spawn child threads and — and this is the important point — both, the “main” thread and the child threads may trigger assertions (i.e. use CFIX_ASSERT and friends). Regardless of the thread an assertion occurs on, it will be recognized by the framework and will lead to the test case being marked as having failed. In case this sounds trivial to you, be informed that not even JUnit properly supports this :)

And while this feature has been supported for user mode tests ever since, the infrastructure for kernel mode unit tests, which was added in version 1.1, has lacked support for this feature: There was no kernel mode counterpart of CfixCreateThread and thus, only single-threaded kernel mode tests were supported.

cfix 1.5 now finally introduces CfixCreateSystemThread: CfixCreateSystemThread is basically a wrapper for PsCreateSystemThread with the added functionality of registering the child thread with cfix. Thus, all threads spawned using CfixCreateSystemThread (rather than using PsCreateSystemThread directly) are allowed to make use of assertions.

Kudos to Matt Oh for beta-testing this feature.

Minor enhacements

Another enhancement related to multi threaded tests is Auto-joining of child threads: After a test routine completes, the framework will now automatically check whether any child threads were created. If this is the case and any of these threads are still running, they will be waited on — not before all threads have terminated will the test run resume. This feature both makes writing multi-threaded tests more convenient (you do not have to wait by yourself) and safer (No more runaway child threads).

Finally, another feature worth noting is that by specifying the -td command line switch, cfix can be directed to not capture a stack backtrace when an assertion fails. Although stack backtraces are usually very helpful, the symbol loading can make their creation quite expensive. Using this switch can therefore speed up the usage of cfix.

Download/Upgrade Now

You see, there are many good reasons to upgrade your cfix installation or — in case you are not using cfix yet — to give cfix a try.

Better yet, check out the Visual Assert AddIn for Visual C++ — it is based on the new cfix 1.5 release and seamlessly integrates with Visual Studio.

Download cfix 1.5.0 Installer
Download cfix 1.5.0 source code

cfix studio renamed to Visual Assert

Back when I began thinking about creating a Visual Studio Add-In for cfix, I needed a code name for the project. After tentatively using the name cfix+ for a while, I quickly settled on cfix studio — given that the project revolved around cfix and Visual Studio, this name pretty much suggested itself.

Soon after going into Beta, however, I had to realize that this name was not without its problems. Most importantly, it makes it hard for users to properly differentiate between cfix and cfix studio. This obviously led to situations where people were not quite sure whether cfix studio is a supplement to, replacement of, or just new version of cfix.

I would not care too much about this ambiguity if the two projects were not very different in terms of licensing: While using the Add-In will require a license to be purchased once it leaves the beta status While the Add-In is freeware (but not open source), the underlying cfix framework is, and will always remain open source and be licensed under the quite permissive LGPL.

As all APIs and libraries unit tests link against are part of the cfix framework, and the cfix framework itself is self-contained, this means that despite the Add-In being closed source, you still get the benefits of open source: Most importantly, there is no lock-in — you can stop using the Add-In and switch back to the command line tools at any time and be all-open source again. You are even free to create a fork of cfix at any time — there really is nothing other than convenience that binds you to using the Add-In.

Given that a growing number of people indeed tends to object to using closed source APIs and fears such vendor lock-in, I consider it important to stress this open source nature of the cfix framework underlying the Add-In.

However, at this point it should also become clear that a name that blurs the distinction between the two projects is counterproductive.

Based on this insight, I opted for dropping the cfix studio name and replacing it by something different: Visual Assert. This name should emphasise that the Add-In may be based on, but really is separate from the cfix framework.

Visual Assert

The new name will be used beginning with the upcoming Beta 2 release.

Vote for cfix

The Automated Testing Institute has elected cfix to be one of the finalists for the Autmation Honors award. The winners of the award will be highlighted in a Special December Edition of the Automated Software Testing Magazine.

If you are a cfix user, be sure to vote for cfix here.

And by the way, I think The Grinder, which is a really neat web performance testing framework, also deserves being voted for…


Categories




About me

Johannes Passing lives in Berlin, Germany and works as a Solutions Architect at Google Cloud.

While mostly focusing on Cloud-related stuff these days, Johannes still enjoys the occasional dose of Win32, COM, and NT kernel mode development.

He also is the author of cfix, a C/C++ unit testing framework for Win32 and NT kernel mode, Visual Assert, a Visual Studio Unit Testing-AddIn, and NTrace, a dynamic function boundary tracing toolkit for Windows NT/x86 kernel/user mode code.

Contact Johannes: jpassing (at) hotmail com

LinkedIn Profile
Xing Profile
Github Profile
Advertisements