Tools « Johannes Passing's Blog

Archive for the 'Tools' Category

Visual Assert 1.1 beta and cfix 1.7 released

Published May 24, 2010 cfix , Testing , Tools , Visual Assert , Visual Studio , Win32 1 Comment

Slightly delayed, Visual Assert 1.1 beta is now available for download. As announced in a previous post, the most important change in the new version is added suport for the latest version of Visual Studio, Visual Studio 2010.

However, the new version also brings a couple of new features that apply to all versions of Visual Studio. Most importantly, cfix and Visual Assert now expose an API that allows developers to plug in custom event sinks. A custom event sink is implemented as a DLL and receives all events the runtime generates during the execution of a test suite. As such, the API is perfectly suited for implementing custom loggers.

To make this new feature easily usable, the Options dialog (Menu: Tools > Options) has been enhanced appropriately. Moreover, the dialog now includes some further options concerning stack size, current directory adjustment, and VC++ directory registration that have not been exposed previously.

Coming along with the new Visual Assert release, a new cfix release, version 1.7, is now available on Sourceforge.

What a weirdo: How the /analyze switch changes its behavior depending on its environment

Published February 13, 2010 Tools , Visual Studio , WDK , Win32 Leave a Comment
Tags: cl, codeanalysis, compiler, sdk, Visual Studio, vsts, Windows

In Visual Studio 2005 Team System (VSTS), the “ultimate” SKU of Visual Studio 2005, Microsoft introduced the /analyze compiler switch. When the /analyze switch is used, the cl compiler not only does its regular checks, but performs a much more thorough static code analysis.

While /analyze is very useful indeed, it was only available in the top SKU — the Standard and Professional versions of Visual Studio lacked support for this compiler switch (this has changed by now, Professional now also supports this feature). As some smart people quickly figured out though, the compilers shipped as part of the Windows SDK did support /analyze, too.

So given that some compilers do support /analyze while other do not, you may well expect that there are two slightly different types of binaries, one that the SDK and VSTS uses, and one that is shipped with other Visual Studio SKUs.

At least this was what I expected. As it turns out though, this is not quite the case.

Where’s /analyze?

For the past two years, I have been developing using Visual Studio 2005 Team System along with Windows SDK 6.0 and WDK 6000 on a Vista x64 machine. Using this setup, I was able to use the /analyze switch in both, “regular” Visual Studio projects and WDK (build.exe-driven) projects. That led me to the conclusion that the WDK 6000 compilers, like the SDK compilers were in fact /analyze-enabled binaries as well.

Switching to a Windows 7 machine with VSTS 2005 and 2008, SDK 7.0, and WDK 6000 did not change this — /analyze kept working fine in all environments.

Then I set up a build server, installed WDK 6000 and Windows SDK 7.0 and attempted to perform a build — to my surprise, though, I got plenty of complaints about the /anayze switch not being supported.

I verified that the right compilers (WDK 6000) were used and compared cl versions between the build machine and my development machine — both were 14.00.50727.220, so everything seemed right. Running cl.exe /? on both machines, however, I noticed that despite versions being the same, this Code Analsis section was missing in the output on the build machine:

                         -CODE ANALYSIS-

/analyze[:WX-] enable code analysis
    WX- - code analysis warnings should not be treated as errors even if /WX is invoked

So obviously, Code Analysis support is enabled or disabled depending on external factors — not the binary itself, but the environment somehow determines whether the /analyze switch is supported or not.

Observing cl.exe /? with Process Monitor on my development machine resulted in the following output:

This trace leaves little room for interpretation: The code analysis features must (mainly) be implemented in c1xxast.dll. c1xxast.dll, however, is not shipped with the WDK itself, nor is it shipped with the non-VSTS SKUs of Visual Studio. So by default, the WDK’s cl will fail to locate the DLL and will revert to “/analyze-disabled mode”.

If, however, you have VSTS or the Windows SDK installed on your machine and your %PATH% happens to include the right directories, cl’s search for c1xxast.dll will succeeded and — tada — /analyze suddenly works. On my development machine, this obviously was the case, whilst on the build machine, it was not.

Compiler version mish-mash

I added the Windows SDK’s bin directory to the build machine’s %PATH% and rerun the build. As I expected, /analyze now worked fine — what I did not quite expect though was that I was now getting dozens of compilation warnings like:

warning C6309: Argument '1' is null: this does not adhere to function 
specification of 'CfixCreateThread'

The reason for this was simple: The WDK cl.exe (remember, version 14.00.50727.220), thanks to a proper %PATH%, now used c1xxast.dll from SDK 7 to perform code analysis — despite the fact that c1xxast.dll actually “belonged” to cl version 15.00.30729.01. So the c1xxast.dll was one generation ahead of the WDK I was using.

The really, really cool thing about cl being able to work with a newer c1xxast.dll is that you can continue using WDK 6000 or 6001 (with W2K support!) and still benefit from the latest-and-greatest static code analysis features.

The reason for getting several warnings on the build machine while not getting similar warnings on my development machine was simply that on my development machine, the VS 2005 directory preceded the SDK directory in my %PATH%. Once I switched the order, I got the same wanings on both machines. This leads me to:

The ugly thing about this, however, is that a tiny change in the order of directories in %PATH% can suddenly make a huge difference w.r.t. code analysis. This is not quite what you’d normally expect.

(The additional compiler warnings, by the way, were a result of the improved analysis checks in cl 15: cl 14 routinely failed to verify the usage of __in vs. __in_opt parameters; cl 15 has become much more precise here and found several mis-attributed function signatures.)

Visual Assert hits RTM, now available for free

Published January 31, 2010 cfix , Testing , Tools , Visual Assert , Visual Studio , Win32 5 Comments
Tags: addin, c, free, plugin, release, rtm, VC, vs

Visual Assert, the unit testing Add-In for Visual Studio/Visual C++ has finally left its beta status and — better yet — is now available for free, both for commercial and non-commercial use.

Visual Assert, based on the cfix 1.6 unit testing framework, allows you to easily write, manage, run, and debug your C/C++ unit tests -– without ever leaving the Visual Studio® IDE. No fiddling with command line tools, no complex configuration, and no boilerplate code required.

Sounds good? Then go straight to the Visual Assert homepage and download the installer!

cfix 1.6 released, simplifies authoring of multi-threaded tests

Published January 31, 2010 cfix , Testing , Tools , Visual Assert 1 Comment
Tags: cfix, functional testing, open source, oss, qa, Testing, unit testing

A new release of cfix, the unit testing framework for C and C++, is now available for download. Besides some minor enhancements like extending the maximum permitted fixture name, cfix 1.6 introduces a major new feature, Anonymous Thread Auto-Registration.

Since its very first release, cfix has supported multi-threaded test cases, i.e. test cases that spawn child threads, each of which potentially making use of the various assertion statements like CFIX_ASSERT. To make this work and ensure that failing assertions are handled properly, however, usage of CfixCreateThread (rather than the native Win32 CreateThread) was mandatory when spawning such threads.

Although using CfixCreateThread (or CfixCreateSystemThread in case of kernel mode code) remains the preferred way to create child threads, there are situations where usage of this API is not possible — for example, when the child threads are created by libraries such as boost.

To support such scenarios, cfix now allows you to annotate a fixture to enable Anonymous Thread Auto-Registration, which means that newly spawned threads are automatically registered with cfix — no usage of CfixCreateThread required. Thanks to this feature, writing multi-threaded tests becomes straightforward — and integrating with libraries such as boost does not pose a problem any more.

For more details about this feature, please refer to the respective section in the cfix documentation.

As always, updated binaries and source code are available on Sourceforge.

Visual Assert Beta 3 released

Published November 24, 2009 cfix , Testing , Tools , Visual Assert Leave a Comment
Tags: addin, c, cfix, tdd, Testing, unit testing, Visual Assert, Visual Studio

A third beta release of Visual Assert is now available for download on www.visualassert.com.

Visual Assert, in case you have not tried it yet, is an Add-In for Visual Studio that adds unit testing capabilities to the Visual C++ IDE: Based on the cfix unit testing framework, Visual Assert allows unit tests to be written, run, and debugged from within the IDE. Pretty much like Junit/Eclipse, TestDriven.Net or MSTest, but for real, native code — code written in C or C++.

Bugs, bugs, bugs, bugs

Alas, there were still a few of them in the previous two beta releases. Luckily though, almost all I received from users or found by internal testing were relatively minor in nature. Still, I want Visual Assert to be as high quality as possible and decided to add this third beta release into the schedule and take the time to focus on — you guessed it — bugfixing, bugfixing, bugfixing, and bugfixing.

Speaking of bug reports, I have to thank all users of Visual Assert and cfix who reported bugs, suggested new features or provided general feedback. Your input has been, and still is highly appreciated. Although I had to postpone any feature suggestions to a later release, I tried hard to resolve all bugs and have them fixed in this new release.

Download, Try it, Share Your Opinion

Of course, using the new Beta version is free. So whether you have already used the previous beta or not, whether you are a unit testing newbie or write unit tests on a daily basis, be sure to give the new version a try. And of course, do not forget to let me know about your feedback, suggestions, found bugs, etc.!

Download Visual Assert Beta 3
Version 1.0.2.3612

cfix 1.5.1 released

Published November 7, 2009 cfix , Testing , Tools Leave a Comment
Tags: bugfix, cfix, qa, release, unittesting

A new version of cfix, the unit testing framework for C and C++ on Windows, is now available on Sourceforge. Despite fixing several minor issues, the new version resolves the following two issues that were reported by users:

Definiting multiple WinUnit fixtures with setup/teardown routines in a single .cpp file leads to a compilation error
A thread handle is leaked during execution of a test (#2889511)

Updated binaries and source code are available for download on Sourceforge.

Btw, in case you use cfix for kernel mode testing and are using WDK 7600, please have a look at my previous post: LTCG issues with the WIN7/amd64 environment of WDK 7600

Launched ntrace.org

Published October 13, 2009 NTrace , Tools Leave a Comment
Tags: dtrace, dynamic tracing, fbt, lille, NTrace, tracing, wcre, Windows, x86

Having given my presentation on NTrace today at the WCRE in Lille/France, I have also opened ntrace.org to the public. NTrace, in case you have missed my previous posts, is a dynamic function boundary tracing system for Windows/x86 I initially developed as part of my Master’s thesis that is capable of performing DTrace-like tracing of both user and kernel mode components.

On the NTrace page, you will now find the paper itself as being published as part of the WCRE proceedings (mind the copyright notice, please) along with two screencasts: One showing how NTrace can be used to trace kernel mode components such as NTFS, and one demonstrating NTrace for user mode tracing.

If you have questions about NTrace or are interested in more details, please feel free to write me an email — my address is jpassing at acm org.

cfix studio Beta 2 to add support for EXE-based unit tests

Published August 6, 2009 cfix , Testing , Tools , Visual Assert , Visual Studio , Win32 Leave a Comment
Tags: addin, cfix, cfix studio, mfc, qa, Testing, Tools, unit test, Visual Studio

N.B. cfix studio was the code name of what has become Visual Assert

The biggest shortcoming of the current cfix studio version certainly is that it requires all tests be implemented in a DLL. Conceptually, keeping test cases separated from the remaining code certainly is a good idea — and implementing tests in a DLL is a way to accomplish this. However, there are many projects in which such separation is either not feasible or just too much effort.

The good news is that with Beta 2, this will finally change: EXEs become first class-citizens in cfix studio and it will not matter any more whether your tests are part of a DLL or EXE project — you can just put them where you think is appropriate.

Take a classic MFC/GUI application project as an example: It is pretty common for these kinds of projects that most, if not all, application logic is part of a single Visual Studio project that compiles into a single EXE. There may be some additional DLLs or LIBs, but by and large, the EXE itself is where most of the interesting things happen.

The upcoming Beta 2 release now allows you to implement all your unit tests as part of the same EXE project. This means that your tests have access to all classes, functions and resources that are part of the project — all of which you would not easily have access to if you implemented the tests in a separate DLL.

Of course, embedding unit tests into an executable raises two questions:

How to strip the tests from the final release?
How on earth will you be able to run these tests without having main() create windows, load files, play sounds, etc each time?

Thankfully, C/C++ has a preprocessor and Visual C++ has the “exclude from build” feature which allows you to exclude certain files whenever the project is built using a specific configuration. Using any of these two features, the first question is easily answered.

The second problem is more tricky — but thankfully, it has already been solved for you: When cfix studio runs unit tests, it is well aware of that running main() might have, let’s say interesting effects — so what it does is simple: It just makes sure that main() is never run! Not only does this ensure that the tests run silently, i.e. without windows popping up etc, it also has the benefit that all unit tests “see” the application in a pristine state: Rather than having to worry about which state main() has brought the application into, you can initialize and clean up any state you need in your Setup and Teardown functions¹.

To make a long story short: You can write unit tests in EXE projects in exactly the same manner as you would in a DLL project. No special considerations needed, no project settings that need to be changed, no additional boilerplate code to write. And when you run the EXE outside cfix studio, i.e. hit F5 in Visual Studio or launch the EXE directly, you will not even notice that the EXE houses some unit tests — everything works as normal.

Sounds good? Then wait a few more days and see it in action!

Remarks
1: Needless to say, all global variables are initialized, constructors are run, etc. All CRT initialization happens as normal; only main()/WinMain() is not run. And yes, it also works for apps that link statically to MFC and therefore do not have a “regular” WinMain().

More Context Menu Handlers for Everyday Use

Published August 5, 2009 Misc , Tools Leave a Comment
Tags: cmd.exe, shell, Tools, verbs, Windows

Although Windows Explorer may actually not be the brightest spot of Windows, it is still, for most users, among the most often used pograms. Customizing it to speed up certain tasks is thus a natural desire.

A while ago, I wrote about how to extend the context menu by new commands that allow MSI packages to be installed/uninstalled with logfiles being created. The registry entries I used were:

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Msi.Package\shell\LoggedInstall]
@="&Logged Install"

[HKEY_CLASSES_ROOT\Msi.Package\shell\LoggedInstall\command]
@="msiexec.exe /l* \"%1-install.log\" /i \"%1\" %*"

[HKEY_CLASSES_ROOT\Msi.Package\shell\LoggedUninstall]
@="L&ogged Uninstall"

[HKEY_CLASSES_ROOT\Msi.Package\shell\LoggedUninstall\command]
@="msiexec.exe /l* \"%1-uninstall.log\" /x \"%1\" %*"

[HKEY_CLASSES_ROOT\Msi.Package\shell\runas\command]
@="msiexec.exe /l* \"%1.log\" /i \"%1\" %*"

While I do not use Windows Installer every day, I am a heavy user of cmd.exe command prompts. Another set of custom verbs I use on my machines therefore deal with opening command line windows. Getting Windows Explorer to open a “normal” command prompt using the context menu is not hard and it has been demonstrated on various places. The idea becomes truly powerful, though, when the commands are specialized to open special kinds of command windows:

A plain command prompt
An elevated command prompt (using elevate.exe)
A WDK command prompt (WLH-chk)
A WDK command prompt (WLHA64-chk)
A Visual Studio 2005 command prompt
etc …

To distinguish the different types of consoles, I like to use different colors — The Visual Studio command prompt is white/green, the elevated prompt is green/blue, and so on. The following script puts it all together (mind the static paths):

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Directory\shell\Open DDK Console here (WLH-chk)]
@="Open DD&K Console here (WLH-chk)"

[HKEY_CLASSES_ROOT\Directory\shell\Open DDK Console here (WLH-chk)\command]
@="C:\\Windows\\System32\\cmd.exe /k C:\\WinDDK\\6000\\bin\\setenv.bat C:\\WinDDK\\6000\\ chk WLH && cd /D %1 && color 1f"

[HKEY_CLASSES_ROOT\Directory\shell\Open DDK Console here (WLHA64-chk)]
@="Open DD&K Console here (WLHA64-chk)"

[HKEY_CLASSES_ROOT\Directory\shell\Open DDK Console here (WLHA64-chk)\command]
@="C:\\Windows\\System32\\cmd.exe /k C:\\WinDDK\\6000\\bin\\setenv.bat C:\\WinDDK\\6000\\ chk AMD64 && cd /D %1 && color 1f"

[HKEY_CLASSES_ROOT\Directory\shell\Open Default Console here]
@="Open Default Conso&le here"

[HKEY_CLASSES_ROOT\Directory\shell\Open Default Console here\command]
@="cmd.exe /K \"title %1 && cd /D %1\""

[HKEY_CLASSES_ROOT\Directory\shell\Open Elevated Console here]
@="Open Ele&vated Console here"

[HKEY_CLASSES_ROOT\Directory\shell\Open Elevated Console here\command]
@="d:\\bin\\elevate.exe /K \"title %1 && color 1a && cd /D %1\""

[HKEY_CLASSES_ROOT\Directory\shell\Open VS.Net 2005 Console here]
@="Open VS.Net 200&5 Console here"

[HKEY_CLASSES_ROOT\Directory\shell\Open VS.Net 2005 Console here\command]
@="cmd.exe /K \"cd /D %1 && \"C:\\Program Files (x86)\\Microsoft Visual Studio 8\\VC\\vcvarsall.bat\" && color 2f\""

Finally, if you perform backups to the cloud from time to time and do want to upload unencrypted data or for other reasons encrypt specific files occasionaly, it may also be practical to have two GPG commands in your context menu — one to (symetrically) encrypt, and one to decrypt:

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\.gpg]
@="GpgFile"

[HKEY_CLASSES_ROOT\GpgFile]

[HKEY_CLASSES_ROOT\GpgFile\shell]

[HKEY_CLASSES_ROOT\GpgFile\shell\Decrypt]
@="Decrypt"

[HKEY_CLASSES_ROOT\GpgFile\shell\Decrypt\command]
@="\"c:\\Program Files (x86)\\GNU\\GnuPG\\gpg.exe\" -d -i -o %1.plain %1"

[HKEY_CLASSES_ROOT\*\shell\GpgSymmetricEncrypt]
@="Encrypt with GPG (symmetric)"

[HKEY_CLASSES_ROOT\*\shell\GpgSymmetricEncrypt\command]
@="\"c:\\Program Files (x86)\\GNU\\GnuPG\\gpg.exe\" -c %1"

Introducing cfix studio, the Visual Studio AddIn for C/C++ Unit Testing

Published June 23, 2009 cfix , Debugging , Testing , Tools , Visual Assert , Visual Studio Leave a Comment

N.B. cfix studio was the code name of what has become Visual Assert

There is little doubt that native code, and C and C++ in particular, is here to stay. And still, it is pretty obvious that when it comes to tools and IDEs, it is the managed world that has gotten most attention from tool vendors over the past years.

While there are lots and lots of useful tools for native development, many of them probably even better than their managed counterparts, there are some areas where the managed language fraction is far ahead: One of these areas certainly is IDE support for unit testing.

JUnit for Eclipse, TestDriven.Net for Visual Studio and MS Test make test-driven development so much more convenient and efficient that it is almost ridiculous that using command line tools is still state of the art for C/C++ development.

That said, there is great news: With cfix studio, there finally is a solution filling in this gap! cfix studio, based on the cfix unit testing framework, is a Visual Studio-AddIn that allows you to easily write, manage, run, and debug your unit tests from within Visual Studio. No fiddling with command line tools, complex configuration, or boilerplate code required!

Among lots of other features, cfix studio also has first-class support for multi-architecture development – you can easily switch back and forth between 32-bit and 64-bit and can even mix tests of different architectures in a single test run. Needless to say, cfix studio, like cfix, is also fully compatible to WinUnit.

If that has caught your interest, you are invited to check out the first beta version of cfix studio:

Download cfix studio Beta 1
Version 1.0.0.3458

It is free, quick to install and comes with a set of example projects. Give it a try — and please let me know about all your crticism, suggestions and other feedback!

Here are some screenshots of cfix studio in action:

The Test Explorer allows you to start a single or set of tests, the Run Window shows the results

Run Window: Viewing test progress

Run Window: Viewing test results and details of a failed assertion

When running in the debugger, a failed assertion will hit a breakpoint and the Run Window will show additional details

About me

Johannes Passing, M.Sc., living in Berlin, Germany.

Besides his consulting work, Johannes mainly focusses on Win32, COM, and NT kernel mode development, along with Java and .Net. He also is the author of cfix, a C/C++ unit testing framework for Win32 and NT kernel mode, Visual Assert, a Visual Studio Unit Testing-AddIn, and NTrace, a dynamic function boundary tracing toolkit for Windows NT/x86 kernel/user mode code.

Contact Johannes: jpassing (at) acm org

Johannes' GPG fingerprint is BBB1 1769 B82D CD07 D90A 57E8 9FE1 D441 F7A0 1BB1.

LinkedIn Profile

Xing Profile

Follow me on Twitter (new)

Johannes Passing's Blog