How to customize test run execution to make your cfix test runs more effective

One of the features introduced back in cfix 1.2 was the ability to customize test execution with the command line parameters -fsr and -fsf. Using these switches can make your test runs more effective and can help simplify debugging — so it is worth spending a minute on this topic.

Assume our test run comprises two fixtures, Fixture A and Fixture B. As fixtures are always run in alphabetic order, and tests run in the order they are defined, the first test to be executed is Test 1 of Fixture A, followed by Test 2 of Fixture A, and so on. Assuming all tests of Fixture A succeed, execution then proceeds with the first test of Fixture B. The following figure illustrates this:

Successful execution of a unit testing suite

Things get interesting when one of the tests fails: Let us assume Test 2 of Fixture A leads to a failed assertion. The default behavior of cfix, which equals the behavior of JUnit and NUnit, is to report the failure and proceed with Test 3 of the same fixture:

Execution resumes after a failed unit test

In many cases, this is the appropriate thing to do — letting the test run to completion and figure out the reasons for the failure afterwards. But in certain scenarios, resuming with Test 3 might not be the smartest thing to do — rather, as soon as a test fails, execution should proceed with the next fixture. This is what -fsf (short-circuit fixture”) does:

Short-circuiting the fixture

Once a test case has failed, all remaining tests of the same fixture are skipped.

When debugging, even resuming at the next fixture is often not desired. Rather, as soon as one test fails, you may immediately want to take a closer a look at the failure, so resuming the run is mute. -fsr (short-circuit run”) does right that: It aborts the run immediately.

Short-circuiting the run

By the way — in Visual Assert, you can quickly access these options in the Options menu in the Test Explorer window:

Visual Assert Test Explorer

Advertisements

Categories




About me

Johannes Passing, M.Sc., living in Berlin, Germany.

Besides his consulting work, Johannes mainly focusses on Win32, COM, and NT kernel mode development, along with Java and .Net. He also is the author of cfix, a C/C++ unit testing framework for Win32 and NT kernel mode, Visual Assert, a Visual Studio Unit Testing-AddIn, and NTrace, a dynamic function boundary tracing toolkit for Windows NT/x86 kernel/user mode code.

Contact Johannes: jpassing (at) acm org

Johannes' GPG fingerprint is BBB1 1769 B82D CD07 D90A 57E8 9FE1 D441 F7A0 1BB1.

LinkedIn Profile
Xing Profile
Github Profile

%d bloggers like this: