Unit Testing

Unit testing is the process of showing that a part of a software system works as far as the requirements created for that part of the system. Unit testing is best if it has the following characteristics:

The software component is tested in isolation with as little interaction with other software components as possible.
The software component is tested using automated tools so that unit tests can be run with every build of the software if required.
All requirements of the software component are tested as part of the unit tests.

Unit testing is not the only type of testing but is definitely a very important part of any testing strategy. Following unit testing, software should go through "integration testing" to show that the components work as expected when put together.

This article describes the "why's" and "how's" of unit testing for the Haiku project.

Why is unit testing important?

A basic concept of software engineering is that the cost of fixing a bug goes up by a factor of 2-10x (depending on the source of the information) the later in the development process it is found. Unit testing is critical to finding implementation bugs within a particular component as quickly as possible.

Unit testing will also help to find requirements problems. If you write the requirements (or use cases) for your component from the BeBook, hopefully the BeBook and your use cases will match the actual Be implementation. A good way to confirm that the BeBook documentation matches Be's implementation is to write your unit tests and run them against the original Be code.

Unit tests will also continue to be maintained and run in the future. As the mailing lists obviously show, many people are looking forward to Haiku post-R1 when new features will be introduced above and beyond BeOS R5. These unit tests will be critical to ensuring that any new feature or even just a bug fix doesn't break existing functionality.

Speaking of bug fixes, consider adding unit tests for any bugs you identify that slipped through your original unit test suite. This will ensure that this bug or a similar one is not re-introduced in the future.

Finally, unit testing is not the be all, end all of testing. As mentioned above, integration testing must be done to show that software components work together. If all unit tests cover all requirements and have run successfully against all components, then a failure has to be due to a bug in the interaction of two or more known working software components.

When should I write my unit tests?

The recommended order for implementing a component is:

Write an interface specification
Write the use case specifications
Write the unit tests
Write an implementation plan
Write the code

Please see the The unit tests are to be written once the use cases are written and before any implementation work is done. The use cases must be done because they determine what the tests will be. You need to write as many tests are required so that all use cases for that component are tested. The use cases should be detailed enough that you can write your unit tests from them.

The unit tests are to be done before implementation for a very good reason. You should be able to run these unit tests against the Be implementation and confirm that they all pass. If they do not pass, then either there is a bug in the unit test itself or you have found a difference between your use cases and the actual implementation. Even if your use cases match the BeBook, if that is not how the actual Be implementation works, we must match the current implementation and not the BeBook. You should go back and modify the use case. Change the use case so that it matches Be's implementation and consider adding a note indicating this doesn't match the BeBook.

Imagine if you completed the implementation and then wrote and ran the unit tests. If you run the tests against your implementation and Be's implementation, you will notice the test passes for your code but fails on Be's. At this point, you will have to change the implementation, change the unit test and change the use case which is more work that if you write the unit tests before the implementation. Worse, if you only ran the unit tests against your implementation and not Be's, you may not notice the problem at all.

What kinds of tests should be in a unit test?

The unit tests you write should cover all the functionality of your software module. That means your unit tests should include:

All standard expected functionality of the software component
All error conditions handled by the software component
Interaction with software components which cannot be decoupled from the target software component
Concurrency tests to show that a software component which is expected to be thread safe (most things are under BeOS) is safe and free from deadlocks.

What framework is being used to do unit testing in Haiku?

The Haiku project has chosen to use CppUnit as the basis of all of our unit tests. This framework provides very useful features and ensures that all unit tests are consistent and can be executed from a single environment.

There are two key components to the framework. First, there is a library called libCppUnit.so which provides all the C++ classes for defining your own testcases. Secondly, there is an executable called "TestRunner" which is capable of executing a set of testcases.

For more information on CppUnit, please refer to this website

What Haiku specific modifications have been made to this framework?

The following are the modifications that have been introduced into the CppUnit framework:

A jamfile has been added for the library and the TestRunner.
Some "bugs" in CppUnit which lead to it not compile under BeOS R5.
The TestRunner has been replaced by our own UnitTester, to support BeOS-style addons. Each test which you can select from the TestRunner is found in the "add-ons" directory at runtime. The original TestRunner required you to change the TestRunner when new tests were added to it.
Changed the output from TestRunner. The output includes a name of the test being run and a run time for the test in microseconds.
Changed the arguments of the assert functions in the TestCase class from std::string to const char *'s due to apparent concurrency problems with std::string under BeOS when testing threaded tests.
Added locking to the TestResults class so that multiple threads can safely add result information at the same time for a single test.
The ThreadedTestCaller class was written to allow us to write tests which contain multiple threads. This is an important class because many BeOS components are thread safe and we need to confirm that the Haiku implementation is also thread safe.

This is the list of the important modifications done to CppUnit at the time this document is being written. For the latest information about modifications to CppUnit, check the code which can be found in the Haiku git repository.

What framework modifications might be required in the future?

This framework will have to evolve as our needs grow. The main issues I think we need to solve are:

The format of the test name is an encoded string representing the class definition of the test class from gcc. It is not a very readable format but given that the test class is often a template class and you would like different names for different instances of the template, this seemed the best compromise. Suggestions welcome.
The threaded test support added into CppUnit forces you to specify the entry point for each thread in your test. If you are doing a test with a BLooper or a BWindow, these classes start a thread of their own. This thread will not be started through the standard entry point so doing "assert's" from one of these threads will not work. Perhaps we need TestBLooper and TestBWindow classes which will work with the assert's.

If you find you need some other features, feel free to add them to CppUnit.

How do I build the framework and current tests for Haiku?

As of writing this document, you can build the framework and all the current tests by performing the following steps:

Checkout the entire repository from the Haiku git repository. There is information at the Haiku site about how to access the git repository.
In a terminal, "cd" into the "generated" directory
Type "jam -q unittests".

When the build is complete, you should find the following files:

tests/haiku/x86/haiku/unittests/UnitTester — this is the executable to use to execute tests.
tests/haiku/x86/haiku/unittests/lib/libcppunit.so — this is CppUnit library which your tests must link against.
tests/haiku/x86/haiku/unittests/lib/*test.so — these are the add-ons containing tests for each of Haiku kits with the respective names.

These are the key files which ensure that the tests can be run.

How do I run tests?

You have a few different options for how you run a test or a series of tests. Before you start however, you must build the code as described in the above section. Once it is built, you can run tests any of these ways:

Execute the UnitTester executable without arguments to run all the tests.
Execute the UnitTester executable with a specific test or test set name to run just that test. The --list option can be used to list all available tests.

How do I write tests for my component?

The first step to writing your tests is to develop a plan for how you will test the functionality.

Once you know the kinds of tests you want, you need to:

For every test you want, define a class which derives from the "TestCase" class in the CppUnit framework.
Within each test class you define, create a "void setUp(void)" and "void tearDown(void)" member function if required. If before executing your test, you need to perform some actions, put those actions in the "setUp()" member. If you need to cleanup after your test, put those actions in the "tearDown()" member.
Within each test class you define, create a member function which takes "void" and returns "void". Within this member function, write the code to execute the test. Whenever you want to ensure that some condition is true during your test, add a line within the member function that looks like "assert(condition)". For example, if the variable "result" must have the value B_OK at a particular point in your test, you should add a line which reads "assert(result = B_OK)".
Create a constructor for all of your test classes that takes a "std::string name" argument and pass that onto the TestCase parent class. Add whatever actions you need to take in the constructor.
Create a destructor for all of your test classes and take whatever actions are appropriate.
Within each test class you define, create a member with the signature "static Test *suite(void)". For a simple test where only one test needs to be run for this class, the contents of this member should look like:
```
return(new TestCaller<ClassName>("", &ClassName::MemberName));
```
Replace "ClassName" with the name of your test class and "MemberName" with the name of the member function you defined your test in. If you need to define more than one test to run from this class, refer to instructions below on how to use the TestSuite class of CppUnit. If you are creating a threaded test, refer to this section.
Create one ".cpp" file for defining the "addonTestFunc()" function. This function must exist in global scope within your test addon. The contents of this ".cpp" file will look something like:
```
#include "TestAddon.h"
Test *addonTestFunc(void)
{
    TestSuite *testSuite = new TestSuite("<TestSuiteName>");
    testSuite->addTest(<ClassName1>::suite());
    testSuite->addTest(<ClassName2>::suite());
    /* etc */
    return(testSuite);
}
```
In the above example, replace <TestSuiteName> with an appropriate name for the group of tests and <ClassName1> and <ClassName2> with the names of the test classes you have defined.
Create a build system around a BeIDE project, Makefile or preferrably a jam file which builds all the necessary code you have written into an addon.
Put this addon into the app_kit/test/add-ons directory and follow the above instructions for how to run your tests.

Are there example tests to base mine on?

There are example tests which you can find in the following directories:

src/tests/kits/app/bmessagequeue
src/tests/kits/support/bautolock
src/tests/kits/support/blocker

There are some things done in these tests which make things a bit more complex, but you may want to do similar things:

Most tests use a ThreadedTestCaller class even in some situations when there aren't actually more than one thread in the test.
All tests are defined as a template class. The test class is a template of the class to test (if that makes sense to you). For example, to test both the Be and Haiku BLocker and not end up with a symbol conflict, the Haiku implementation of BLocker is actually in a namespace called "Haiku". So, the tests must be run against the classes "::BLocker" and "Haiku::BLocker". The easiest way to do this was to make the class to be tested a template and define it for both "::BLocker" and "Haiku::BLocker".

Even with the complexity, I think this code provides a pretty good example of how to write your tests.

How do I write a test with multiple threads?

If you have a test which you want to define that requires more than one thread of execution (most likely a concurrency test of you code), you need to use the ThreadedTestCaller class. The steps which differ from the above description on how to write a test case are:

In your test class, define a member function for each thread you will be starting. All of these member functions must take "void" and return "void". If all the threads in your test perform the exact same actions, it is OK to just define one member function. Usually in the tests I have written, I have called these member functions "TestThread1()", "TestThread2()", etc.
If your "static Test *suite()" function for your test class, you must return a ThreadedTestCaller. Imagine that the test class name is "MyTestClass" and you want two threads which run member functions "TestThread1()" and "TestThread2()". That code would look like:
```
Test *MyTestClass::suite(void)
{
    MyTestClass *theTest = new MyTestClass("");
    ThreadedTestCaller<MyTestClass> *threadedTest =
        new TreadedTestCaller<MyTestClass>("", theTest);
    threadedTest->addThread(":Thread1", &MyTestClass::TestThread1);
    threadedTest->addThread(":Thread2", &MyTestClass::TestThread2);
    return(threadedTest);
}
```
If you need to, you can put a number of ThreadedTestCaller instances into a TestSuite and return them in the suite() member function. Examples of this can be found in the BLocker and BMessageQueue test examples.

Otherwise the steps are the same as for other tests. The code gets much more complex if you define your test classes as templates as the examples do.