7.1 Unit Tests

Unit tests are small pieces of software that test specific conditions in your source code. Typically, it is the developer's responsibility to write and maintain unit tests. Unit tests are often used to test such conditions as boundaries, unusual data types, interfaces between components, and any complex operation that needs to be continually verified as the software changes. Unit tests should be run regularly as the software is being built. Each development team uses their own methodology for building software. We recommend that the following simple rules be applied for working with unit tests:

• Any piece of code checked into the main code base must have a corresponding unit test.

• Code cannot be checked into the main code base until it passes its unit tests.

• Unit tests should be kept current, being updated as the code is updated.

• Unit tests should be part of a framework that runs regularly (every night, every week, and so on) and reports meaningful results.

A unit test can be something as simple as:

int main()
{
   ... test some stuff
  return 0;
}

A simple framework can lend organization to your unit test strategy. We include a unit test framework with the following features:

• Each piece of functionality can be placed in a separate test function so it can be tested in isolation. New tests can be added by simply creating a new function.

• A few simple macros are included that make it easy to verify certain conditions and report any that are not correct.

• All exceptions are caught and reported as errors.

7.1.1 Using the Unit Test Framework

This section provides an overview of how you use the unit test framework that we provide.

EXAMPLE We have written unit tests for almost every component we present in this book. All unit tests are included on the CD-ROM. Here, we use one of the apBString unit tests as an example: UTFUNC(Pel8) { setDescription ("Pel8 tests"); Pel8 b; Pel16 w; Pel32 l; Pel32s ls; float f; double d; std::string s; apBString bstr; bstr << (Pel8) 123; bstr >> b; bstr.rewind (); bstr >> w; bstr.rewind (); bstr >> l; bstr.rewind (); bstr >> ls; bstr.rewind (); bstr >> f; bstr.rewind (); bstr >> d; bstr.rewind (); bstr >> s; bstr.rewind (); VERIFY (b == 123); VERIFY (w == 123); VERIFY (l == 123); VERIFY (ls== 123); VERIFY (f == 123); VERIFY (d == 123); VERIFY (s.compare ("123") == 0); bstr >> b; VERIFY (bstr.eof()); } This function tests that a byte can be written to the binary string and then read back in multiple formats, to verify that the conversions were made correctly.

You can use the provided framework to write your own unit tests. Our framework encourages you to write a number of small, isolated tests instead of a large complex test. We strongly recommend that you test as much as possible in your unit tests, as we demonstrate in the following portion of the apRect unit test:

UTFUNC(rect)
{
  setDescription ("Rect");

  apRect rect (0, 1, 2, 3);
  VERIFY (rect.x0() == 0);
  VERIFY (rect.y0() == 1);
  VERIFY (rect.width() == 2);
  VERIFY (rect.height() == 3);
  ...
}

This function is testing trivial inline functions and the apRect constructor. Do not assume that this simply works, or that other test functions will indirectly test these member functions. You should test these functions directly. Here is the unit test for the apRect default constructor:

UTFUNC(defaultctor)
{
  setDescription ("default ctor");

  apRect rect;
  VERIFY (rect.x0() == 0);
  VERIFY (rect.y0() == 0);
  VERIFY (rect.width() == 0);
  VERIFY (rect.height() == 0);
  ...
}

Your unit test file contains one or more UTFUNC() functions as well as a main() function. If you want to include any custom pre- or post-processing, you can do so, as follows:

int main()
{
  // Add any pre-processing here
  bool state = apUnitTest::gOnly().run ();
  // Add any post-processing here
  apUnitTest::gOnly().dumpResults (std::cout);
  return state;
}

apUnitTest is a Singleton object that contains a list of all unit tests to run. The results for each unit test are stored internally and can be displayed when dumpResults() is called. Unit test functions should not generate any output on their own, unless that is the point of the test. Any extra input/output can skew the execution time measurements.

7.1.2 Design of the Unit Test Framework

Figure 7.1 illustrates the overall design of the unit test framework.

There is a base class, apUnitTestFunction, from which unit tests are derived using the UTFUNC() macro. There is a unit test framework object, apUnitTest, that maintains a list of all the unit tests, runs them, and displays the results. Each of these components is described in this section.

apUnitTestFunction Base Class

Each unit test is derived from the apUnitTestFunction base class using the UTFUNC() macro. The complete apUnitTestFunction base class is shown below.

class apUnitTestFunction
{
public:
  apUnitTestFunction (const std::string& name);

  enum eResult {eNotRun, eRunning, eUnknown, eSuccess, eFailure};

  const std::string& name         () const { return name_;}
  eResult            result       () const { return result_;}
  double             elapsed      () const { return elapsed_;}
  const std::string& message      () const { return message_;}
  const std::string& description  () const
  { return description_;}
  std::string        resultString () const;

  void setDescription (const std::string& s) { description_ = s;}

  void run (bool verbose = false);
  // Run this unit test. Called by the unit test framework
protected:
  virtual void test() = 0;
  // All unit tests define this function to perform a single test

  bool verify (bool state, const std::string& message="");
  // Fails test if state is false. Used by VERIFY() macro

  void addMessage (const std::string& message);
  // Adds the message string to our messages

  bool         verbose_;     // true for verbose output

  eResult      result_;      // Result of this unit test
  std::string  name_;        // Unit test name (must be unique)
  std::string  description_; // Description of function
  std::string  message_;     // Message, usual a failure message
  double       elapsed_;     // Execution time, in seconds
};

The run() method runs a single unit test, measures its execution time, creates a catch handler to deal with any unexpected exceptions, and determines the result. Note that the actual unit test is defined within the test() method.

The implementation of the run() method is shown here.

void apUnitTestFunction::run ()
{
  std::string error;

  apElapsedTime time;
  try {
    test ();
  }
  catch (const std::exception& ex) {
    // We caught an STL exception
    error = std::string("Exception '") + ex.what() + "' caught";
    addMessage (error);
    result_ = eFailure;
  }
  catch (...) {
    // We caught an unknown exception
    error = "Unknown exception caught";
    addMessage (error);
    result_ = eFailure;
  }
  elapsed_ = time.sec ();

  // Make sure the test() function set a result or set eUnknown
  if (result_ != eSuccess && result_ != eFailure)
    result_ = eUnknown;
}

Note that the source code also includes a verbose mode to display immediate results, which we have removed for the sake of brevity.

apUnitTest Object

Our unit test framework object, apUnitTestObject, maintains a list of unit tests, runs all of the unit tests in order, and displays the results of those tests. Its definition is shown here.

class apUnitTest
{
public:
  static apUnitTest& gOnly ();

  bool run (bool verbose = false);
  // Run all the unit tests. Returns true if all tests are ok

  void dumpResults (std::ostream& out);
  // Dump results to specified stream

  int size () const { return static_cast<int>(tests_.size());}
  const apUnitTestFunction* retrieve (int index) const;
  // Retrieves the specific test, or NULL if invalid index

  void addTest (const std::string& name,
                apUnitTestFunction* test);
  // Used by our macro to add another unit test

private:
  apUnitTest ();
  static apUnitTest* sOnly_;  // Points to our only instance

  std::vector<apUnitTestFunction*> tests_; // Array of tests
};

A std::vector maintains our list of unit tests. The run() method steps through the list, in order, and executes all the unit tests, as shown.

bool apUnitTest::run ()
{
  bool state = true;

  for (unsigned int i=0; i<tests_.size(); i++) {
    apUnitTestFunction* test = tests_[i];
    test->run ();
    if (test->result() != apUnitTestFunction::eSuccess)
      state = false;
  }
  return state;
}

EXAMPLE In this example, we look at the output for the apBString unit test. (Note that we include the complete source code for the apBString unit test on the CD-ROM.) Running the unit test produces the following output: [View full width] Unit Test started at Thu Jul 04 23:39:46 2002 Unit Test finished at Thu Jul 04 23:39:46 2002 Test 1: Success : ctor : Constructor and simple accessor tests: 0 sec Test 2: Success : Pel8 : Pel8 tests : 0 sec Test 3: Success : Pel16 : Pel16 tests : 0 sec Test 4: Success : Pel32 : Pel32 tests : 0 sec Test 5: Success : Pel32s : Pel32s tests : 0 sec Test 6: Success : float : float tests : 0 sec Test 7: Success : double : double tests : 0 sec Test 8: Success : string : string tests : 0 sec Test 9: Success : eof : eof tests : 0 sec Test 10: Success : data : data tests : 0 sec Test 11: Success : bstr : bstr tests : 0 sec Test 12: Success : dump : dump tests : 0 sec Test 13: Success : point : point tests : 0 sec Test 14: Success : rect : rect tests : 0 sec Passed: 14, Failed: 0, Other: 0

The execution times are all reported as 0 because each test is very simple. This unit test framework is portable across many platforms and the results are similar on each platform.

We can simulate a failure by adding a simple unit test function to our framework, as shown:

UTFUNC (failing)
{
  setDescription ("Always will fail");
  VERIFY (1 == 2);
}

The output would include these additional lines:

Test 15: ***** Failure ***** : failing : Always will fail : 0 sec
  Messages:
1 == 2

Passed: 14, Failed: 1, Other: 0

Notice that the conditional is included as part of the failure message.

Macros in the Unit Test Framework

The unit test framework uses two macros: UTFUNC() and VERIFY(). In general, we tend to avoid macros; however, they are very useful in our unit test framework. Figure 7.2 provides a quick overview of the syntax used in macros.

Note that parameters used in macros are not checked for syntax; rather, they are treated as plain text. Parameters can contain anything, even unbalanced braces. This can result in very obscure error messages that are difficult to resolve.

The UTFUNC() macro creates a unit test function of the specified name by deriving an object from the apUnitTestFunction base class. UTFUNC() is defined as follows:

#define UTFUNC(utx)                            \
class UT##utx : public apUnitTestFunction      \
{                                              \
UT##utx ();                                    \
static UT##utx sInstance;                      \
void test ();                                  \
};                                             \
UT##utx UT##utx::sInstance;                    \
UT##utx::UT##utx () : apUnitTestFunction(#utx) \
{                                              \
  apUnitTest::gOnly().addTest(#utx,this);      \
}                                              \
void UT##utx::test ()

For example, the preprocessor expands the UTFUNC(rect) macro into the following code:

class UTrect : public apUnitTestFunction
{
UTrect ();
static UTrect sInstance;
void test ();
};
UTrect UTrect::sInstance;
UTrect::UTrect () : apUnitTestFunction("rect")
{
  apUnitTest::gOnly().addTest("rect",this);
}
void UTrect::test ()

Every unit test function creates a new object with one static instance. These objects are constructed during static initialization, and automatically call addTest() to add themselves to the list of unit test functions to be run. Note that the last line of the expanded macro is the test() method, and your unit function becomes its definition.

The VERIFY() macro is much simpler than UTFUNC(). It verifies that a specified condition is true. Its definition is as follows:

#define VERIFY(condition) verify (condition, #condition)

Let's look at the following example:

VERIFY (rect.x0() == 0);

The preprocessor expands this macro into the following code:

verify(rect.x0() == 0, "rect.x0() == 0");

The VERIFY() macro calls a verify() method that is defined in the apUnitTestFunction base class, as shown.

bool apUnitTestFunction::verify (bool state,
                                 const std::string& message)
{
  if (!state) {
    result_ = eFailure;
    addMessage (message);
    if (verbose_)
      std::cout << " FAILURE " << name_.c_str() << " : "
                << message.c_str() << std::endl;
  }
  else if (result_ != eFailure)
    // Make sure we mark the unit test success, if possible.
    result_ = eSuccess;
  return state;
}

state is the result of the conditional expression. If the result is false, a failure message, including the string of the conditional, is written to an internal log. This failure message is displayed after all of the unit tests have been run.

setDescription() is a method that lets you include more descriptive information about the test. It is very useful if you have a number of tests and wish to clarify what they do.

7.1.3 Extending the Unit Test Framework

The unit test framework that we have included is only a beginning of a complete solution. We recommend using this framework as a basis to construct a fully automated unit test framework that runs unit tests at regular intervals, such as each night. An automated framework would do the following:

• Obtain the most recent sources for your application. This involves obtaining a copy of the sources from whatever configuration management package you use.

• Compile and build any intermediate libraries.

• Compile each unit test.

• Execute all unit tests, capture the results, and record which unit tests fail.

• Generate a report, including a summary or detailed information about the tests.

• Email the report to the appropriate software developers and engineers.

Our experience is that at least half of all unit test failures are not actually failures in the objects being tested. Failures tend to occur when an object has been modified, but the unit test lags behind.