Revision as of 08:42, 2 June 2021

A good set of tests is a critical requirement for modern software development, both to ensure the quality of the product and to help developers to quickly modify code without introducing bugs or causing regressions. The literature on software testing is vast; excellent tutorials and references are available both online and in print, see Literature Survey.

All developers are encouraged to review tests and ensure that any work is covered by both acceptance and unit tests.

Current Test Architecture

Acceptance Tests

Acceptance tests in Gnucash are based on a home-grown (or anonymously sourced) set of macros and functions which can be found in src/test-core and src/engine/test-core. The quality, scope, and coverage of these tests varies greatly; some parts of GnuCash are tested lightly or not at all while others are tested fairly extensively. Although I've labelled them "acceptance tests", in many cases they're written more as unit tests whose scope is a single function rather than a whole module.

Unit Tests

Unit testing, invented by Kent Beck in the early 1990s, seeks to test the public interface of classes as thoroughly as possible: All member functions should be tested with as much variation of their parameters as possible, with an emphasis on corner cases. Tests should avoid dependence on the implementation to avoid brittleness.

GnuCash has adopted the GLib testing framework to facilitate unit testing of GObject-based classes and GLib-dependent code. Muslim Choclov wrote unit tests for the most important modules in LibQOF as a GSoC2011 project. Work continues to get all of LibQOF and the engine fully tested to facilitate major architectural changes needed to make GnuCash a proper database application.

For C++ code GnuCash has adopted the Google Test Framework.

See also https://developer.gnome.org/glib/stable/gtester.html for the actual command line tool to run tests.

Running Tests

 make check

or

 ninja check

builds and runs the full test suite. N.B. Some tests depend on additional libraries and the cmake built-in target test does not build them. Once the check target has been built check and test are synonyms.

These targets run the tests using the ctest command. Its default is to run all tests and to write the output to CMAKE_BUILD_DIRECTORY/Testing/Temporary/LastTest.log. Both can be overridden with command-line options, e.g.

 ctest -V -R test-engine.*

will run all of the tests whose target names begin with test-engine and will output the results to stderr. Individual tests can be built by naming their targets, e.g.

 ninja test-gnc-timezone

and those written in C which produce an executable can be run directly:

 bin/test-gnc-timezone

while those that don't can be run using ctest -R.

Travis CI

We have set up an automatic build and test bot on Travis CI to run the full test suite after every commit and for every Github pull request. The Travis build and test environment is configured through the .travis.yml file in the GnuCash root directory. Individuals with Github forks can easily arrange to run Travis on commits to their repositories as well from the Github repository's Settings Tab: Select Integrations & Services then select Travis CI from the drop down list, enter your password, and fill out the form.

Travis keeps track of the status of each branch's last test and will email the pusher if a push fails to build or test or if the previous push failed and the latest one didn't. Obviously if Travis reports that the branch is fixed no action is required. If Travis reports a failure, follow the link in the email and examine the output. If it indicates a problem with the commit then you should push a fix, but sometimes it will indicate an unrelated error. There are some tests that fail at random; you can look at the build history tab on the Travis page to see old failures and to compare the failing tests to see which tests they are. In those cases rerunning the build on Travis (the button is at the top left of the build page) will usually produce a passing test and a happy Travis.

For tests that fail on Travis but not on your local development system it can be helpful to keep a VM around that duplicates the Travis build environment, which you can determine by examining .travis.yml. Note that the default Travis environment is Ubuntu Precise (12.04), which maint uses; master changes that to Ubuntu Trusty (14.04) via the .travis.yml

 dist: trusty

Policy

• All new non-GUI code should include thorough unit tests. Automated testing of the GUI tends to be brittle, so GUI modifications should be hand tested in as many OS environments as possible before being committed.
• make check run from the top build directory should pass before commits are pushed.

Unit Test Policies

• Getter/Setter functions which only set or retrieve an instance member variable do not need to be tested.
• Convenience functions which only wrap another function to change the function's name or to provide a default argument do not require testing.
• Composed functions, or functions which simply string together a series of calls to other functions, need not be tested if the called functions are all tested, have no side effects, and where the composed function has only one flow of control.
• There is some disagreement among testing gurus about whether a function's parameter variations should be exercised in a single test function or separately in a test function per function call. Use your judgement here. Remember that the dictum of Agile Development is to write a little bit at a time and to refactor as often as you need to. That applies as much to test code as it does to production code. It's OK to change your mind!
• Similarly there is tension among the gurus about how much to make a test program dependent upon, and how much to use mock objects to replace actual dependency code. Keeping in mind the goal of a short code-compile-test cycle, use your judgement. That said, at present much of Gnucash is rather interdependent and doesn't virtualize functions -- a requirement for applying mocks. If you're writing new modules, do use modern OO techniques to minimize interdependence, and where it's necessary make sure to use virtual functions so that linking the rest of Gnucash isn't needed to test your work.

Writing Tests

Google Test based tests

Tests on C++ code should be written using GoogleTest and GoogleMock.

There are plenty of websites devoted to writing unit tests. There are also a couple of encyclopedic books on the topic, XUnit Test Patterns by George Mezaros and Working Effectively With Legacy Code by Michael Feathers.

Tests should be written in a test subdirectory of the directory containing the implementation file that you're testing. Create one if it doesn't already exist.

Once you've written your test you need to arrange for it to be compiled and run by the build system. The test directory needs a CMakeLists.txt if it doesn't already have one.

There's a handy function, gnc_add_test() that takes care of most of the boilerplate of adding a test to the build system. It takes four arguments: The test name, which must be unique in the whole build system and should be test-file-name where "file-name" is the file containing the functions you're testing; the source files for the test; a variable containing the include paths; and a variable containing the library flags and other cmake targets that the test depends upon. For example, suppose that we want to test foodir/foo.c. This file calls functions in libgncmod-engine and needs to include files from the libgnucash/engine directory. We'd use the following setup in CMakeLists.txt:

 set(test_foo_SOURCES ../foo.c test-foo.c)
 set(test_foo_INCLUDES ${CMAKE_SOURCE_DIR}/libgnucash/engine)
 set(test_foo_LIBS gncmod-engine)
 gnc_add_test(test-foo "${test_foo_SOURCES}" test_foo_INCLUDES test_foo_LIBS)

That gets the test built and run. We also need to make sure that the source files are included in the distribution tarball, so at the bottom of CMakeLists.txt we need:

 set(test_foodir_sources_DIST "${test_foo_SOURCES}")
 set(test_foodir_DIST CMakeLists.txt "${test_foodir_sources_DIST})

And in foodir/CMakeLists.txt:

  set(foodir_DIST "${foodir_local_DIST}" "${test_foodir_DIST}" PARENT_SCOPE)

Finally, don't forget to add all of your new files to git when you commit your test!

GLib-test based tests

Several of our C libraries have unit tests written using the GLib test framework. While GoogleTest is preferred for writing new tests for C++ code it may be useful to write tests using this framework for C code before converting it to C++ to help prevent breaking something. This is still unit testing so the general references above still apply.

To set up unit testing in a directory:

• Create a "test" directory if there isn't one already
• Create a CMakeLists.txt in that test directory, again if there isn't one already.
• The setup of of a GLib test program in CMakeLists.txt is the same as for a GoogleTest test, see above.
• Copy test-templates/test-module.c to your test directory, rename it, and create a target for it in CMakeLists.txt
• Run

 test-templates/make-testfile "Your Name <you@your.email.address>" path/to/source

passing the path to the source file you want to write tests for. This will create a template test file for you with all of the necessary functions prototyped and commented out and a populated test suite function with the individual tests commented out.

• Uncomment the definition for a function that you want to test and write your test function. Write Setup and Teardown functions as needed. Uncomment the execution line for the test function, adjusting the setup and teardown function names as necessary, in the test suite function at the end of the file.

There is a unit test support module with some useful functions for controlling logging and signals in common/test-core. If you use it, add

 test-core

to your test program's LIBS variable or add

 ${CMAKE_SOURCE_DIR}/test-core/unittest-support.c

to the SOURCES variable; in either case add

 #include <unittest-support.h>

in your test-suite file. You have two options for the actual test programs. You can write a bunch of separate programs with a few tests each or you can group several files containing tests into a single program.

Many Little Programs

• Make a copy of test-templates/testmain.c for each program, renaming it appriopriately, in your test directory.
• Create fixtures and test functions and register the test functions in main(); there are comments in the file to guide you.
• Set up a target in Makefile.am as described above for each program.

Test Suites

A test suite is a collection of test functions registered in a test-suite function; main() runs the test-suite functions. This makes it easier to group tests into separate files with a master test program file to contain main(). We'll call the master test program source file the module file; it's conventionally named after the directory it's testing, e.g. test-engine.c. Normally you'll have a test-suite for each source file in the directory named utest-filename.c, e.g. utest-Split.c.

Note that as the C++ conversion progresses it may be necessary to compile some of these files as C++ in which case the extension will be .cpp instead of .c.

Legacy Tests

In addition to GoogleTest and GLib Testing based tests GnuCash has several older tests based on a private framework. Most of these are module tests rather than unit tests and most don't do a good job of localizing test failures, both of which can make debugging test failures difficult. The framework isn't well documented so if you need to get into these tests you'll need to study the source code in test-stuff.h, test-stuff.c, test-engine-stuff.h, and test-engine-stuff.cpp.

Scheme Tests

We want to test Scheme as well. New tests should be written using SRFI-64. "Legacy" tests are just scheme forms, some are unit tests and some are larger integration tests. Until recently there was no standard style or form other than that they had to exit with 0 for success or something else for failure so that ctest can tell which pass and which fail.

Scheme tests are added to CMakeLists.txt with the command gnc_add_scheme_test; multiple tests can be added with gnc_add_scheme_tests. They don't create executables in CMAKE_BUILD_DIRECTORY/bin so to run one by itself use

 ctest -R label-regex

where label-regex is a Regular Expression not a Glob. For example

 ctest -R .*-barchart

will run all of the tests whose names end in -barchart. Adding a -V to the ctest options will send the output to stdout instead of CMAKE_BUILD_DIRECTORY/Testing/Temporary/LastTest.log

Test Coverage Status

Unit Tests

LibQOF

Done:

• qofbackend
• qofbook
• qofinstance
• qofobject
• qofsession
• gnc-date
• kvp_frame

Engine

Done:

• Account
• Split
• Transaction

Known Test Needs

• GtkAction callbacks referenced directly in GtkBuilder UI files need at a minimum "presence testing" so that make check will fail if the callbacks don't compile for some reason.

Literature Survey

• Best unit test tutorials for beginners

• Working Effectively with Legacy Code
  • Summary

• Test Patterns: Refactoring Test Code The skeleton of this excellent manual for writing and improving unit tests is online; there is a pointer there for purchasing the book as well. While the book focuses on xUnit-style test frameworks (meaning jUnit and its many derivatives) most of the principles and patterns are applicable to any unit test code.