Revision as of 23:30, 24 May 2014

A good set of tests is a critical requirement for modern software development, both to ensure the quality of the product and to help developers to quickly modify code without introducing bugs or causing regressions. The literature on software testing is vast; excellent tutorials and references are available both online and in print, see Literature Survey.

All developers are encouraged to review tests and ensure that any work is covered by both acceptance and unit tests.

Current Test Architecture

Acceptance Tests

Acceptance tests in Gnucash are based on a home-grown (or anonymously sourced) set of macros and functions which can be found in src/test-core. The quality, scope, and coverage of these tests varies greatly; some parts of GnuCash are not tested lightly or not at all, others are tested fairly extensively. While I've labelled the "acceptance tests", in many cases they're written more as unit tests whose scope is a single function rather than a whole module.

Unit Tests

Unit testing, invented by Kent Beck in the early 1990s, seeks to test all execution paths in single functions with as much isolation from the rest of the system as possible.

GnuCash has adopted the GLib testing framework to facilitate unit testing. Muslim Choclov wrote unit tests for the most important modules in LibQOF as a GSoC2011 project. Work continues to get all of LibQOF and the engine fully tested to facilitate major architectural changes needed to make GnuCash a proper database application.

See also https://developer.gnome.org/glib/stable/gtester.html for the actual command line tool to run tests.

Running Unit Tests

The collection of all implemented tests in gnucash is run by running

make check

This can be run in any subdirectory to limit the number of tests. If a directory has both acceptance and unit tests, one can further limit the testing to the unit tests by running

 make test

which you'd run also if you're working on unit tests and need to recompile. Note that make test doesn't always work outside of the test directory itself. You can also run the unit tests in a directory with the test program, and restrict the tests run with the appropriate suite-name or even test case:

test-engine -p /engine/Account

will run only the Account tests. The test program is really an alias for gtester; read the man page for more details and options, e.g. the --verbose argument.

Unit Test Policies

• Getter/Setter functions which only set or retrieve an instance member variable do not need to be tested.
• Convenience functions which only wrap another function to change the function's name or to provide a default argument do not require testing.
• Composed functions, or functions which simply string together a series of calls to other functions, need not be tested if the called functions are all tested, have no side effects, and where the composed function has only one flow of control.
• There is some disagreement among testing gurus about whether all execution paths in a function should be exercised in a single test function or separately in a test function per execution path. Use your judgement here. Remember that the dictum of Agile Development is to write a little bit at a time and to refactor as often as you need to. That applies as much to test code as it does to application code. It's OK to change your mind!
• Similarly there is tension among the gurus about how much to make a test program dependent upon, and how much to use mock objects to replace actual dependency code. Keeping in mind the goal of a short code-compile-test cycle, use your judgement. That said, at present much of Gnucash is rather interdependent and doesn't virtualize functions -- a requirement for applying mocks. If you're writing new modules, do use modern OO techniques to minimize interdependence, and where it's necessary make sure to use virtual functions so that linking the rest of Gnucash isn't needed to test your work.

Adding Unit Tests

To set up unit testing in a directory:

• Create a "test" directory if there isn't one already
• Create a Makefile.am in that test directory, again if there isn't one already.
• Add the line

 include $(top_srcdir)/test-templates/Makefile.decl

in the Makefile.am and in every parent directory's Makefile.am that doesn't already have that line. If a parent directory has more than one subdirectory and not all of them have unit tests, create or add to the variable TEST_SUBDIRS.

• You will be adding one or more test programs. These are declared by adding their names to the TEST_PROGS variable, then defining the usual automake variables for each. For example:

 TEST_PROGS += test-foo test-bar
 noinst_PROGRAMS = $(TEST_PROGS)
 
 test_foo_SOURCES = foo.c baz.c utest-foo.c
 test_foo_HEADERS = foo.h 
 test_foo_LDADD   = ../libgncmod-pepper.la
 (And so on, also for test_bar)

There is a test support module with some useful functions for controlling logging and signals in src/test-core. If you use it, add

 ${top_builddir}/src/test-core/libtest-core.la

to your test program's LDADD variable.

Test Coverage Status

Unit Tests

LibQOF

Done:

• qofbackend
• qofbook
• qofinstance
• qofobject
• qofsession
• gnc-date
• kvp_frame

Engine

Done:

• Account
• Split
• Transaction

Known Test Needs

• GtkAction callbacks referenced directly in GtkBuilder UI files need at a minimum "presence testing" so that make check will fail if the callbacks don't compile for some reason.

Literature Survey

• Best unit test tutorials for beginners

• Working Effectively with Legacy Code
  • Summary

• Test Patterns: Refactoring Test Code The skeleton of this excellent manual for writing and improving unit tests is online; there is a pointer there for purchasing the book as well. While the book focuses on xUnit-style test frameworks (meaning jUnit and its many derivatives) most of the principles and patterns are applicable to any unit test code.