serval-dna/doc/Testing.md

Serval DNA Testing
==================
[Serval Project][], June 2014

[Serval DNA][] is tested using a suite of [test scripts](../tests/) written in
the [Bash][] shell scripting language, using the Serval Project's own [Bash
Test Framework][].  These scripts are [integration tests][] focussed on the
Serval DNA component and its external interfaces.

Test Framework
--------------

The [Bash Test Framework][] performs common testing work, so that test
developers can focus on the specifics of their test cases and test cases
contain a minumum of [boilerplate code][]:

 * creates a temporary working directory to isolate each test case
 * invokes each test case's set-up, test, finalise, and tear-down functions in
   a defined order, guaranteeing to always call the latter two
 * provides a rich set of assertion functions
 * records the outcome of each test case: PASS, FAIL or ERROR
 * records a detailed log of the execution of each test case
 * removes temporary working directories and files after each test case
 * kills any stray processes after each test case
 * runs test cases in parallel if so directed
 * reports progress during execution

Some features that may be added in future are:

 * conformance with [Test Anything Protocol][TAP]
 * support for a SKIP test outcome
 * formal versioning of the Test Framework and parts of its API, to catch
   incompatibilities between test scripts and Framework upgrades

Prerequisites
-------------

The [Bash Test Framework][] requires the following execution environment:

 * [Bash][] version 3.2.48 or later
 * [GNU grep][] version 2.7 or later
 * [GNU sed][] version 4.2 or later
 * [GNU awk][] version 3.1 or later
 * [pgrep][] and [pkill][] version 593 or later (Solaris) or from procps-ng 3.3
   or later (Linux)

Before running any tests, all the executables and other artifacts under test
(ie, the **servald** executable), plus all test utilities, must be
[built](../INSTALL.md).

Test scripts
------------

Executing a test script without any arguments causes it to run all the test
cases that it defines, one at a time.  The script will terminate once all test
cases have been run, and its exit status will be zero only if all test cases
reported PASS.

Every test script uses the [Bash Test Framework][] to parse its command line,
so the following options are supported by all test scripts:

 * __`-l`__ or __`--list`__ causes the script to print a list of all its test
   cases on standard output, instead of executing them

 * __`-t`__ or __`--trace`__ sets the Bash `-x` option during execution of each
   test case, which adds much more detail to the test logs

 * __`-v`__ or __`--verbose`__ causes test logs to be sent to standard output
   during execution of the tests, so the developer can watch a test as it runs
   (this version is incompatible with running tests in parallel)

 * __`-E`__ or __`--stop-on-error`__ causes the test script to stop running new
   test cases as soon as any test reports ERROR, and to wait for currently
   running test cases to finish

 * __`-F`__ or __`--stop-on-failure`__ causes the test script to stop running
   new test cases as soon as any test reports FAIL, and to wait for currently
   running test cases to finish

 * __`-j N`__ or __`--jobs=N`__ causes up to __N__ test cases to be run
   concurrently, which can greatly speed the rate of completion of a large test
   run, since most tests spend much of their time either sleeping or i/o bound

 * __`-f PREFIX`__ or __`--filter=PREFIX`__ causes only those test cases whose
   names begin with __PREFIX__ to be executed

 * __`-f N`__ or __`--filter=N`__ causes only test case number __N__ to be
   executed (test cases are numbered in the order they are defined in the
   script)

 * __`-f M-N`__ or __`--filter=M-N`__ causes only test cases numbers __M__
   through to __N__ (inclusive) to be executed (test cases are numbered in the
   order they are defined in the script); if __M__ is omitted then all cases up
   to number __N__ are executed; if __N__ is omitted then all test cases from
   number __M__ and above are executed

 * __`-f M,N,...`__ or __`--filter=M,N,...`__ causes only test cases __M__ and
   __N__ (... etc.) to be executed (test cases are numbered in the order they
   are defined in the script)

There are other options as well.  To see a complete and up-to-date summary, use
the __`--help`__ option:

    $ ./tests/all --help

Aggregate scripts
-----------------

Some test scripts simply aggregate other scripts, providing a convenient way to
execute many tests with a single command.  Aggregate scripts behave in all
respects like a normal test script: the command line options and exit status
are the same.

The most notable aggregate script is [tests/all](../tests/all), which runs all
available tests except long-running, resource-hungry “stress” tests:

    $ ./tests/all
    1 [PASS.] (logging) By default, only errors and warnings are logged to stderr
    2 [PASS.] (logging) Configure all messages logged to stderr
    3 [PASS.] (logging) Configure no messages logged to stderr
    4 [PASS.] (logging) By Default, all messages are appended to a configured file
    ...
    158 [PASS.] (rhizomeprotocol) One way direct pull bundle from configured peer
    159 [PASS.] (rhizomeprotocol) Two-way direct sync bundles with configured peer
    160 [PASS.] (directory_service) Publish and retrieve a directory entry
    161 [PASS.] (directory_service) Ping via relay node
    161 tests, 161 pass, 0 fail, 0 error
    $

Test logs
---------

All test scripts write their test logs into the `testlog` sub-directory
(relative to the current working directory), which has the following structure:

    ./testlog/
        SCRIPTNAME/
            1.FirstTestCaseName.RESULT/
                log.txt
                ... other files...
            2.SecondTestCaseName.RESULT/
                log.txt
                ... other files...
        SECONDSCRIPTNAME/
            1.first_test_case_name.RESULT/
                log.txt
                ... other files...
            2.second_test_case_name.RESULT/
                log.txt
                ... other files...
        ... more script directories...

where `SCRIPTNAME` and `SECONDSCRIPTNAME` are the names of the test scripts,
`FirstTestCaseName`, `first_test_case_name`, etc. are the names of the tests
within those scripts, and `RESULT` is either `ERROR`, `FAIL` or `PASS`.  An
aggregate test script writes logfiles for all the test cases it includes under
its own SCRIPTNAME, not under the names of the scripts it includes.

Whenever a test script starts, it deletes its `testlog/SCRIPTNAME` directory
and all its contents, so the logs from previous runs are lost.

Every test case produces a `log.txt` file, and may also produce other files to
assist diagnosis in case of failure or to supplement a pass result, eg,
performance statistics, code coverage data, network packet logs for
reproducibility.

Source code coverage
--------------------

The [Bash Test Framework][] has command-line options to support per-test-case
[source code test coverage][] analysis using [GCC][] and [gcov(1)][].  An
aggregate coverage analysis can easily be generated with no special options to
test scripts.

To generate code coverage information for [Serval DNA][], modify the standard
[build](../INSTALL.md) procedure by adding CFLAGS and LDFLAGS arguments to the
`./configure` step:

    ...
    $ ./configure CFLAGS='-g -O0 --coverage' LDFLAGS='--coverage'
    $ make
    ...

This will generate one [GCNO][] file for every object file, in the same
directory as the object file.

Once **servald** has been built using these flags, invoking it will generate
some [GCDA][] coverage data files, one per source file, in the same directory
as the [GCNO][] files.  Repeated invocations will accumulate coverage data in
the same files.  The environment variables `GCOV_PREFIX` and
`GCOV_PREFIX_STRIP` can be used to change the directory where the [GCDA][] data
files are written.

### Aggregate code coverage

To generate aggregate code coverage for a test run:

    $ make covzero
    $ ./tests/all
    ...
    $ make covhtml
    $ www-browser ./coverage_html/index.html
    ...

The coverage report will reflect exactly the accumulated coverage of all tests
run between `make covzero` and `make covhtml`.  The above example runs all
tests (except stress tests) but any combination may be run, including manual
invocations of **servald**.  The **servald** executable must be invoked at
least once after `make covzero`, or `make covhtml` will fail with an error, for
lack of coverage data.

If more tests are run without invoking `make covzero`, then the coverage data
will sum with the existing coverage data since the last `make covzero`.

### Per-test-case code coverage

**Note**: Per-test-case coverage support is of very limited use because of
deficiencies in the coverage data processing utilities (see below).

If the __`--coverage`__ option is given to a test script, then it sets the
`GCOV_PREFIX` and `GCOV_PREFIX_STRIP` environment variables while running each
test case, causing each case's generated [GCDA][] coverage data files to be
created under the case's own log directory:

    ./testlog/
        SCRIPTNAME/
            N.TestCaseName.RESULT/
                log.txt
                gcov/
                    home/username/src/serval-dna/objs_servald/cli.gcda
                    home/username/src/serval-dna/objs_servald/commandline.gcda
                    ...
                    home/username/src/serval-dna/objs_servald/nacl/src/crypto_auth_hmacsha256_ref/hmac.c
                    ...

In theory, these per-test-case [GCDA][] data files could be merged to produce
coverage data for any desired combination of test cases, but there is currently
no command-line utility available to perform this merge.  The code for merging
undoubtably exists in the *libgcov* [atexit(3)][] callback, which sums the
process's accumulated execution counts into any existing [GCDA][] files, but no
work has been done to extract this code into a utility.

If the __`--geninfo`__ option is given (which implies `--coverage`), the test
framework will invoke [geninfo][] after each test case completes, to generate
one [lcov][] *tracefile* per case named `coverage.info` located in the case's
own log directory:

    ./testlog/
        SCRIPTNAME/
            N.TestCaseName.RESULT/
                log.txt
                coverage.info

**Note**: The `--geninfo` option must be accompanied by at least one
__`--gcno-dir=PATH`__ option, or the `TFW_GCNO_PATH` environment variable must
be set to a list of colon-separated directory paths.  The test framework
recursively searches all these directories looking for [GCNO][] files, which it
then supplies to [geninfo][], which uses them to find the source files and
[GCDA][] files produced by `--coverage`.

The per-test-case tracefiles produced by [geninfo][] may be merged together
using the `lcov --add-tracefile` option, and may also be combined into a single
coverage report by passing many tracefile arguments to the [genhtml][] utility.
Unfortunately, both of these operations are prohibitively slow, which makes the
`--geninfo` option of limited use for the time being.

-----
**Copyright 2013 Serval Project Inc.**
![CC-BY-4.0](./cc-by-4.0.png)
This document is available under the [Creative Commons Attribution 4.0 International licence][CC BY 4.0].


[Serval Project]: http://www.servalproject.org/
[CC BY 4.0]: ./LICENSE-DOCUMENTATION.md
[Serval DNA]: ../README.md
[Bash]: http://en.wikipedia.org/wiki/Bash_(Unix_shell)
[Bash Test Framework]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:bash_test_framework
[GNU grep]: http://www.gnu.org/software/grep/
[GNU sed]: http://www.gnu.org/software/sed/
[GNU awk]: http://www.gnu.org/software/gawk/
[pgrep]: http://en.wikipedia.org/wiki/Pgrep
[pkill]: http://en.wikipedia.org/wiki/Pkill
[integration tests]: http://en.wikipedia.org/wiki/Integration_testing
[boilerplate code]: http://en.wikipedia.org/wiki/Boilerplate_code
[TAP]: http://en.wikipedia.org/wiki/Test_Anything_Protocol
[source code test coverage]: http://en.wikipedia.org/wiki/Code_coverage
[GCC]: https://gcc.gnu.org/
[gcov(1)]: https://gcc.gnu.org/onlinedocs/gcc/Gcov.html
[GCNO]: https://gcc.gnu.org/onlinedocs/gcc-3.4.2/gcc/Gcov-Data-Files.html
[GCDA]: https://gcc.gnu.org/onlinedocs/gcc-3.4.2/gcc/Gcov-Data-Files.html
[lcov]: http://ltp.sourceforge.net/archive/old_pages/coverage/lcov.php
[geninfo]: http://ltp.sourceforge.net/coverage/lcov/geninfo.1.php
[genhtml]: http://ltp.sourceforge.net/coverage/lcov/genhtml.1.php
[atexit(3)]: http://man7.org/linux/man-pages/man3/atexit.3.html