Commit Graph

107 Commits

Author SHA1 Message Date
Andrew Bettison
6824977800 Improve test framework: begin/end_fixture
Two new functions to allow parts of a test case test_xxx() function
to be treated as setup, so that a failure will result in an ERROR
not a FAIL
2015-03-02 18:49:50 +10:30
Andrew Bettison
188a67d3c1 Improve test framework: tfw_shopt, tfs_shopt_restore 2014-06-30 11:11:01 +09:30
Andrew Bettison
4fbaf8865a Improve test framework: create_file
create_file '-' sends to standard output
create_file --label=LABEL
create_file passes other options to tfw_createfile
2014-06-19 09:52:17 +09:30
Andrew Bettison
3d537f9645 Improve test framework: test coverage support 2014-06-06 14:47:54 +09:30
Andrew Bettison
13cf8b6209 Improve test framework: command-line options
Now supports -jN for --jobs=N, -fPRE for --filter=PRE, etc.
Reformatted usage message
2014-06-06 12:36:06 +09:30
Andrew Bettison
64237e4ab4 Improve test framework: return after fail()
So that functions can be used during teardown()
2014-05-22 15:32:16 +09:30
Jeremy Lakeman
342572730b Don't busy wait for forked processes to terminate 2014-05-12 13:44:50 +09:30
Andrew Bettison
fd8195fa4f Improve test framework: execute --error-on-fail 2014-05-02 13:45:31 +09:30
Andrew Bettison
24ed57c19b Improve test framework: execute --stdout-file=PATH
Also execute --stderr-file=PATH
2014-04-30 17:12:25 +09:30
Andrew Bettison
5035abde34 Improve test framework: assert_fork_is_running
Also assert_fork_is_not_running
Added 'framework' test case for fork primitives
2014-04-30 11:10:20 +09:30
Andrew Bettison
525aee88c5 Fix test framework bug: needed shopt -s extglob 2014-04-30 11:08:53 +09:30
Andrew Bettison
78d0e29768 Improve test framework: shell backtrace from forks
Was not showing the point where the fork() was invoked
2014-04-29 15:09:18 +09:30
Andrew Bettison
08e02b18db Improve test framework: tfw_run() function 2014-03-03 15:14:34 +10:30
Andrew Bettison
5093bff12a Improve test framework: sleep(1) arguments
Always pass exactly one argument to sleep(1) since the BSD sleep (used
on OSX) does not sum its arguments like GNU sleep(1)

Perform more stringent checking on various timeout parameters to ensure
they are well-formed decimal floats
2014-02-17 17:11:31 +10:30
Andrew Bettison
135df30b1b Improve test framework: assertGrep --ignore-case 2013-12-19 19:14:52 +10:30
Jeremy Lakeman
3b14543750 Sleep while waiting for forked processes to complete 2013-12-09 11:22:44 +10:30
Andrew Bettison
4f0aa3a41c Improve test framework: include %label in fork log 2013-12-02 16:07:37 +10:30
Andrew Bettison
9953949c6e Improve test framework: forked commands inherit stdin
Remove the subshell parentheses around the command invoked using fork(),
so that the forked command inherits the file descriptors.  This allows
it to be used like "fork some command < input".
2013-12-02 15:56:32 +10:30
Andrew Bettison
1d14f43050 Improve test framework: --timeout option in usage message 2013-12-01 05:13:33 +10:30
Andrew Bettison
874a3a9ef7 Improve test framework: shell backtraces
When in a the finalise part of a test (possibly as a result of a
failure or error), make sure that shell backtraces do not list the
test case functions that failed, since they have already been
reported in a backtrace at the point of failure
2013-11-20 15:03:48 +10:30
Andrew Bettison
c5014baf62 Improve test framework: better fork functions
Replace forkKillAll with fork_terminate_all

Replace forkWaitAll with fork_wait_all

Add fork_terminate and fork_wait functions

Introduce fork labels '%<alphanumeric>' as arguments to all fork
functions

Treat a forked process exit status 143 (killed with SIGTERM) as a
failure or error -- it happens when a forked process is terminated
using fork_terminate or fork_terminate_all
2013-11-20 15:01:37 +10:30
Andrew Bettison
8ca34749de Improve test framework: slightly clearer logging 2013-11-19 17:27:25 +10:30
Andrew Bettison
a90eced640 Fix test framework: bug in assertStdoutIs() et al
Could overwrite the latest stdout or stderr file produced by execute()
2013-11-06 23:56:33 +10:30
Andrew Bettison
7caebb97be Cosmetic changes to test framework
Re-order function definitions to group them more logically, and add a
few more comments.  No functional change whatsoever.
2013-11-04 17:40:02 +10:30
Andrew Bettison
34c5522e48 Improve test framework: tfw_cat --hexdump, TFWSTDOUT
Add -h|--hexdump option to tfw_cat() and fix faulty logic in tfw_cat()
-v option wof files not ending in \n (newline)

Add TFWSTDOUT and TFWSTDERR env vars set by execute() with absolute
path names of files containing stdout and stderr respectively (as an
alternative to using replayStdout() and replayStderr()).
2013-11-04 17:40:02 +10:30
Andrew Bettison
b1b5a79f6f Improve test framework: timeout on executing commands
Default timeout is 60 seconds, can be overridden by --timeout=N on command-line
or by TFW_EXECUTE_TIMEOUT var or by --timeout option to execute() primitive

Cull timeout sleep processes created by wait_until() primitive
2013-11-04 17:40:01 +10:30
Andrew Bettison
ef2cc1404c Improve test framework: tfw_cmp_version()
Also add unit tests for test framework itself
2013-10-14 16:45:56 +10:30
Andrew Bettison
55efc73b7e Improve test framework: assertGrep --fixed-strings option 2013-09-05 09:19:14 +09:30
Andrew Bettison
3164172c08 Work around broken Vim syntax highlighting 2013-08-30 11:20:19 +09:30
Andrew Bettison
80280de998 Improve test framework: set TFWSOURCE env var 2013-04-29 15:10:16 +09:30
Andrew Bettison
226fcf3754 Improve test framework: fix some minor awk bugs 2013-03-06 13:08:10 +10:30
Andrew Bettison
6f3c2b99c9 Improve test framework: Solaris bug
The 'includeTests' function failed if /bin/bash was not available or not
the same as the Bash used to run the top test script.  This failed on
Solaris, where /usr/bin/bash is too old, and a privately-built Bash must
be used to run the test scripts.
2013-03-06 11:45:12 +10:30
Andrew Bettison
b8e32bdd66 Improve test framework: tfw_quietly, tfw_multicolumn
New tfw_quieten() is similar to tfw_nolog(), but only silences the chatty log
output for asserts and other actions that succeed.  Failures are still logged.

New tfw_multicolumn() encapsulates an awk(1) script to format a list into
columns that fit into $COLUMNS width.

Improve exit status handling for test ERROR and FAIL conditions.
2013-02-11 18:18:04 +10:30
Andrew Bettison
3e2b1524bb Improve test framework: fork concurrent processes
Add fork(), forkKillAll() and forkWaitAll() framework primitives for running
many concurrent background processes within a single test case.
2013-02-08 18:46:30 +10:30
Andrew Bettison
2ca7bab7c0 Improve test framework: create_file --append 2013-02-08 12:07:56 +10:30
Andrew Bettison
b3a0b0e965 Improve test framework: fix Mac OS X bugs
Mac OS X sort(1) does not support -V (version number compare), and sed(1) does
not support the \+ regex operator.
2013-02-07 17:24:28 +10:30
Andrew Bettison
819e9bb8b2 Improve test framework: fix job control bugs
The non-deterministic bug of "job %N terminated without result" is now fixed.
The cause was that Bash job numbers are allocated as max(current job numbers) +
1, so if the most recent job exited was not harvested before the next job was
started, then the job number would get re-used, deleting the un-harvested job's
result file.

SIGINT (ctrl-C interrupt) handling has been improved for the -j (parallel) and
non-parallel cases.

Test case execution in the non-parallel case is faster, by fixing a bug that
waited for a one-second timeout before starting each test job.
2013-02-07 17:06:30 +10:30
Andrew Bettison
e01b2cacd8 Improve test framework: duplicate test names
Now that tests can be included from other scripts, there is a possibility of
test name collision, so the main loop logic now uses the test sequence number
instead of the name internally to identify the test.
2013-02-07 09:44:44 +10:30
Andrew Bettison
3e35cf33d2 Improve test framework: includeTests() function
Support for test scripts to aggregate other test scripts.
2013-02-06 18:49:29 +10:30
Andrew Bettison
ac18b895e3 Improve test framework: better progress output 2013-01-31 13:52:57 +11:00
Andrew Bettison
7b984bbd92 Improve test framework: one log directory per test case
Log file is now in testlog/SCRIPTNAME/N.TESTNAME.RESULT/log.text

Add tfw_preserve() function to copy files/directories into log directory
2013-01-31 12:39:46 +11:00
Andrew Bettison
b77df878c1 Improve test framework: wait_until [ ! ]* command 2013-01-10 12:11:24 +10:30
Andrew Bettison
b8de07679d Improve test framework: --line=N..M for assertGrep()
Support the --line option for assertGrep, assertStdoutGrep and assertStderrGrep
primitives to select a line or range of lines to which the grep will be
applied.
2012-11-07 20:27:11 +10:30
Andrew Bettison
2b5b7fa896 Improve test framework: tfw_nolog() 2012-11-05 14:43:07 +10:30
Andrew Bettison
1ee62b7587 Fix bug in test framework: local $njobs used by other function 2012-11-05 12:07:44 +10:30
Andrew Bettison
03f71dcccf Improve rhizome stress test
Generate test files from 2KiB to 32MiB in size, instead of all about 13 bytes
in size.  Guess what?  Found a bug!
2012-10-31 18:17:06 +10:30
Andrew Bettison
d38ce66c95 Improve test framework: create_file()
New executable in Makefile 'all' target: tfw_createfile.

New str.h function: str_to_ll_scaled().
2012-10-31 18:13:33 +10:30
Andrew Bettison
ed44dfe720 Improve test framework: finally() function
If supplied, the finally_TestCaseName() function, otherwise the finally()
function, is called after the test_TestCaseName() function terminates (with any
result).  An assertion failure in the finally() function causes the test to
FAIL, but execution continues.  So it is like teardown() but produces FAIL not
ERROR.  Useful for clean-up that must be performed as part of the system under
test.
2012-10-18 11:38:44 +10:30
Andrew Bettison
7317f9b56c Add a new 'rhizomeprotocol' test case: stress test
Refactored some testdefs.sh and testdefs_rhizome.sh, not yet complete.
2012-10-17 18:10:39 +10:30
Andrew Bettison
d8f06d0582 Improve test framework: tfw_cat and shell backtrace
tfw_cat() now behaves better if the file is missing a final newline.

Shell backtrace lines no longer prefixed with log prefix (timestamp).
2012-09-28 15:30:47 +09:30