Commit Graph

94 Commits

Author SHA1 Message Date
Andrew Bettison
5093bff12a Improve test framework: sleep(1) arguments
Always pass exactly one argument to sleep(1) since the BSD sleep (used
on OSX) does not sum its arguments like GNU sleep(1)

Perform more stringent checking on various timeout parameters to ensure
they are well-formed decimal floats
2014-02-17 17:11:31 +10:30
Andrew Bettison
135df30b1b Improve test framework: assertGrep --ignore-case 2013-12-19 19:14:52 +10:30
Jeremy Lakeman
3b14543750 Sleep while waiting for forked processes to complete 2013-12-09 11:22:44 +10:30
Andrew Bettison
4f0aa3a41c Improve test framework: include %label in fork log 2013-12-02 16:07:37 +10:30
Andrew Bettison
9953949c6e Improve test framework: forked commands inherit stdin
Remove the subshell parentheses around the command invoked using fork(),
so that the forked command inherits the file descriptors.  This allows
it to be used like "fork some command < input".
2013-12-02 15:56:32 +10:30
Andrew Bettison
1d14f43050 Improve test framework: --timeout option in usage message 2013-12-01 05:13:33 +10:30
Andrew Bettison
874a3a9ef7 Improve test framework: shell backtraces
When in a the finalise part of a test (possibly as a result of a
failure or error), make sure that shell backtraces do not list the
test case functions that failed, since they have already been
reported in a backtrace at the point of failure
2013-11-20 15:03:48 +10:30
Andrew Bettison
c5014baf62 Improve test framework: better fork functions
Replace forkKillAll with fork_terminate_all

Replace forkWaitAll with fork_wait_all

Add fork_terminate and fork_wait functions

Introduce fork labels '%<alphanumeric>' as arguments to all fork
functions

Treat a forked process exit status 143 (killed with SIGTERM) as a
failure or error -- it happens when a forked process is terminated
using fork_terminate or fork_terminate_all
2013-11-20 15:01:37 +10:30
Andrew Bettison
8ca34749de Improve test framework: slightly clearer logging 2013-11-19 17:27:25 +10:30
Andrew Bettison
a90eced640 Fix test framework: bug in assertStdoutIs() et al
Could overwrite the latest stdout or stderr file produced by execute()
2013-11-06 23:56:33 +10:30
Andrew Bettison
7caebb97be Cosmetic changes to test framework
Re-order function definitions to group them more logically, and add a
few more comments.  No functional change whatsoever.
2013-11-04 17:40:02 +10:30
Andrew Bettison
34c5522e48 Improve test framework: tfw_cat --hexdump, TFWSTDOUT
Add -h|--hexdump option to tfw_cat() and fix faulty logic in tfw_cat()
-v option wof files not ending in \n (newline)

Add TFWSTDOUT and TFWSTDERR env vars set by execute() with absolute
path names of files containing stdout and stderr respectively (as an
alternative to using replayStdout() and replayStderr()).
2013-11-04 17:40:02 +10:30
Andrew Bettison
b1b5a79f6f Improve test framework: timeout on executing commands
Default timeout is 60 seconds, can be overridden by --timeout=N on command-line
or by TFW_EXECUTE_TIMEOUT var or by --timeout option to execute() primitive

Cull timeout sleep processes created by wait_until() primitive
2013-11-04 17:40:01 +10:30
Andrew Bettison
ef2cc1404c Improve test framework: tfw_cmp_version()
Also add unit tests for test framework itself
2013-10-14 16:45:56 +10:30
Andrew Bettison
55efc73b7e Improve test framework: assertGrep --fixed-strings option 2013-09-05 09:19:14 +09:30
Andrew Bettison
3164172c08 Work around broken Vim syntax highlighting 2013-08-30 11:20:19 +09:30
Andrew Bettison
80280de998 Improve test framework: set TFWSOURCE env var 2013-04-29 15:10:16 +09:30
Andrew Bettison
226fcf3754 Improve test framework: fix some minor awk bugs 2013-03-06 13:08:10 +10:30
Andrew Bettison
6f3c2b99c9 Improve test framework: Solaris bug
The 'includeTests' function failed if /bin/bash was not available or not
the same as the Bash used to run the top test script.  This failed on
Solaris, where /usr/bin/bash is too old, and a privately-built Bash must
be used to run the test scripts.
2013-03-06 11:45:12 +10:30
Andrew Bettison
b8e32bdd66 Improve test framework: tfw_quietly, tfw_multicolumn
New tfw_quieten() is similar to tfw_nolog(), but only silences the chatty log
output for asserts and other actions that succeed.  Failures are still logged.

New tfw_multicolumn() encapsulates an awk(1) script to format a list into
columns that fit into $COLUMNS width.

Improve exit status handling for test ERROR and FAIL conditions.
2013-02-11 18:18:04 +10:30
Andrew Bettison
3e2b1524bb Improve test framework: fork concurrent processes
Add fork(), forkKillAll() and forkWaitAll() framework primitives for running
many concurrent background processes within a single test case.
2013-02-08 18:46:30 +10:30
Andrew Bettison
2ca7bab7c0 Improve test framework: create_file --append 2013-02-08 12:07:56 +10:30
Andrew Bettison
b3a0b0e965 Improve test framework: fix Mac OS X bugs
Mac OS X sort(1) does not support -V (version number compare), and sed(1) does
not support the \+ regex operator.
2013-02-07 17:24:28 +10:30
Andrew Bettison
819e9bb8b2 Improve test framework: fix job control bugs
The non-deterministic bug of "job %N terminated without result" is now fixed.
The cause was that Bash job numbers are allocated as max(current job numbers) +
1, so if the most recent job exited was not harvested before the next job was
started, then the job number would get re-used, deleting the un-harvested job's
result file.

SIGINT (ctrl-C interrupt) handling has been improved for the -j (parallel) and
non-parallel cases.

Test case execution in the non-parallel case is faster, by fixing a bug that
waited for a one-second timeout before starting each test job.
2013-02-07 17:06:30 +10:30
Andrew Bettison
e01b2cacd8 Improve test framework: duplicate test names
Now that tests can be included from other scripts, there is a possibility of
test name collision, so the main loop logic now uses the test sequence number
instead of the name internally to identify the test.
2013-02-07 09:44:44 +10:30
Andrew Bettison
3e35cf33d2 Improve test framework: includeTests() function
Support for test scripts to aggregate other test scripts.
2013-02-06 18:49:29 +10:30
Andrew Bettison
ac18b895e3 Improve test framework: better progress output 2013-01-31 13:52:57 +11:00
Andrew Bettison
7b984bbd92 Improve test framework: one log directory per test case
Log file is now in testlog/SCRIPTNAME/N.TESTNAME.RESULT/log.text

Add tfw_preserve() function to copy files/directories into log directory
2013-01-31 12:39:46 +11:00
Andrew Bettison
b77df878c1 Improve test framework: wait_until [ ! ]* command 2013-01-10 12:11:24 +10:30
Andrew Bettison
b8de07679d Improve test framework: --line=N..M for assertGrep()
Support the --line option for assertGrep, assertStdoutGrep and assertStderrGrep
primitives to select a line or range of lines to which the grep will be
applied.
2012-11-07 20:27:11 +10:30
Andrew Bettison
2b5b7fa896 Improve test framework: tfw_nolog() 2012-11-05 14:43:07 +10:30
Andrew Bettison
1ee62b7587 Fix bug in test framework: local $njobs used by other function 2012-11-05 12:07:44 +10:30
Andrew Bettison
03f71dcccf Improve rhizome stress test
Generate test files from 2KiB to 32MiB in size, instead of all about 13 bytes
in size.  Guess what?  Found a bug!
2012-10-31 18:17:06 +10:30
Andrew Bettison
d38ce66c95 Improve test framework: create_file()
New executable in Makefile 'all' target: tfw_createfile.

New str.h function: str_to_ll_scaled().
2012-10-31 18:13:33 +10:30
Andrew Bettison
ed44dfe720 Improve test framework: finally() function
If supplied, the finally_TestCaseName() function, otherwise the finally()
function, is called after the test_TestCaseName() function terminates (with any
result).  An assertion failure in the finally() function causes the test to
FAIL, but execution continues.  So it is like teardown() but produces FAIL not
ERROR.  Useful for clean-up that must be performed as part of the system under
test.
2012-10-18 11:38:44 +10:30
Andrew Bettison
7317f9b56c Add a new 'rhizomeprotocol' test case: stress test
Refactored some testdefs.sh and testdefs_rhizome.sh, not yet complete.
2012-10-17 18:10:39 +10:30
Andrew Bettison
d8f06d0582 Improve test framework: tfw_cat and shell backtrace
tfw_cat() now behaves better if the file is missing a final newline.

Shell backtrace lines no longer prefixed with log prefix (timestamp).
2012-09-28 15:30:47 +09:30
Daniel O'Connor
741ffd0d2d Nuke debug message 2012-09-06 17:28:40 +09:30
Daniel O'Connor
7a06f55078 SHELL is not set by the shell, use BASH which is set by bash. 2012-09-06 17:19:51 +09:30
Daniel O'Connor
9f22073ab6 Detect Solaris and use GNU versions of awk, sed and grep.
Don't use %N as a date format unless we're on Linux.
Avoid using GNUisms with cmp & sort.
2012-09-06 16:41:24 +09:30
Paul Gardner-Stephen
2dfc056354 Add absapath function since Solaris lacks it.
Tests still don't work due to ancient bash though.
2012-09-06 11:55:49 +09:30
Andrew Bettison
9149c80297 Improve test framework: assert '!' command...
The assert command now supports leading '!' args in its command, each negating
the sense of the following command.
2012-08-30 09:33:15 +09:30
Andrew Bettison
25fbd6ffe0 Improve test framework: colours on Max OS X
Fixes #7.
2012-08-24 16:30:41 +09:30
Andrew Bettison
3a6b424c2d Fix test framework: job tracking on Mac OS X
Caused by difference in basic regular expressions supported by sed(1) on Mac
and on Linux: Mac does not support the \+ repeater, have to use \{1,\} instead.
2012-08-24 16:02:21 +09:30
Daniel O'Connor
d793dbc948 Restore testframework.sh that I previously rewound. 2012-08-24 15:36:17 +09:30
Daniel O'Connor
0dd2145ed4 assert that the port number matches between calls to overlay_interface_init_any() 2012-08-24 15:21:23 +09:30
Andrew Bettison
3d82983888 Improve test framework: better log messages from assertGrep 2012-08-20 18:43:47 +09:30
Andrew Bettison
a14c8427cb Improve test framework: kill residual processes 2012-08-20 11:28:03 +09:30
Andrew Bettison
d678680d40 Improve test framework: assertExpr '~' operator
The '~' (regular expression match) operator does not implicitly anchor the
expression to the ends of the text, ie, does not put ^...$ around the pattern.
That leaves the test writer free to specify a full or anchored or interior
match.
2012-08-17 17:56:42 +09:30
Andrew Bettison
9310a15cbd Improve test framework: better output from tfw_cat 2012-08-16 16:58:41 +09:30