Commit Graph

75 Commits

Author SHA1 Message Date
Andrew Bettison
b8e32bdd66 Improve test framework: tfw_quietly, tfw_multicolumn
New tfw_quieten() is similar to tfw_nolog(), but only silences the chatty log
output for asserts and other actions that succeed.  Failures are still logged.

New tfw_multicolumn() encapsulates an awk(1) script to format a list into
columns that fit into $COLUMNS width.

Improve exit status handling for test ERROR and FAIL conditions.
2013-02-11 18:18:04 +10:30
Andrew Bettison
3e2b1524bb Improve test framework: fork concurrent processes
Add fork(), forkKillAll() and forkWaitAll() framework primitives for running
many concurrent background processes within a single test case.
2013-02-08 18:46:30 +10:30
Andrew Bettison
2ca7bab7c0 Improve test framework: create_file --append 2013-02-08 12:07:56 +10:30
Andrew Bettison
b3a0b0e965 Improve test framework: fix Mac OS X bugs
Mac OS X sort(1) does not support -V (version number compare), and sed(1) does
not support the \+ regex operator.
2013-02-07 17:24:28 +10:30
Andrew Bettison
819e9bb8b2 Improve test framework: fix job control bugs
The non-deterministic bug of "job %N terminated without result" is now fixed.
The cause was that Bash job numbers are allocated as max(current job numbers) +
1, so if the most recent job exited was not harvested before the next job was
started, then the job number would get re-used, deleting the un-harvested job's
result file.

SIGINT (ctrl-C interrupt) handling has been improved for the -j (parallel) and
non-parallel cases.

Test case execution in the non-parallel case is faster, by fixing a bug that
waited for a one-second timeout before starting each test job.
2013-02-07 17:06:30 +10:30
Andrew Bettison
e01b2cacd8 Improve test framework: duplicate test names
Now that tests can be included from other scripts, there is a possibility of
test name collision, so the main loop logic now uses the test sequence number
instead of the name internally to identify the test.
2013-02-07 09:44:44 +10:30
Andrew Bettison
3e35cf33d2 Improve test framework: includeTests() function
Support for test scripts to aggregate other test scripts.
2013-02-06 18:49:29 +10:30
Andrew Bettison
ac18b895e3 Improve test framework: better progress output 2013-01-31 13:52:57 +11:00
Andrew Bettison
7b984bbd92 Improve test framework: one log directory per test case
Log file is now in testlog/SCRIPTNAME/N.TESTNAME.RESULT/log.text

Add tfw_preserve() function to copy files/directories into log directory
2013-01-31 12:39:46 +11:00
Andrew Bettison
b77df878c1 Improve test framework: wait_until [ ! ]* command 2013-01-10 12:11:24 +10:30
Andrew Bettison
b8de07679d Improve test framework: --line=N..M for assertGrep()
Support the --line option for assertGrep, assertStdoutGrep and assertStderrGrep
primitives to select a line or range of lines to which the grep will be
applied.
2012-11-07 20:27:11 +10:30
Andrew Bettison
2b5b7fa896 Improve test framework: tfw_nolog() 2012-11-05 14:43:07 +10:30
Andrew Bettison
1ee62b7587 Fix bug in test framework: local $njobs used by other function 2012-11-05 12:07:44 +10:30
Andrew Bettison
03f71dcccf Improve rhizome stress test
Generate test files from 2KiB to 32MiB in size, instead of all about 13 bytes
in size.  Guess what?  Found a bug!
2012-10-31 18:17:06 +10:30
Andrew Bettison
d38ce66c95 Improve test framework: create_file()
New executable in Makefile 'all' target: tfw_createfile.

New str.h function: str_to_ll_scaled().
2012-10-31 18:13:33 +10:30
Andrew Bettison
ed44dfe720 Improve test framework: finally() function
If supplied, the finally_TestCaseName() function, otherwise the finally()
function, is called after the test_TestCaseName() function terminates (with any
result).  An assertion failure in the finally() function causes the test to
FAIL, but execution continues.  So it is like teardown() but produces FAIL not
ERROR.  Useful for clean-up that must be performed as part of the system under
test.
2012-10-18 11:38:44 +10:30
Andrew Bettison
7317f9b56c Add a new 'rhizomeprotocol' test case: stress test
Refactored some testdefs.sh and testdefs_rhizome.sh, not yet complete.
2012-10-17 18:10:39 +10:30
Andrew Bettison
d8f06d0582 Improve test framework: tfw_cat and shell backtrace
tfw_cat() now behaves better if the file is missing a final newline.

Shell backtrace lines no longer prefixed with log prefix (timestamp).
2012-09-28 15:30:47 +09:30
Daniel O'Connor
741ffd0d2d Nuke debug message 2012-09-06 17:28:40 +09:30
Daniel O'Connor
7a06f55078 SHELL is not set by the shell, use BASH which is set by bash. 2012-09-06 17:19:51 +09:30
Daniel O'Connor
9f22073ab6 Detect Solaris and use GNU versions of awk, sed and grep.
Don't use %N as a date format unless we're on Linux.
Avoid using GNUisms with cmp & sort.
2012-09-06 16:41:24 +09:30
Paul Gardner-Stephen
2dfc056354 Add absapath function since Solaris lacks it.
Tests still don't work due to ancient bash though.
2012-09-06 11:55:49 +09:30
Andrew Bettison
9149c80297 Improve test framework: assert '!' command...
The assert command now supports leading '!' args in its command, each negating
the sense of the following command.
2012-08-30 09:33:15 +09:30
Andrew Bettison
25fbd6ffe0 Improve test framework: colours on Max OS X
Fixes #7.
2012-08-24 16:30:41 +09:30
Andrew Bettison
3a6b424c2d Fix test framework: job tracking on Mac OS X
Caused by difference in basic regular expressions supported by sed(1) on Mac
and on Linux: Mac does not support the \+ repeater, have to use \{1,\} instead.
2012-08-24 16:02:21 +09:30
Daniel O'Connor
d793dbc948 Restore testframework.sh that I previously rewound. 2012-08-24 15:36:17 +09:30
Daniel O'Connor
0dd2145ed4 assert that the port number matches between calls to overlay_interface_init_any() 2012-08-24 15:21:23 +09:30
Andrew Bettison
3d82983888 Improve test framework: better log messages from assertGrep 2012-08-20 18:43:47 +09:30
Andrew Bettison
a14c8427cb Improve test framework: kill residual processes 2012-08-20 11:28:03 +09:30
Andrew Bettison
d678680d40 Improve test framework: assertExpr '~' operator
The '~' (regular expression match) operator does not implicitly anchor the
expression to the ends of the text, ie, does not put ^...$ around the pattern.
That leaves the test writer free to specify a full or anchored or interior
match.
2012-08-17 17:56:42 +09:30
Andrew Bettison
9310a15cbd Improve test framework: better output from tfw_cat 2012-08-16 16:58:41 +09:30
Andrew Bettison
3c736b1f6c Improve test framework: orderly shutdown on signal
Uses bash job control internally instead of PIDs, because job control
allows all processes in a job to be killed.

Changes the way _tfw_shopt/_tfw_shopt_restore work, to avoid "unrestored shopt"
errors on interrupt.
2012-08-16 12:17:15 +09:30
Andrew Bettison
e1fb64e9d4 Improve test framework: assertExpr
Change internal implementation of assertExpr to use shell commands test(1) and
grep(1) instead of awk(1), so that we can write 'assertExpr "$var" '~' "$rexp"
where $rexp is a grep regular expression.  (awk(1) does not support \{N,M\} in
regular expressions.)
2012-08-15 18:46:57 +09:30
Andrew Bettison
18424a4f1e Improve test framework: expose shellarg() function 2012-08-02 15:42:18 +09:30
Andrew Bettison
91fd0a7801 Improve test framework: always log stderr on some assertions 2012-07-31 17:47:09 +09:30
Andrew Bettison
04b95d2590 Merge branch 'andrew' into 'master' 2012-07-25 18:04:16 +09:30
Andrew Bettison
ba69ae616f Improve test framework: fix tfw_core_backtrace
Was not actually printing a gdb backtrace
2012-07-25 17:02:57 +09:30
Andrew Bettison
6b07b4c22e Improve test framework: support gdb core dumps
Add --core-backtrace option to execute() function
Add tfw_core_backtrace() function
2012-07-24 15:38:36 +09:30
Andrew Bettison
10fd1c1fc6 Improve test framework: log message from assertStdoutIs 2012-07-23 15:18:11 +09:30
Andrew Bettison
117a3191ab Fix test framework: shorter tmp dir name
So that socket names do not exceed system limit
Also add some comments explaining a few obscure points
2012-07-18 16:29:56 +09:30
Andrew Bettison
c8a538337d Improve test framework: do not use $BASHPID
Also do not allow -j and --verbose to be given together
2012-07-18 14:44:28 +09:30
Andrew Bettison
801a0fb7b0 Improve test framework: --filter by test numbers 2012-07-16 18:40:30 +09:30
Andrew Bettison
e67d09cf11 Fix bug in test framework output 2012-07-16 18:40:30 +09:30
Andrew Bettison
cc39a02443 Improve test framework
Add 'wait_until' primitive
Improve option parsing
2012-07-11 16:51:29 +09:30
Andrew Bettison
17733c631c Improve test framework: clearer log messages 2012-07-06 13:18:43 +09:30
Andrew Bettison
7eb4293491 Improve test framework: receive TFW_LOGDIR env var
Always place test logs in servald build root, regardless of current
working directory
2012-07-04 14:54:37 +09:30
Andrew Bettison
f76ef9a7e6 Improve test framework: log timestamps 2012-07-02 12:26:58 +09:30
Andrew Bettison
7b4b37aff5 Improve test framework: provide $TFWUNIQUE 2012-07-02 10:59:34 +09:30
Andrew Bettison
5f4dfc5fd4 Improve test framework: colour results 2012-06-28 18:14:52 +09:30
Andrew Bettison
f7483c2c15 Fix test framework: broken --filter logic 2012-06-28 15:18:48 +09:30