New ".rowid" output field from rhizome add, import, extract, export
operations. (Also added missing ".inserttime" and "date" fields to
some operations.)
Use new "rhizome add file" .rowid output field to check output of of
/restful/rhizome/bundlelist.json
Write test case assertions using jq(1) utility, increase from four
bundles to 100.
Fix bugs in HTTP server content generation logic.
Make payload content generator read payload 4KiB at a time, to
always read on filesystem block boundaries for performance. Increase
size of payload in relevant test case.
Replaced 'int has_author' manifest element with new 'enum authorship'
element to record the result of author authentication, to avoid
repeating expensive crypto operations.
Separated the handling of bundle secret arguments from author lookup and
authentication. The new rhizome_apply_bundle_secret(m,bsk) is now
called at the top level to set the manifest secret key (if it
validates), and thereafter there is no need to pass the 'bsk' argument
to any other functions, as they can simply check the 'haveSecret' field
of the manifest.
Removed rhizome_extract_privatekey() which combined author lookup and
bundle secret validation, and replaced it with functions that only deal
with the author: rhizome_lookup_author() and rhizome_authenticate_author().
Renamed other functions to make their purpose and effect clearer.
Formalised the semantics of only storing AUTHENTICATED author SIDs in
the 'author' column of the MANIFESTS table, which necessitated a change
to a 'rhizomeops' test case: when adding a file using a BK-less
manifest, the author column is set to null, so the Rhizome list output
does not show the bundle as ".fromhere" and does not give an author for
that bundle.
Replace generic rhizome_manifest_set() and rhizome_manifest_set_ll()
with per-field setter functions, eg, rhizome_manifest_set_filesize().
Struct rhizome_manifest elements for all known fields, to replace the
use of rhizome_manifest_get() and rhizome_manifest_get_ll() everywhere:
sender, recipient, service, name, date, bundle_key.
Add boolean validity flags for binary blob types, to avoid having to compare
with many bytes of all-zero to detect presence, eg, has_sender, has_recipient,
has_author, has_bundle_key. These maintained by the setter functions.
Rename existing manifest struct elements to be the same as their field
names: fileLength -> filesize, journalTail -> tail.
More use of unsigned int, size_t and uint64_t for payload sizes, offsets, byte
counts, etc. especially in rhizome_store.c and meshms.c. More uniform use of
size_t to dimension memory buffers. Fix some printf(3) style format strings
for 64-bit correctness on 32-bit systems. Use new constant RHIZOME_SIZE_UNSET
instead of -1 to indicate unknown dimension, and explicitly assert its absence
before comparisons and arithmetic, for safety.
Replace some 'int' loop variables with 'unsigned' where appropriate.
Fix bugs discovered in MeshMS bundle private/public key generation and
bundle secret key handling for export/extract commands.
Instrument the first MeshMS test case to aid debugging.
New debug config flag: debug.manifest logs all modifications to all manifest
fields by setter functions.
Rename debug config flag: debug.rhizome_bind -> debug.rhizome_sql_bind.
Replace setup_curl_7() function in separate test scripts with a single,
general setup_curl() function in testdefs.sh which takes the minimum
version number as its argument.
Caused by tests running so slowly that nodes become unreachable due to
timeout before the assertion of reachability is made. The timeout was
hard coded at five times the tick interval (2.5 seconds).
Introduced new config options: mdp.iftype.*.reachable_timeout_ms and
interface.*.mdp.reachable_timeout_ms to allow the timeout to be
configured. Configure the timeout to 60 seconds in 'routing' tests.
Changed the config schema to move options interfaces.*.mdp_tick_ms and
interfaces.*.packet_interval into interfaces.*.mdp.tick_ms and
interfaces.*.mdp.packet_interval (introduced new sub-struct 'mdp').
Rename 'secret:' field to '.secret:' (non-manifest-fields start with '.'
using the same convention as rhizome list)
Add '.author' and 'BK' fields to "rhizome add"
Add 'BK' field to "rhizome import"
Fix 'rhizomeops' tests to assert no 'author' and 'BK' output fields from
"rhizome add" with no author
Fiz testdefs.sh and testdefs_rhizome.sh to support new output fields
The recent work on abstract local socket names means that every instance has
its own unique abstract socket name anyway, so there is no need for test
scripts to use $TFWUNIQUE in MDP and Monitor socket names to avoid
intereference when run concurrently.
- completely event driven and non-blocking
- modelling of tx & rx serial buffers that will truncate on overfilling
- each radio takes turns to send a packet, with approx TDMA overhead
- add explicit transaction around blob writes so the commit can be retried
- remove status columns
- remove SID's from message log output to reduce size and complexity
- new interface.x.point_to_point config
- disable local echo for point to point dummy interfaces
- add ME and YOU sid abbreviation codes after learning other sid
Indepent configuration of show_pid, show_time and log level for each
destination
Update test scripts for new config options
Include xprintf.c in MDP client source files (now used by log.c)
sqlite_void_exec() and its ilk now return the count of changed rows, not
just zero, on success
sqlite_exec_prepared() and its ilk now return the count of rows (number
of step results SQLITE_ROW), instead of just zero, on success
rhizome_clean() function now produces an optional report of its changes
rhizome_fail_write() and rhizome_finish_write() now log WARNings not
ERRORs if the SQL DELETE FROM FILES or DELETE FROM FILEBLOBS statements
fail
Refactor rhizome_open_read() to use sqlite_exec_int64()
Ensure that 'rhizome extract' and 'rhizome dump' commands return exit
status of 1 in "not found" conditions, not 255, which is reserved for
errors
Test cases for four new commands: 'rhizome delete bundle', 'rhizome
delete manifest', 'rhizome delete payload' and 'rhizome delete file'
(no test case for 'rhizome clean' yet)
Output is same format as "keyring add" command
Fixed minor bug in rolling a random DID -- last byte was not random
Some code improvements, including better support for sid_t
Replace explicit [<pin,pin...>] command-line arguments with --entry-pin=<pin>
options. Update test scripts accordlingly.
Add --keyring-pin=<pin> option (with one test case).
Use new test framework fork() support to run several rhizome direct
push/pull/sync processes at once to a single server.
Speed up setup for Rhizome Transfer stress test.
Rename tests to start with "Stress" prefix
Refactor StressRhizomeAdd test to use conventional utilities and be more
readable. Remove trailing "sleep 10".
Renamed rhizome extract file to rhizome dump file
Added rhizome extract file [manifest] [filepath] [pins]
Modified tests to use the appropriate command, assuming that MeshMS payloads will be encrypted
Introduce CFINCOMPATIBLE config parse result flag.
Sort interface rules by unsigned integer key.
Legacy and modern 'interfaces' config styles are now incompatible.
Validate config_network_interface struct to enforce that only exactly one of
'match' and 'dummy' options are set.
Add test cases for 'interface' config option.
Change a test case: configuration options are now case sensitive.
Fix config file load and parse logic in conf.c, always copy 'debug' flags
from config.debug.
The config schema 'interfaces' option is no longer MANDATORY.
Introduce new CLIFLAG_PERMISSIVE_CONFIG to supress bad-config ERROR messages
from the 'config set' and 'config get' commands.
Refactor cli_execute() into cli_parse() and cli_invoke(). Use *const* struct
command_line_option everywhere.
Instead of specifying a file path, if the file is given as '-' then the content
of the manifest is output in the 'manifest' key-value pair instead of written
to a file.
This allows the output fields of any command to contain nul characters, which
paves the way for extracting a manifest or other binary data directly instead
of having to write it into a temporary file.
Replace the main-loop scheduled periodic alarm with an "activate" alarm that is
scheduled whenever a fetch candidate is added to any queue, unless the alarm is
already scheduled.
Replace the "rhizome.fetch_interval_ms" config item with
"rhizome.fetch_delay_ms" [default 50], which is the number of milliseconds
between adding a fetch candidate and firing the "activate" alarm. This allows
time for a few more Rhizome advertisment packets to arrive after the first one,
before deciding which fetches to start first.
Add new `is_scheduled()` alarm primitive.
Add ".readonly" output field, add logic to detect bundle author and update
MANIFESTS table 'author' column accordingly.
Add two 'rhizomeops' test cases to deal with authorless logic.
Rename some 'rhizomeops' test cases for consistency and to help filtering.
Add test case for new feature of the "rhizome add" command: if the author SID
is not specified (empty arg) then it searches the keyring for the author.
Removed "authorSid" argument from several functions that also take a struct
rhizome_manifest * arg, since the author, if known, is now supplied in the
struct.
Improve return value handling and refactored some rhizome crypto code.
Replace ".selfsigned" column with ".author" and ".fromhere" columns in
output of "rhizome list" command. (Note that a "sender" column is
already present.)
Add 'author' field to struct rhizome_manifest.
Log all fully rendered SQL statements on DEBUG_RHIZOME.
Update 'rhizomeops' test cases and improve the assert_rhizome_list()
test function to be able to assert authorship of files.
The extract_stdout_keyvalue() function now accepts an optional delimiter
argument, to override its default of ':'. This makes it useful for extracting
the output of "servald config get" commands.
Fixes#21. The problem was caused when the double-fork logic used in "servald
start" was clobbered in 5103176. This meant that the servald daemon process on
Android no longer had a PPID=1, but the PID of the long-lived
"org.servalproject" parent process which called the JNI entry point. Killing
the servald process then caused it to become a zombie process, since the
org.servalproject does not habitually call wait(2). That caused the "servald
stop" logic to send five SIGHUPs to the zombie without any error, making it
appear that the process was not dying.
Reinstated the double-fork logic and added a new test case to ensure that the
daemon process does not become a zombie on being killed prematurely.
The "rhizome direct push" command (and also sync) was not waiting for the
server's HTTP response, so it was exiting before the server had finished
storing the bundle, which led to a race with the subsequent "assert
bundle_received_by" test. Fixed by adding the missing code to receive the HTTP
response.
Refactored the code used for parsing HTTP responses in rhizome_fetch.c, and
used it in rhizome_direct_http.c.
Rename Push, Pull, Sync to DirectPush, DirectPull, DirectSync.
Move HttpAdd test to just after HttpImport test.
Tighten up the assertions in Rhizome Direct test cases.
Factor the "assert_rhizome_list" test out of assert_received() to avoid having
to water down assert_rhizome_list(), which currently takes an exact list of all
files that must be present in the list, and is more valuable that way.
and made corresponding adjustments. Oddly push and pull work again.
Maybe there is some subtle bug in the test framework?
Anyway, now to fix the sync test. #9
as hex if not supplying a manifest template.
modified rhizomeprotocol test of manifest-less HTTP import to set
that option. That test now passes.
ALL rhizomeprotocol tests now pass. #9
file import. It now rungs in rhizomeprotocol test suite, but fails
because the BK= field is missing (but the file is imported).
Also removed the assertions about receiving bundles from the push
test as it was non-deterministic. The bundle reception is already
tested, so there is no loss.
Investigating missing BK=.
Also test 8 (sync) fails post-merge, although push and pull work
intepdendently. Will investigate that post-merge. #9
Must be enabled by using rhizome.api.addfile.*
Certainly polishing to be done, including using filename supplied
during HTTP POST. Now to fix that, and make it all work with
final rhizomeprotocol test case.
rhizomeprotocol test cases 8 and 9 currently fail post-merge. #9
associated file before checking if it was already in the database.
Rhizome Direct can supply a manifest without associated file if
the file is already in the database, and so it was breaking.
Also removed "assert bundle_received_by"'s from rhizome direct
pull and sync tests because they are not needed, and were failing
because the same file contents was used for the files being
exchanged, and so file storage was not occurring, and thus the log
message being looked for was not being produced.
Push, pull and sync tests in rhizomeprotocol now pass, leaving
only two tests in error. #9
signatures were not being recorded. Also fixed separate issue
where rhizomeprotocols tests expected selfsigned to be 1 on
receiver end, when it will never be, because the BK doesn't match.
rhizome direct push test in rhizomeprotocols now passes. #9
Now the 'rhizomeprotocol' Push test case now passes. It should be renamed to
DirectPush.
Much refactoring of the Rhizome Direct HTTP request parsing. Now uses
strbuf_sprintf() instead of snprintf() in many places to check for buffer
overrun and ensure terminating nul. Still more of this kind of work is needed.
Improved debug that needs to be made conditional on DEBUG_RHIZOME_RX and
DEBUG_RHIZOME_TX. Some just needs removal.
Explicitly create identities in 'dnaprotocol' fixtures because
start_servald_instances() no longer does it.
The create_single_identity() function was not setting NAMEA, NAMEB, etc. when
they contained a space, because of shell misquoting.
(cherry picked from commit 9d8707a0ad)
Cherry picked from commit 87133cb2de, conflicts
in tests/rhizomeprotocol resolved by hand: bundle_received_by() now takes two
args before list of instances.
The 'rhizomeprotocol' tests now use create_single_identity() in fixtures (which
tests the 'keyring list' command).
Use foreach_instance() function in various places, and replace functions that
used to loop over instances with functions that work on the current instance.
Cherry picked from commit e994626429, conflicts
in testdefs.sh resolved manually (start_servald_instances() sets DUMMYx
variable).
Move create_rhizome_identities() from testdefs_rhizome.sh to
create_identities() in testdefs.sh
Rename create_identity() in testdefs.sh to create_single_identity(), uses
create_identities() to do its work.
All test cases pass on Solaris (see issue #16) except the 'jni' tests because
there is no Java compiler available on Solaris, and the new 'rhizomeprotocol'
tests from the rhizomedirect branch, which have always failed.
Replace explicit inline binary data from HttpImport test, so that Git stops
thinking the shell script is a binary file.
Put curl(1) detection into common fixture function.
Explicitly create identities in 'dnaprotocol' fixtures because
start_servald_instances() no longer does it.
The create_single_identity() function was not setting NAMEA, NAMEB, etc. when
they contained a space, because of shell misquoting.
The 'rhizomeprotocol' tests now use create_single_identity() in fixtures (which
tests the 'keyring list' command).
Use foreach_instance() function in various places, and replace functions that
used to loop over instances with functions that work on the current instance.
Move create_rhizome_identities() from testdefs_rhizome.sh to
create_identities() in testdefs.sh
Rename create_identity() in testdefs.sh to create_single_identity(), uses
create_identities() to do its work.
Servald starts DNA helper, receives startup ACK, sends requests, receives
responses, handles malformed helper responses, echoes dnahelper stderr lines to
log, sends MDP reply packet, waits for dead helper process, all asynchronously.
Shuts down helper process during servald shutdown.
Remaining issues:
- Does not impose a timeout on helper responses.
- Only the first URI is reported by the "dna lookup" command.
Rhizome manifest parser now parses and validates all known fields, informs
about unsupported fields, and unpacks fields into relevant struct manifest
elements where appropriate. Is also stricter about whitespace.
Rhizome fetch code now logs debug messages if DEBUG_RHIZOME_RX bit is on.
Was not transmitting actual HTTP server port in rhizome announcements, was
always transmitting port 4110.
When trying for a free HTTP server port, sometimes bind() succeeds but listen()
fails with EADDRINUSE, so new logic to deal with that.
Fix endless recursion if error or debug logged while reading config file
Fix 'config del' logic
Log messages made before log file can be opened are buffered and written
once the file is open
Do not log to file in ANDROID version, just to Android's log system
... as part of working on work on DNA helper callout facility.
(DNA replies now contain a token that can be used to match them
against DNA requests as a happy side-effect).
results from each SID/servald instance.
Fixed bug in "keyring list" that was returning keyrings in
lower case instead of normalised form using overlay_render_sid()
Tests assert that stderr contains no ERROR: lines after a successful exit
Rewrote sqlite_exec_int64() to separate error outcomes from legitimate
result values
Changed several WHY() calls to DEBUG()
Improved test framework
Improve regular expressions for common data types in test scripts
Revert column count field delimiter in "rhizome list" from ":" to "\n"
Add a few more test cases
The "rhizome file add" command assumes service=file if no manifest supplied
or the manifest lacks a service field.
The "rhizome extract manifest" command includes the service in its CLI output.
Config debug flags are now represented as individual items, eg:
servald config set debug.verbose yes
servald config set debug.rhizome yes
instead of aggregated, eg:
servald config set debug verbose,rhizome <-- no longer works
Added a debug flag "debug.all" that can be set and cleared, over which
individual flag sets/clears take priority.
Slimmed down dna.c by moving some globals and functions to the files where they
are used.
Default command line (help message) is now the new style, not the deprecated
style.
Does not depend on ServalDResult class - appends output fields to a supplied
list instead, and returns the integer status
Does not depend on ServalDReentranceError class - uses
java.lang.IllegalStateException instead
- new test cases to cover new payload with different versions
- exit status of 2 when duplicate is detected
- tighten up other test cases
- fix bug in sqlite_exec_int64()
- adding a bundle using an existing manifest with an incorrect payload should
fail with an error
- adding a bundle using an existing manifest to update the payload to a new
version should succeed
- improve format of "dna rhizome list" output to one bundle per line
- new execute() option: --exit-status=N
equivalent to assertExitStatus --stderr '==' N
- new executeOk() function, shortcut for execute --exit-status=0
