- support mdp clients over loopback UDP
Note this is using an environment variable to specify the port number
In future we expect to support environments where this port is already bound
- monitor mdp sockets in a single & separate thread
Introduce "enum meshms_status" to replace int -1,0,1 for return values
from MeshMS internal functions
Add test cases to check that status 2 is returned when attempting to
send a message from an identity that is not known (unlocked)
Include local (my) and remote (their) SIDs in JSON responses for
conversationlist.json and messagelist.json, so that the same JSON
structures can be used in future for non-SID-specific queries.
Refactor MeshMS message iterator, rename "sender" and "recipient" fields
to "my" and "their", for consistency with the rest of the MeshMS source
code, and because "sender" and "recipient" are properties that apply to
a single message or single ply, not to a message thread (conversation).
Change HTTP request buffer pointers from (const char*) to (char*)
because some Rhizome operations can modify received data in-place, eg,
when decrypting it.
Refactor a lot of Rhizome bundle storage code to use the new "enum
rhizome_payload_status" instead of mysterious int values to represent
the outcome of the operation.
Formalise add-bundle result in "enum rhizome_bundle_status"
Rewrite rhizome_manifest_finalise(), rhizome_find_duplicate() and
rhizome_add_manifest() to return enum rhizome_bundle_status
New function rhizome_manifest_check_stored() that compares a manifest
with its stored counterpart and returns enum rhizome_bundle_status
Remove redundant rhizome_manifest_check_sanity(), consolidating all
manifest validation rules in rhizome_manifest_validate(), which now
checks the 'id' field is present, and that 'sender' and 'recipient' are
both present for MeshMS
Correct manifest finalisation logic: set the 'finalised' flag in
rhizome_manifest_validate(), not in rhizome_manifest_verify() (which
sets 'selfSigned'), and consistently clear 'finalised' flag in all
attribute setter functions
Remove manifest 'ttl' field and all references thereof (leaving unused
space in Rhizome BAR)
Rename some payload functions for clarity
Allows any valid "service" manifest field in the "rhizome add file"
command
Many improvements in Rhizome manifest parsing; stricter manifest syntax
rules (no comment or blank lines, field names must be alphanumeric
identifiers), faster preliminary manifest inspection when receiving
manifest advertisements or syncing manifests
The 'development' branch introduces "struct socket_address" which
coincidentally fixed the recently encountered Linux kernel 3.12
recvmsg(2) EINVAL problem, so that 'rhizomeprotocol' tests which fail on
the 'anyservice' branch will pass after this merge
Move validation checks into new function rhizome_manifest_validate()
Remove rhizome_manifest 'errors' field
Replace rhizome_manifest 'warnings' with 'malformed'
Replace rhizome_manifest 'manifest_bytes' with 'manifest_body_bytes' and
refactor to use 'manifest_all_bytes' in all manifest i/o
Refactor rhizome_manifest_verify() and reverse sense of return value to
match rhizome_manifest_validate()
New function rhizome_manifest_inspect() -- lightweight manifest parser
used when receiving Rhizome advertisements
New 'rhizomeops' test case for invalid manifest "service" field values,
now passes
Replace forkKillAll with fork_terminate_all
Replace forkWaitAll with fork_wait_all
Add fork_terminate and fork_wait functions
Introduce fork labels '%<alphanumeric>' as arguments to all fork
functions
Treat a forked process exit status 143 (killed with SIGTERM) as a
failure or error -- it happens when a forked process is terminated
using fork_terminate or fork_terminate_all
Now tokens are included in each row instead of a single token in the
enclosing JSON object. Only puts tokens on the first row, and
thereafter all rows with a rowid higher than for the previous row that
had a token.
New ".rowid" output field from rhizome add, import, extract, export
operations. (Also added missing ".inserttime" and "date" fields to
some operations.)
Use new "rhizome add file" .rowid output field to check output of of
/restful/rhizome/bundlelist.json
Write test case assertions using jq(1) utility, increase from four
bundles to 100.
Fix bugs in HTTP server content generation logic.
Make payload content generator read payload 4KiB at a time, to
always read on filesystem block boundaries for performance. Increase
size of payload in relevant test case.
Replaced 'int has_author' manifest element with new 'enum authorship'
element to record the result of author authentication, to avoid
repeating expensive crypto operations.
Separated the handling of bundle secret arguments from author lookup and
authentication. The new rhizome_apply_bundle_secret(m,bsk) is now
called at the top level to set the manifest secret key (if it
validates), and thereafter there is no need to pass the 'bsk' argument
to any other functions, as they can simply check the 'haveSecret' field
of the manifest.
Removed rhizome_extract_privatekey() which combined author lookup and
bundle secret validation, and replaced it with functions that only deal
with the author: rhizome_lookup_author() and rhizome_authenticate_author().
Renamed other functions to make their purpose and effect clearer.
Formalised the semantics of only storing AUTHENTICATED author SIDs in
the 'author' column of the MANIFESTS table, which necessitated a change
to a 'rhizomeops' test case: when adding a file using a BK-less
manifest, the author column is set to null, so the Rhizome list output
does not show the bundle as ".fromhere" and does not give an author for
that bundle.
Replace generic rhizome_manifest_set() and rhizome_manifest_set_ll()
with per-field setter functions, eg, rhizome_manifest_set_filesize().
Struct rhizome_manifest elements for all known fields, to replace the
use of rhizome_manifest_get() and rhizome_manifest_get_ll() everywhere:
sender, recipient, service, name, date, bundle_key.
Add boolean validity flags for binary blob types, to avoid having to compare
with many bytes of all-zero to detect presence, eg, has_sender, has_recipient,
has_author, has_bundle_key. These maintained by the setter functions.
Rename existing manifest struct elements to be the same as their field
names: fileLength -> filesize, journalTail -> tail.
More use of unsigned int, size_t and uint64_t for payload sizes, offsets, byte
counts, etc. especially in rhizome_store.c and meshms.c. More uniform use of
size_t to dimension memory buffers. Fix some printf(3) style format strings
for 64-bit correctness on 32-bit systems. Use new constant RHIZOME_SIZE_UNSET
instead of -1 to indicate unknown dimension, and explicitly assert its absence
before comparisons and arithmetic, for safety.
Replace some 'int' loop variables with 'unsigned' where appropriate.
Fix bugs discovered in MeshMS bundle private/public key generation and
bundle secret key handling for export/extract commands.
Instrument the first MeshMS test case to aid debugging.
New debug config flag: debug.manifest logs all modifications to all manifest
fields by setter functions.
Rename debug config flag: debug.rhizome_bind -> debug.rhizome_sql_bind.
Replace setup_curl_7() function in separate test scripts with a single,
general setup_curl() function in testdefs.sh which takes the minimum
version number as its argument.
Caused by tests running so slowly that nodes become unreachable due to
timeout before the assertion of reachability is made. The timeout was
hard coded at five times the tick interval (2.5 seconds).
Introduced new config options: mdp.iftype.*.reachable_timeout_ms and
interface.*.mdp.reachable_timeout_ms to allow the timeout to be
configured. Configure the timeout to 60 seconds in 'routing' tests.
Changed the config schema to move options interfaces.*.mdp_tick_ms and
interfaces.*.packet_interval into interfaces.*.mdp.tick_ms and
interfaces.*.mdp.packet_interval (introduced new sub-struct 'mdp').
Rename 'secret:' field to '.secret:' (non-manifest-fields start with '.'
using the same convention as rhizome list)
Add '.author' and 'BK' fields to "rhizome add"
Add 'BK' field to "rhizome import"
Fix 'rhizomeops' tests to assert no 'author' and 'BK' output fields from
"rhizome add" with no author
Fiz testdefs.sh and testdefs_rhizome.sh to support new output fields
The recent work on abstract local socket names means that every instance has
its own unique abstract socket name anyway, so there is no need for test
scripts to use $TFWUNIQUE in MDP and Monitor socket names to avoid
intereference when run concurrently.
- completely event driven and non-blocking
- modelling of tx & rx serial buffers that will truncate on overfilling
- each radio takes turns to send a packet, with approx TDMA overhead
- add explicit transaction around blob writes so the commit can be retried
- remove status columns
- remove SID's from message log output to reduce size and complexity
- new interface.x.point_to_point config
- disable local echo for point to point dummy interfaces
- add ME and YOU sid abbreviation codes after learning other sid
Indepent configuration of show_pid, show_time and log level for each
destination
Update test scripts for new config options
Include xprintf.c in MDP client source files (now used by log.c)
sqlite_void_exec() and its ilk now return the count of changed rows, not
just zero, on success
sqlite_exec_prepared() and its ilk now return the count of rows (number
of step results SQLITE_ROW), instead of just zero, on success
rhizome_clean() function now produces an optional report of its changes
rhizome_fail_write() and rhizome_finish_write() now log WARNings not
ERRORs if the SQL DELETE FROM FILES or DELETE FROM FILEBLOBS statements
fail
Refactor rhizome_open_read() to use sqlite_exec_int64()
Ensure that 'rhizome extract' and 'rhizome dump' commands return exit
status of 1 in "not found" conditions, not 255, which is reserved for
errors
Test cases for four new commands: 'rhizome delete bundle', 'rhizome
delete manifest', 'rhizome delete payload' and 'rhizome delete file'
(no test case for 'rhizome clean' yet)
Output is same format as "keyring add" command
Fixed minor bug in rolling a random DID -- last byte was not random
Some code improvements, including better support for sid_t
Replace explicit [<pin,pin...>] command-line arguments with --entry-pin=<pin>
options. Update test scripts accordlingly.
Add --keyring-pin=<pin> option (with one test case).
Use new test framework fork() support to run several rhizome direct
push/pull/sync processes at once to a single server.
Speed up setup for Rhizome Transfer stress test.
Rename tests to start with "Stress" prefix
Refactor StressRhizomeAdd test to use conventional utilities and be more
readable. Remove trailing "sleep 10".
Renamed rhizome extract file to rhizome dump file
Added rhizome extract file [manifest] [filepath] [pins]
Modified tests to use the appropriate command, assuming that MeshMS payloads will be encrypted
Introduce CFINCOMPATIBLE config parse result flag.
Sort interface rules by unsigned integer key.
Legacy and modern 'interfaces' config styles are now incompatible.
Validate config_network_interface struct to enforce that only exactly one of
'match' and 'dummy' options are set.
Add test cases for 'interface' config option.
Change a test case: configuration options are now case sensitive.
Fix config file load and parse logic in conf.c, always copy 'debug' flags
from config.debug.
The config schema 'interfaces' option is no longer MANDATORY.
Introduce new CLIFLAG_PERMISSIVE_CONFIG to supress bad-config ERROR messages
from the 'config set' and 'config get' commands.
Refactor cli_execute() into cli_parse() and cli_invoke(). Use *const* struct
command_line_option everywhere.
Instead of specifying a file path, if the file is given as '-' then the content
of the manifest is output in the 'manifest' key-value pair instead of written
to a file.
This allows the output fields of any command to contain nul characters, which
paves the way for extracting a manifest or other binary data directly instead
of having to write it into a temporary file.
Replace the main-loop scheduled periodic alarm with an "activate" alarm that is
scheduled whenever a fetch candidate is added to any queue, unless the alarm is
already scheduled.
Replace the "rhizome.fetch_interval_ms" config item with
"rhizome.fetch_delay_ms" [default 50], which is the number of milliseconds
between adding a fetch candidate and firing the "activate" alarm. This allows
time for a few more Rhizome advertisment packets to arrive after the first one,
before deciding which fetches to start first.
Add new `is_scheduled()` alarm primitive.
Add ".readonly" output field, add logic to detect bundle author and update
MANIFESTS table 'author' column accordingly.
Add two 'rhizomeops' test cases to deal with authorless logic.
Rename some 'rhizomeops' test cases for consistency and to help filtering.
