Undo a change from late 2015 that allowed the format=text+binarysig
parameter of the rhizome/manifest MIME content-type to be optional -- it
is easy to provide that parameter via libcurl, and the lbard code has
just been updated to provide it.
Make Content-Type handling consistent throughout the Serval DNA code by
using 'struct mime_content_type' to represent both parsed (request) and
sent (response) Content-Type fields, replacing ad-hoc in-line string
literals.
Fix some TODOs by adding the "serval/sid", "rhizome/bid" and
"rhizome/bundlesecret" MIME types, which remain optional for the time
being to preserve REST API compatibility with clients that do not set
the Content-Type of their 'bundle-author', 'bundle-id' and
'bundle-secret' parameters.
This introduces a new way of linking Serval executables and dynamic
libraries from static libraries like libservald.a -- called
"feature-driven" linking.
The Makefile now links servald and serval-tests from libservald.a,
rather than from an explicit list of object (.o) files. Thanks to the
section-based method for registering functions such as HTTP handlers,
CLI commands and MDP handlers, these object files had become
"stand-alone" and hence were no longer included in the link because
there was no unresolved reference that required them to be linked in.
The new "feature.h" provides the DECLARE_FEATURE(name) macro that each
stand-alone source file uses to declare the named feature(s) it
provides. Each executable can call the USE_FEATURE(name) macro in any
of its explicitly-linked source files to cause the corresponding
object(s) to be included in the link, eg, servald_features.c.
The DEFINE_BINDING() macro has been extended so that every individual
MDP binding is given a feature name based on its port number macro, eg,
"mdp_binding_MDP_PORT_ECHO".
Some features have been factored into their own separate source files so
they can be omitted or included in a build independently of each other:
- the MDP bindings for MDP_PORT_DNALOOKUP, MDP_PORT_ECHO,
MDP_PORT_TRACE, MDP_PORT_KEYMAPREQUEST, MDP_PORT_RHIZOME_xxx,
MDP_PORT_PROBE, MDP_PORT_STUN, MDP_PORT_STUNREQ
- the CLI "log" and "echo" commands
- the CLI "rhizome direct" command
The JNI source files are only compiled if the <jni.h> header is present,
otherwise they are omitted from libservald.so.
Add RHIZOME_BUNDLE_STATUS_MANIFEST_TOO_BIG enum option to indicate
that the manifest exceeded 8 KiB in size.
Refactor rhizome_add_manifest() and rhizome_manifest_finalise()
to return 'struct rhizome_bundle_result' instead of 'enum
rhizome_bundle_status', so that that their detailed failure messages
can reach the HTTP API layer instead of just being logged.
Fix HTTP response status codes produced Rhizome direct HTTP requests
to be consistent with the Rhizome RESTful API.
- Introduce the new 'struct rhizome_bundle_result' that contains a
rhizome_bundle_status enum value and an optional, nul-terminated
string that provides an explanation of the cause of the status; add
functions to construct, query, and free the struct
- Replace 'enum rhizome_add_result' with 'struct rhizome_bundle_result',
removing an unnecessary level of enum interpretation
- Make rhizome_fill_manifest() return 'struct rhizome_bundle_result' and
add logic to check that the supplied author SID is correct (previous
behaviour was: if an incorrect author SID was supplied but the correct
author could be found in the keyring, then the incorrect one was
silently ignored)
- Simplify the response code in rhizome_restful.cc to take advantage of
the new 'struct rhizome_bundle_result'; in particular, the mapping
from 'enum rhizome_bundle_status' codes to HTTP status codes is now
expressed in a single switch statement
- Fix some minor failures in test scripts revealed by the changes
Original DEBUG() and DEBUGF() macros renamed to _DEBUG() and _DEBUGF()
New DEBUG() and DEBUGF() macros, first argument is flagname
New DEBUGF2(foo, bar, ...) macro does if(config.debug.foo||config.debug.bar) test
Replace almost all config.debug.xxx references to IF_DEBUG(xxx)
Add RHIZOME_BUNDLE_STATUS_READONLY enum value
Tighten up switch statements on bundle and payload status enums (no
default labels)
Rename some recently added enum entries
Return bundle status and payload status in HTTP responses
Add test for failing to decrypt a foreign encrypted bundle payload,
fix bug that caused an assertion failure
Add tests for fetching a non-existent manifest and fetching bundles
whose payload blob is not in the store
If no instance directory specified, then use paths like
/etc/serval/serval.conf
/var/cache/serval
/var/log/serval
/var/run/serval
etc. for files, instead of all in a single directory.
Log all directory creation as INFO messages.
Interpretation of log.file.directory_path has changed slightly.
Updated servald configuration tech doc.
Change HTTP request buffer pointers from (const char*) to (char*)
because some Rhizome operations can modify received data in-place, eg,
when decrypting it.
Refactor a lot of Rhizome bundle storage code to use the new "enum
rhizome_payload_status" instead of mysterious int values to represent
the outcome of the operation.
Formalise add-bundle result in "enum rhizome_bundle_status"
Rewrite rhizome_manifest_finalise(), rhizome_find_duplicate() and
rhizome_add_manifest() to return enum rhizome_bundle_status
New function rhizome_manifest_check_stored() that compares a manifest
with its stored counterpart and returns enum rhizome_bundle_status
Remove redundant rhizome_manifest_check_sanity(), consolidating all
manifest validation rules in rhizome_manifest_validate(), which now
checks the 'id' field is present, and that 'sender' and 'recipient' are
both present for MeshMS
Correct manifest finalisation logic: set the 'finalised' flag in
rhizome_manifest_validate(), not in rhizome_manifest_verify() (which
sets 'selfSigned'), and consistently clear 'finalised' flag in all
attribute setter functions
Remove manifest 'ttl' field and all references thereof (leaving unused
space in Rhizome BAR)
Rename some payload functions for clarity
Instead of int64_t. Fixes some -Wsign-compare warnings.
Replace sqlite_exec_int64() with sqlite_exec_uint64().
Also store rowid as uint64_t, and use 0 not -1 to indicate
unset.
New UNUSED(param) macro suppresses the warning for a specific parameter
In some cases, removed the unused parameters. In others, used the
parameter in a DEBUGF() or assert() statement to document the intent.
Move validation checks into new function rhizome_manifest_validate()
Remove rhizome_manifest 'errors' field
Replace rhizome_manifest 'warnings' with 'malformed'
Replace rhizome_manifest 'manifest_bytes' with 'manifest_body_bytes' and
refactor to use 'manifest_all_bytes' in all manifest i/o
Refactor rhizome_manifest_verify() and reverse sense of return value to
match rhizome_manifest_validate()
New function rhizome_manifest_inspect() -- lightweight manifest parser
used when receiving Rhizome advertisements
New 'rhizomeops' test case for invalid manifest "service" field values,
now passes
Replaced 'int has_author' manifest element with new 'enum authorship'
element to record the result of author authentication, to avoid
repeating expensive crypto operations.
Separated the handling of bundle secret arguments from author lookup and
authentication. The new rhizome_apply_bundle_secret(m,bsk) is now
called at the top level to set the manifest secret key (if it
validates), and thereafter there is no need to pass the 'bsk' argument
to any other functions, as they can simply check the 'haveSecret' field
of the manifest.
Removed rhizome_extract_privatekey() which combined author lookup and
bundle secret validation, and replaced it with functions that only deal
with the author: rhizome_lookup_author() and rhizome_authenticate_author().
Renamed other functions to make their purpose and effect clearer.
Formalised the semantics of only storing AUTHENTICATED author SIDs in
the 'author' column of the MANIFESTS table, which necessitated a change
to a 'rhizomeops' test case: when adding a file using a BK-less
manifest, the author column is set to null, so the Rhizome list output
does not show the bundle as ".fromhere" and does not give an author for
that bundle.
Replace generic rhizome_manifest_set() and rhizome_manifest_set_ll()
with per-field setter functions, eg, rhizome_manifest_set_filesize().
Struct rhizome_manifest elements for all known fields, to replace the
use of rhizome_manifest_get() and rhizome_manifest_get_ll() everywhere:
sender, recipient, service, name, date, bundle_key.
Add boolean validity flags for binary blob types, to avoid having to compare
with many bytes of all-zero to detect presence, eg, has_sender, has_recipient,
has_author, has_bundle_key. These maintained by the setter functions.
Rename existing manifest struct elements to be the same as their field
names: fileLength -> filesize, journalTail -> tail.
More use of unsigned int, size_t and uint64_t for payload sizes, offsets, byte
counts, etc. especially in rhizome_store.c and meshms.c. More uniform use of
size_t to dimension memory buffers. Fix some printf(3) style format strings
for 64-bit correctness on 32-bit systems. Use new constant RHIZOME_SIZE_UNSET
instead of -1 to indicate unknown dimension, and explicitly assert its absence
before comparisons and arithmetic, for safety.
Replace some 'int' loop variables with 'unsigned' where appropriate.
Fix bugs discovered in MeshMS bundle private/public key generation and
bundle secret key handling for export/extract commands.
Instrument the first MeshMS test case to aid debugging.
New debug config flag: debug.manifest logs all modifications to all manifest
fields by setter functions.
Rename debug config flag: debug.rhizome_bind -> debug.rhizome_sql_bind.
