Move validation checks into new function rhizome_manifest_validate()
Remove rhizome_manifest 'errors' field
Replace rhizome_manifest 'warnings' with 'malformed'
Replace rhizome_manifest 'manifest_bytes' with 'manifest_body_bytes' and
refactor to use 'manifest_all_bytes' in all manifest i/o
Refactor rhizome_manifest_verify() and reverse sense of return value to
match rhizome_manifest_validate()
New function rhizome_manifest_inspect() -- lightweight manifest parser
used when receiving Rhizome advertisements
New 'rhizomeops' test case for invalid manifest "service" field values,
now passes
New ".rowid" output field from rhizome add, import, extract, export
operations. (Also added missing ".inserttime" and "date" fields to
some operations.)
Use new "rhizome add file" .rowid output field to check output of of
/restful/rhizome/bundlelist.json
Replaced 'int has_author' manifest element with new 'enum authorship'
element to record the result of author authentication, to avoid
repeating expensive crypto operations.
Separated the handling of bundle secret arguments from author lookup and
authentication. The new rhizome_apply_bundle_secret(m,bsk) is now
called at the top level to set the manifest secret key (if it
validates), and thereafter there is no need to pass the 'bsk' argument
to any other functions, as they can simply check the 'haveSecret' field
of the manifest.
Removed rhizome_extract_privatekey() which combined author lookup and
bundle secret validation, and replaced it with functions that only deal
with the author: rhizome_lookup_author() and rhizome_authenticate_author().
Renamed other functions to make their purpose and effect clearer.
Formalised the semantics of only storing AUTHENTICATED author SIDs in
the 'author' column of the MANIFESTS table, which necessitated a change
to a 'rhizomeops' test case: when adding a file using a BK-less
manifest, the author column is set to null, so the Rhizome list output
does not show the bundle as ".fromhere" and does not give an author for
that bundle.
Replace generic rhizome_manifest_set() and rhizome_manifest_set_ll()
with per-field setter functions, eg, rhizome_manifest_set_filesize().
Struct rhizome_manifest elements for all known fields, to replace the
use of rhizome_manifest_get() and rhizome_manifest_get_ll() everywhere:
sender, recipient, service, name, date, bundle_key.
Add boolean validity flags for binary blob types, to avoid having to compare
with many bytes of all-zero to detect presence, eg, has_sender, has_recipient,
has_author, has_bundle_key. These maintained by the setter functions.
Rename existing manifest struct elements to be the same as their field
names: fileLength -> filesize, journalTail -> tail.
More use of unsigned int, size_t and uint64_t for payload sizes, offsets, byte
counts, etc. especially in rhizome_store.c and meshms.c. More uniform use of
size_t to dimension memory buffers. Fix some printf(3) style format strings
for 64-bit correctness on 32-bit systems. Use new constant RHIZOME_SIZE_UNSET
instead of -1 to indicate unknown dimension, and explicitly assert its absence
before comparisons and arithmetic, for safety.
Replace some 'int' loop variables with 'unsigned' where appropriate.
Fix bugs discovered in MeshMS bundle private/public key generation and
bundle secret key handling for export/extract commands.
Instrument the first MeshMS test case to aid debugging.
New debug config flag: debug.manifest logs all modifications to all manifest
fields by setter functions.
Rename debug config flag: debug.rhizome_bind -> debug.rhizome_sql_bind.
Use open(2)/read(2)/write(2) instead of fopen(3)/fread(3)/fwrite(3) in
several places to avoid unnecessary buffering
Fix a bug in Rhizome HTTP add's handling of unconfigured manifest
template file
Improve some debug and error logging for file i/o
Remove stowSid() at last
Change API of tohex() and strbuf_tohex(), to pass string length instead of
binary byte count. This allows odd numbers of hex digits to be produced.
Remove alloca_tohex_sid(); replace with alloca_tohex_sid_t()
New alloca_tohex_sid_t_trunc() macro
OUT()s or where return() is used instead of RETURN().
Added OUT() to end of all functions using IN() that lacked it to
make it easier to statically analyse this invariant.
Fixed several return instead of RETURNs detected through use of
this tool. #49
Overhauled the file fetch queue logic in rhizome_fetch.c.
Now the 'rhizomeprotocol' stress test passes in approximately 5 minutes on my
2009-vintage Dell laptop.
Added a call to rhizome_enqueue_suggestions() in rhizome_fetch_close() so that
a new Rhizome GET request is sent as soon as a fetch slot becomes free, instead
of waiting for the (default 5 second) timer to trigger the next GET.
Introduce __WHENCE__ macro and a block comment in log.h explaining it.
In "primitive" kinds of functions, rename 'whence' arguments to '__whence' and
use WHYF(), WARNF(), DEBUGF() macros instead of calling logMessage() directly.
Add test case for new feature of the "rhizome add" command: if the author SID
is not specified (empty arg) then it searches the keyring for the author.
Removed "authorSid" argument from several functions that also take a struct
rhizome_manifest * arg, since the author, if known, is now supplied in the
struct.
Improve return value handling and refactored some rhizome crypto code.
Replaces (const char *file, unsigned int line, const char *function) arguments
to all logging functions, simplifies malloc/free tracking code in
overlay_buffer.c and Rhizome manifest alloc/free tracking in rhizome_bundle.c.
Use __HERE__ macro instead of (__FILE__, __LINE__, __FUNCTION__) everywhere.
Special __NOWHERE__ macro is equivalent to (NULL, 0, NULL).
Declare net.c functions in new "net.h" header, so log.c doesn't have to pull
in the entire "serval.h" just to use write_str().
Facilitates progress on issue #2.
Servald starts DNA helper, receives startup ACK, sends requests, receives
responses, handles malformed helper responses, echoes dnahelper stderr lines to
log, sends MDP reply packet, waits for dead helper process, all asynchronously.
Shuts down helper process during servald shutdown.
Remaining issues:
- Does not impose a timeout on helper responses.
- Only the first URI is reported by the "dna lookup" command.
Do not add 'filehash' var to manifest if filesize=0
Do not accept 'filehash' var when parsing manifest with filesize=0
When responding to a new rhizome advertisement, do not try to HTTP
request a payload if filesize=0, just import the manifest directly
Various operations, eg "rhizome file add", do not report 'filehash'
fields where 'filesize' is zero
Do not delete rows from MANIFESTS table which have empty filehash
Various related bug fixes
Rhizome manifest parser now parses and validates all known fields, informs
about unsupported fields, and unpacks fields into relevant struct manifest
elements where appropriate. Is also stricter about whitespace.
Rhizome fetch code now logs debug messages if DEBUG_RHIZOME_RX bit is on.
that is called only when needed, and marks a manifest as finalised
if the verifcation fails. reading a manifest now never sets
finalised flag, as either _finalise() or _verify() must be called.
Move rhizome_new_manifest() out of rhizome_read_manifest_file() so that the
out-of-manifest report shows the names of the functions where the manifests
were really allocated.
Replace with WHY() or WHYF() everywhere
Improve logMessage() to handle file/line/function printing internally
Mainly so that log messages from manifest alloc/free in rhizome_bundle.c
can make it into the Android log
The sequence start time was only getting set in overlay_interface_init(), which
was not always called, or was called after overlay_gettime_ms() had already
been used.
Added FATALF(), FATAL() and FATAL_perror() macros.
Removed a bunch of debug statements no longer needed in monitor.c and server.c.
- If a bundle has the same payload hash and manifest.name as one already in
the rhizome store, then the "dna rhizome add" command will not add it
- Now passes fourth test case in tests/dna_rhizome
Now successfully fetches manifests for files we already have in
the database, using the existing database copy of the file.
Also added filter to stop same file being fetched more than once at
same time..
