Refactor a lot of Rhizome bundle storage code to use the new "enum
rhizome_payload_status" instead of mysterious int values to represent
the outcome of the operation.
Formalise add-bundle result in "enum rhizome_bundle_status"
Rewrite rhizome_manifest_finalise(), rhizome_find_duplicate() and
rhizome_add_manifest() to return enum rhizome_bundle_status
New function rhizome_manifest_check_stored() that compares a manifest
with its stored counterpart and returns enum rhizome_bundle_status
Remove redundant rhizome_manifest_check_sanity(), consolidating all
manifest validation rules in rhizome_manifest_validate(), which now
checks the 'id' field is present, and that 'sender' and 'recipient' are
both present for MeshMS
Correct manifest finalisation logic: set the 'finalised' flag in
rhizome_manifest_validate(), not in rhizome_manifest_verify() (which
sets 'selfSigned'), and consistently clear 'finalised' flag in all
attribute setter functions
Remove manifest 'ttl' field and all references thereof (leaving unused
space in Rhizome BAR)
Rename some payload functions for clarity
Instead of int64_t. Fixes some -Wsign-compare warnings.
Replace sqlite_exec_int64() with sqlite_exec_uint64().
Also store rowid as uint64_t, and use 0 not -1 to indicate
unset.
New UNUSED(param) macro suppresses the warning for a specific parameter
In some cases, removed the unused parameters. In others, used the
parameter in a DEBUGF() or assert() statement to document the intent.
Remove two redundant calls to rhizome_retrieve_manifest() in meshms.c,
revealed by an assert() in the stricter manifest parsing code
Fix header files included by socket.h
Allows any valid "service" manifest field in the "rhizome add file"
command
Many improvements in Rhizome manifest parsing; stricter manifest syntax
rules (no comment or blank lines, field names must be alphanumeric
identifiers), faster preliminary manifest inspection when receiving
manifest advertisements or syncing manifests
The 'development' branch introduces "struct socket_address" which
coincidentally fixed the recently encountered Linux kernel 3.12
recvmsg(2) EINVAL problem, so that 'rhizomeprotocol' tests which fail on
the 'anyservice' branch will pass after this merge
Move validation checks into new function rhizome_manifest_validate()
Remove rhizome_manifest 'errors' field
Replace rhizome_manifest 'warnings' with 'malformed'
Replace rhizome_manifest 'manifest_bytes' with 'manifest_body_bytes' and
refactor to use 'manifest_all_bytes' in all manifest i/o
Refactor rhizome_manifest_verify() and reverse sense of return value to
match rhizome_manifest_validate()
New function rhizome_manifest_inspect() -- lightweight manifest parser
used when receiving Rhizome advertisements
New 'rhizomeops' test case for invalid manifest "service" field values,
now passes
Remove unused rhizome_str[n]_is_...() functions in favour of
str[n]_to_rhizome_..._t() functions
Ensure that all str_to_..._t() functions accept a NULL 'dst' pointer
so they can be used easily to validate strings
Now tokens are included in each row instead of a single token in the
enclosing JSON object. Only puts tokens on the first row, and
thereafter all rows with a rowid higher than for the previous row that
had a token.
New ".rowid" output field from rhizome add, import, extract, export
operations. (Also added missing ".inserttime" and "date" fields to
some operations.)
Use new "rhizome add file" .rowid output field to check output of of
/restful/rhizome/bundlelist.json
Content generator functions now take arguments describing the buffer
they are to fill, and respond with a struct containing the number of
bytes filled, and the number of free bytes needed before being called
again.
The HTTP response logic now fills the buffer as much as possible before
calling write(2) by topping it up instead of waiting for it to be
completely emptied before generating more content.
Separate function rhizome_list_commit() which, if not called, causes the
next list re-query to include the last row that was fetched with
rhizome_list_next(), rather than excluding it.
Remove 'rowcount' element
Order by descending ROWID, which is functionally the same as descending
inserttime but more reliable
Replace '_offset' cursor element with rowid first-last range to record
the expanding window of rows already returned, which allows release and
re-open of cursor mid-listing without missing rows or producing duplicates
Replaced 'int has_author' manifest element with new 'enum authorship'
element to record the result of author authentication, to avoid
repeating expensive crypto operations.
Separated the handling of bundle secret arguments from author lookup and
authentication. The new rhizome_apply_bundle_secret(m,bsk) is now
called at the top level to set the manifest secret key (if it
validates), and thereafter there is no need to pass the 'bsk' argument
to any other functions, as they can simply check the 'haveSecret' field
of the manifest.
Removed rhizome_extract_privatekey() which combined author lookup and
bundle secret validation, and replaced it with functions that only deal
with the author: rhizome_lookup_author() and rhizome_authenticate_author().
Renamed other functions to make their purpose and effect clearer.
Formalised the semantics of only storing AUTHENTICATED author SIDs in
the 'author' column of the MANIFESTS table, which necessitated a change
to a 'rhizomeops' test case: when adding a file using a BK-less
manifest, the author column is set to null, so the Rhizome list output
does not show the bundle as ".fromhere" and does not give an author for
that bundle.
Replace generic rhizome_manifest_set() and rhizome_manifest_set_ll()
with per-field setter functions, eg, rhizome_manifest_set_filesize().
Struct rhizome_manifest elements for all known fields, to replace the
use of rhizome_manifest_get() and rhizome_manifest_get_ll() everywhere:
sender, recipient, service, name, date, bundle_key.
Add boolean validity flags for binary blob types, to avoid having to compare
with many bytes of all-zero to detect presence, eg, has_sender, has_recipient,
has_author, has_bundle_key. These maintained by the setter functions.
Rename existing manifest struct elements to be the same as their field
names: fileLength -> filesize, journalTail -> tail.
More use of unsigned int, size_t and uint64_t for payload sizes, offsets, byte
counts, etc. especially in rhizome_store.c and meshms.c. More uniform use of
size_t to dimension memory buffers. Fix some printf(3) style format strings
for 64-bit correctness on 32-bit systems. Use new constant RHIZOME_SIZE_UNSET
instead of -1 to indicate unknown dimension, and explicitly assert its absence
before comparisons and arithmetic, for safety.
Replace some 'int' loop variables with 'unsigned' where appropriate.
Fix bugs discovered in MeshMS bundle private/public key generation and
bundle secret key handling for export/extract commands.
Instrument the first MeshMS test case to aid debugging.
New debug config flag: debug.manifest logs all modifications to all manifest
fields by setter functions.
Rename debug config flag: debug.rhizome_bind -> debug.rhizome_sql_bind.
Use open(2)/read(2)/write(2) instead of fopen(3)/fread(3)/fwrite(3) in
several places to avoid unnecessary buffering
Fix a bug in Rhizome HTTP add's handling of unconfigured manifest
template file
Improve some debug and error logging for file i/o
Remove stowSid() at last
Change API of tohex() and strbuf_tohex(), to pass string length instead of
binary byte count. This allows odd numbers of hex digits to be produced.
Remove alloca_tohex_sid(); replace with alloca_tohex_sid_t()
New alloca_tohex_sid_t_trunc() macro
Use the new sqlite_bind() and sqlite_prepare_bind() calls instead.
Add cmp_sid_t() function, use it instead of memcmp().
Use alloca_tohex_sid_t() in preference to alloca_tohex_sid() when the
argument is a sid_t.
Revise sqlite_prepare() macro et al and underlying functions to use bind
varargs instead of sprintf(3)-style varargs.
Convert all SQL function calls in rhizome_database.c
- Define a function per page that is responsible for parsing requests
- Define a "generator" callback function for filling the buffer for a response
- Remove features that have never been used
- Remove functions that are no longer used
sqlite_void_exec() and its ilk now return the count of changed rows, not
just zero, on success
sqlite_exec_prepared() and its ilk now return the count of rows (number
of step results SQLITE_ROW), instead of just zero, on success
rhizome_clean() function now produces an optional report of its changes
rhizome_fail_write() and rhizome_finish_write() now log WARNings not
ERRORs if the SQL DELETE FROM FILES or DELETE FROM FILEBLOBS statements
fail
Refactor rhizome_open_read() to use sqlite_exec_int64()
Ensure that 'rhizome extract' and 'rhizome dump' commands return exit
status of 1 in "not found" conditions, not 255, which is reserved for
errors
Test cases for four new commands: 'rhizome delete bundle', 'rhizome
delete manifest', 'rhizome delete payload' and 'rhizome delete file'
(no test case for 'rhizome clean' yet)
- we no longer hit the database for every outgoing packet, attempting to announce bundles
- we no longer advertise manifests periodically
- when an interesting bar arrives, we ask for the manifest to be announced, which uses the existing packet format
