Closes#2.
Rewrite all Rhizome db query code using new retry primitives defined in
"rhizome.h": sqlite_step_retry(), sqlite_retry(), sqlite_retry_done(), etc.
Replace all calls to sqlite3_prepare_v2() with sqlite_prepare() which does
proper error logging.
Fix bug: re-invoking sqlite3_blob_close() on SQLITE_BUSY return causes process
to abort. Use an explicit BEGIN...COMMIT around the blob writing code instead.
Tested using repeated invocations of batphone/tests/meshms1.
Delete deprecated Rhizome db code in rhizome_crypto.c that has been replaced
with keyring file.
Much refactoring and removal of cruft.
SQL query errors are now logged with the filename, line number and function
where they were invoked, not of the low-level function that discovered the
error. This makes use of the new __HERE__ notation introduced last commit.
All the queries that used sqlite_exec_void() and sqlite_exec_int64() and
sqlite_exec_strbuf() now do a sleep-retry while the Rhizome db is locked.
There are other queries that still need conversion, and some old infinite
retry logic that needs replacing.
Add sqlite_exec_void_retry() function, use it in
rhizome_update_file_priority(). This should be reviewed to ensure that the
server process never sleeps.
The general problem remains of what the servald process should do if the
database is locked when it tries to update. Simplest solution is to sleep and
retry, but that blocks all other services and would hurt VoMP. A better
solution would be for each Rhizome operation to collect its database updates
into a single transaction and place that in a work queue that gets called using
schedule() (or even watch() if a file-descriptor event can somehow be used when
the database becomes available). Another solution is perhaps to perform all
Rhizome operations in a dedicated process that can block indefinitely on the
database without affecting servald responsiveness.
Do not add 'filehash' var to manifest if filesize=0
Do not accept 'filehash' var when parsing manifest with filesize=0
When responding to a new rhizome advertisement, do not try to HTTP
request a payload if filesize=0, just import the manifest directly
Various operations, eg "rhizome file add", do not report 'filehash'
fields where 'filesize' is zero
Do not delete rows from MANIFESTS table which have empty filehash
Various related bug fixes
Tests assert that stderr contains no ERROR: lines after a successful exit
Rewrote sqlite_exec_int64() to separate error outcomes from legitimate
result values
Changed several WHY() calls to DEBUG()
Improved test framework
Improve regular expressions for common data types in test scripts
Revert column count field delimiter in "rhizome list" from ":" to "\n"
Add a few more test cases
that is called only when needed, and marks a manifest as finalised
if the verifcation fails. reading a manifest now never sets
finalised flag, as either _finalise() or _verify() must be called.
Move rhizome_new_manifest() out of rhizome_read_manifest_file() so that the
out-of-manifest report shows the names of the functions where the manifests
were really allocated.
Was calling exit() instead of returning -1 on error, which was causing
the serval app to exit in the JNI "start" command if rhizome could not
create its db file.
Use DEBUGF() instead of fprintf(stderr) in many places.
confValueGet() was not stripping \r\n off end of returned string.
rhizome_set_datastore_path() was keeping the pointer it was given (from
conValueGet()'s rotor) instead of making a strdup() copy.
The "rhizome file add" command assumes service=file if no manifest supplied
or the manifest lacks a service field.
The "rhizome extract manifest" command includes the service in its CLI output.
Config debug flags are now represented as individual items, eg:
servald config set debug.verbose yes
servald config set debug.rhizome yes
instead of aggregated, eg:
servald config set debug verbose,rhizome <-- no longer works
Added a debug flag "debug.all" that can be set and cleared, over which
individual flag sets/clears take priority.
Slimmed down dna.c by moving some globals and functions to the files where they
are used.
Default command line (help message) is now the new style, not the deprecated
style.
Create datastore directory and all parent dirs if it does not exist
More robust error checking and buffer overflow protection
New mkdirs() and mkdirsn() functions
The sequence start time was only getting set in overlay_interface_init(), which
was not always called, or was called after overlay_gettime_ms() had already
been used.
Added FATALF(), FATAL() and FATAL_perror() macros.
Removed a bunch of debug statements no longer needed in monitor.c and server.c.
- argv is (const char *const *) everywhere, to avoid having to strdup() all the
Java arg strings
- changed (char*) to (const char*) in lots of places to fix cascading compiler
warnings as a result of argv constness
- fixed a bug in "config get" command when used without 'variable' arg
- new test cases to cover new payload with different versions
- exit status of 2 when duplicate is detected
- tighten up other test cases
- fix bug in sqlite_exec_int64()
- adding a bundle using an existing manifest with an incorrect payload should
fail with an error
- adding a bundle using an existing manifest to update the payload to a new
version should succeed
- improve format of "dna rhizome list" output to one bundle per line
- If a bundle has the same payload hash and manifest.name as one already in
the rhizome store, then the "dna rhizome add" command will not add it
- Now passes fourth test case in tests/dna_rhizome
