Commit Graph

208 Commits

Author SHA1 Message Date
gardners
17ea1b69b3 don't look at all BARs if we are too slow. 2013-02-20 15:36:29 +10:30
Jeremy Lakeman
e4d6520c46 Don't request manifests that are already in the fetch queue 2013-02-20 15:36:25 +10:30
Jeremy Lakeman
ba1800012e Only advertise BAR's on an alarm
- we no longer hit the database for every outgoing packet, attempting to announce bundles
- we no longer advertise manifests periodically
- when an interesting bar arrives, we ask for the manifest to be announced, which uses the existing packet format
2013-02-20 15:36:23 +10:30
Andrew Bettison
cf13a4b51e Add 'rhizome.clean_on_open' config option 2013-02-20 14:43:20 +10:30
Jeremy Lakeman
3a93fee8a5 New JNI interface to better support cursor result sets 2013-01-23 14:43:50 +10:30
Jeremy Lakeman
ad01836d0e Use database query to respond to rhizome list, add name argument 2013-01-17 11:45:45 +10:30
Jeremy Lakeman
6dcc434541 Add more manifest fields as database columns 2013-01-16 10:56:09 +10:30
Jeremy Lakeman
b25ff45438 Importing the same bundle twice returns duplicate flag 2013-01-15 10:32:48 +10:30
Jeremy Lakeman
09617b9ce0 Set sqlite temp directory explicitly, increase database timeout 2013-01-06 19:43:14 +10:30
Jeremy Lakeman
7978bc0b1a Add logging for sqlite errors 2013-01-06 14:18:36 +10:30
Jeremy Lakeman
c7cf80b352 Use sqlite user schema version to remember state 2013-01-06 13:04:49 +10:30
Jeremy Lakeman
e26522bdc6 Minor improvements to command line time measurements 2013-01-03 14:16:33 +10:30
Jeremy Lakeman
2e761c17fa Only consider duplicate manifests that we can re-author 2013-01-03 12:23:50 +10:30
Jeremy Lakeman
2967d1e00f Don't check for duplicates if an existing id has been supplied 2013-01-03 11:18:30 +10:30
Jeremy Lakeman
8b045dd1a6 Extract files based on the manifest id
Renamed rhizome extract file to rhizome dump file
Added rhizome extract file [manifest] [filepath] [pins]
Modified tests to use the appropriate command, assuming that MeshMS payloads will be encrypted
2013-01-02 12:58:05 +10:30
Jeremy Lakeman
6c7ba438a3 Refactor rhizome extract manifest into more useful methods 2013-01-02 11:15:03 +10:30
Jeremy Lakeman
a492c05d64 Re-add database cleanup 2012-12-31 15:09:12 +10:30
Jeremy Lakeman
898cddfcd3 Tidy up rhizome extract return code 2012-12-31 13:21:37 +10:30
Jeremy Lakeman
5b48a85f7d Don't clean up database on every open call 2012-12-28 13:17:04 +10:30
Jeremy Lakeman
c8fbef9016 Fix transaction error checking 2012-12-28 12:47:21 +10:30
Jeremy Lakeman
04eaf99243 Added transaction around file blob creation to ensure rowid is correctly retrieved 2012-12-28 12:10:42 +10:30
Jeremy Lakeman
57439b2162 Rhizome extract file will now log an error on hash mismatch 2012-12-28 11:34:22 +10:30
Jeremy Lakeman
3f45623d04 New stress test, rhizome add while transferring 2012-12-27 16:51:31 +10:30
Jeremy Lakeman
72bc597e92 Add encryption support to writing of rhizome content 2012-12-27 09:39:10 +10:30
Jeremy Lakeman
f64de66b34 Refactor add via HTTP to be equivalent to command line add 2012-12-20 15:36:07 +10:30
Jeremy Lakeman
181d1363f1 Migrate command line import to new storage api 2012-12-19 16:16:49 +10:30
Jeremy Lakeman
1a26a3e452 Reduce log spam 2012-12-12 10:02:43 +10:30
Andrew Bettison
5985df751d Overhaul debug flags
Replace debugflags_t and DEBUG_XXX bit masks with config schema "debug.xxx"
entries.

No more support for "debug.all".
2012-12-11 15:59:46 +10:30
Andrew Bettison
08fc41f893 Merge branch 'master' 2012-12-07 14:12:43 +10:30
Andrew Bettison
8425882ffc Merge branch 'newconfig' into 'master'
Conflicts:
	commandline.c
	conf.h
	dataformats.c
	log.h
	overlay_address.c
	overlay_interface.c
	packetformats.c
	rhizome.c
	serval.h
	tests/directory_service
	vomp_console.c
2012-12-07 14:09:55 +10:30
gardners
be9cefc5ba fixed bug when reporting hash of files during rhizome extract file. 2012-12-07 13:57:26 +10:30
gardners
f827c7c1c8 fixed various problems following split of files into fileblobs 2012-12-04 16:47:45 +10:30
gardners
3ace113be1 fixed error in orphan file cleanup sql statement. 2012-12-04 16:19:01 +10:30
gardners
fc0c134cbf added speed indication on completion of rhizome transfer.
split files table into files and fileblobs to avoid pathological
delays when updating datavalid flag in files table (sqlite copies
the whole table row, including possibly large blob).
2012-12-04 16:03:56 +10:30
Andrew Bettison
caa209fc1d Integrate new config into servald
Not passing any tests yet, but compiles and links and simple uses do not
SEGV.
2012-12-04 14:12:28 +10:30
gardners
de7a4ce2e3 fixed bugs with rhizome streaming direct to database.
that now works, but rhizome direct pull now fails.
2012-12-03 20:51:14 +10:30
gardners
adda5b5571 fixed bug when dropping incomplete file fetch row. 2012-12-03 17:10:21 +10:30
gardners
b39c71b206 factor out code for creating empty blob for inserting file into. 2012-12-03 16:19:01 +10:30
gardners
3e7e29f5f5 improve handling of incomplete files in rhizome database (clean
all out on start up, but only clean out specific ones when inserting).
2012-12-03 12:20:50 +10:30
Andrew Bettison
c4b90a108b Fix compile warnings on Solaris (gcc 4.4) 2012-11-28 14:13:25 +10:30
Andrew Bettison
21a0f31ae4 Merge branch 'whacktmps' into 'master' 2012-11-15 13:41:45 +10:30
Andrew Bettison
b04564b6c9 Fix bug in 'rhizome list' command
Missing RETURN(), was using 'return' instead.
2012-11-12 14:38:50 +10:30
Andrew Bettison
3d55a019c0 Quieten some unconditional DEBUG messages 2012-11-08 10:23:57 +10:30
Andrew Bettison
3c7eb7b058 Add missing sqlite3_blob_close()
Fixes #33.
2012-10-30 16:12:40 +10:30
Jeremy Lakeman
de95bb3971 Gracefully disable rhizome if the database cannot be opened
- close database after every command line operation
- don't cache rhizome enabled configuration
- don't send advertisements unless the database is open and the web server is running
- don't provess advertisements unless the database is open
2012-10-29 13:05:03 +10:30
Andrew Bettison
259b9a9d24 Logging/debug improvements: __WHENCE__, __whence
Introduce __WHENCE__ macro and a block comment in log.h explaining it.

In "primitive" kinds of functions, rename 'whence' arguments to '__whence' and
use WHYF(), WARNF(), DEBUGF() macros instead of calling logMessage() directly.
2012-10-16 17:00:03 +10:30
Andrew Bettison
52ccd07020 Issue #17, improve speed of "rhizome extract manifest"
In the case that the MANIFESTS 'author' column is not NULL, do not perform a
full bundle secret verification in order to clear the '.readonly' flag, just
check whether the author's SID is present in the keyring with a proper-size
rhizome secret.
2012-10-15 18:33:44 +10:30
Andrew Bettison
f6d9e6cb0e Issue #17, improve "rhizome extract manifest" logic
Do not store negatives in the MANIFESTS 'author' column, just positives, so
that newly-unlocked identities are still checked.

Comment in rhizome_database.c about the semantics of the MANIFESTS.author
column.
2012-10-15 18:08:31 +10:30
Andrew Bettison
c43a7ba8e6 Issue #17, improve "rhizome extract manifest" command
Add ".readonly" output field, add logic to detect bundle author and update
MANIFESTS table 'author' column accordingly.

Add two 'rhizomeops' test cases to deal with authorless logic.

Rename some 'rhizomeops' test cases for consistency and to help filtering.
2012-10-15 16:48:04 +10:30
Andrew Bettison
f4d0b7a16a Fix some minor bugs in recent commits 2012-10-10 18:13:31 +10:30
Andrew Bettison
2a8cfb0404 Quieten Rhizome advertisement debug logging
Introduce DEBUG_RHIZOME_ADS and sqlite_set_debugmask(), use in
rhizome_packetformats.c advertisement mustering and processing code.
2012-10-10 13:22:30 +10:30
Andrew Bettison
3678522872 Issue #17, add AUTHOR column to Rhizome MANIFESTS table
Replace ".selfsigned" column with ".author" and ".fromhere" columns in
output of "rhizome list" command.  (Note that a "sender" column is
already present.)

Add 'author' field to struct rhizome_manifest.

Log all fully rendered SQL statements on DEBUG_RHIZOME.

Update 'rhizomeops' test cases and improve the assert_rhizome_list()
test function to be able to assert authorship of files.
2012-10-09 17:48:06 +10:30
Andrew Bettison
89343c69cd Merge branch 'rhizomedirect' into 'master' 2012-10-05 17:45:30 +09:30
gardners
d685953ff5 Fixed storage of signatures with manifests in a codepath that
was missed in commit 1c72778c50.
2012-10-05 05:53:11 +02:00
Andrew Bettison
3040d2b665 Add Rhizome DB retries on SQL statement preparation
It turns out that if the DB is locked, sqlite_prepare_v2() call can return
SQLITE_BUSY.  The retry logic (implemented for issue #2) only provided for
sqlite_step() to return SQLITE_BUSY.  It was a fairly straightforward matter to
extend the retry logic to cover statement preparation in an equally general
fashion.

The problem was observed while diagnosing failures in the rhizomeprotocol
DirectPush test case: the "servald rhizome list" command was failing due to a
locked database.  See issue #9.
2012-10-04 14:30:20 +09:30
Andrew Bettison
37d4b4fa3a Issue #9, fixed a compiler warning and bug
The warning was that 'fd' could be used uninitialised in rhizome_store_file(),
and it was true!
2012-10-04 11:42:13 +09:30
gardners
295f7c68ec some rhizome schema cleanups, including importantly making filesize
an integer not text column, so that FILESIZE BETWEEN clauses behave
numerically rather than lexographically.  Added an index for good
measure. #9
2012-10-03 17:54:39 +02:00
gardners
132cfd6fec Fixed bug in rhizome_store_file(), where it tried to open the
associated file before checking if it was already in the database.
Rhizome Direct can supply a manifest without associated file if
the file is already in the database, and so it was breaking.
Also removed "assert bundle_received_by"'s from rhizome direct
pull and sync tests because they are not needed, and were failing
because the same file contents was used for the files being
exchanged, and so file storage was not occurring, and thus the log
message being looked for was not being produced.
Push, pull and sync tests in rhizomeprotocol now pass, leaving
only two tests in error.  #9
2012-10-03 15:50:43 +02:00
gardners
1c72778c50 fixed problem with rhizome direct push / manifest import where
signatures were not being recorded.  Also fixed separate issue
where rhizomeprotocols tests expected selfsigned to be 1 on
receiver end, when it will never be, because the BK doesn't match.
rhizome direct push test in rhizomeprotocols now passes. #9
2012-10-03 12:28:23 +02:00
gardners
92a5423b17 Moved rhizome direct imports to be in-process instead of
out-of-process. rhizomeprotocol push test still fails.
Investigating. #9
2012-10-03 12:28:21 +02:00
Andrew Bettison
bbe6d12693 Issue #9, fix 'rhizome direct push' command
Now the 'rhizomeprotocol' Push test case now passes.  It should be renamed to
DirectPush.

Much refactoring of the Rhizome Direct HTTP request parsing.  Now uses
strbuf_sprintf() instead of snprintf() in many places to check for buffer
overrun and ensure terminating nul.  Still more of this kind of work is needed.

Improved debug that needs to be made conditional on DEBUG_RHIZOME_RX and
DEBUG_RHIZOME_TX.  Some just needs removal.
2012-10-03 17:51:37 +09:30
Jeremy Lakeman
ac3864ff20 Reuse command line parsing framework with monitor interface 2012-09-25 13:31:34 +09:30
Paul Gardner-Stephen
bba6839656 Initial stab at porting to Solaris.
It compiles without warning (with CC=gcc) but doesn't link
because NaCL doesn't build yet.
2012-09-05 20:42:50 +09:30
Andrew Bettison
3f1e0dd763 Finish sleep-retry on locked Rhizome db
Closes #2.

Rewrite all Rhizome db query code using new retry primitives defined in
"rhizome.h": sqlite_step_retry(), sqlite_retry(), sqlite_retry_done(), etc.
Replace all calls to sqlite3_prepare_v2() with sqlite_prepare() which does
proper error logging.

Fix bug: re-invoking sqlite3_blob_close() on SQLITE_BUSY return causes process
to abort.  Use an explicit BEGIN...COMMIT around the blob writing code instead.

Tested using repeated invocations of batphone/tests/meshms1.

Delete deprecated Rhizome db code in rhizome_crypto.c that has been replaced
with keyring file.
2012-08-24 15:38:39 +09:30
Andrew Bettison
bb4819aa83 Issue #2, sleep-retry logic in all Rhizome operations
Much refactoring and removal of cruft.

SQL query errors are now logged with the filename, line number and function
where they were invoked, not of the low-level function that discovered the
error.  This makes use of the new __HERE__ notation introduced last commit.
2012-08-23 17:43:35 +09:30
Andrew Bettison
fd3da58a7c Issue #2, add sleep-retry logic for most database queries
All the queries that used sqlite_exec_void() and sqlite_exec_int64() and
sqlite_exec_strbuf() now do a sleep-retry while the Rhizome db is locked.

There are other queries that still need conversion, and some old infinite
retry logic that needs replacing.
2012-08-22 19:09:30 +09:30
Andrew Bettison
d8fd9fa411 Refactor nanosleep(2) calls into sleep_ms() function 2012-08-21 13:35:33 +09:30
Andrew Bettison
5cbc2167ec Start on problem of database lock errors
Add sqlite_exec_void_retry() function, use it in
rhizome_update_file_priority().  This should be reviewed to ensure that the
server process never sleeps.

The general problem remains of what the servald process should do if the
database is locked when it tries to update.  Simplest solution is to sleep and
retry, but that blocks all other services and would hurt VoMP.  A better
solution would be for each Rhizome operation to collect its database updates
into a single transaction and place that in a work queue that gets called using
schedule() (or even watch() if a file-descriptor event can somehow be used when
the database becomes available).  Another solution is perhaps to perform all
Rhizome operations in a dedicated process that can block indefinitely on the
database without affecting servald responsiveness.
2012-08-20 18:43:48 +09:30
Andrew Bettison
12d8bc0c12 Use new typedef time_ms_t everywhere 2012-08-09 12:14:32 +09:30
Andrew Bettison
3c09a1ec7d Fix broken log WARN messages in "rhizome list" 2012-07-17 18:07:02 +09:30
Andrew Bettison
bc11b16d96 Remove spurious DEBUG() statement 2012-07-17 15:33:04 +09:30
Andrew Bettison
b50943d09c Fix handling of empty payload (filesize = 0)
Do not add 'filehash' var to manifest if filesize=0
Do not accept 'filehash' var when parsing manifest with filesize=0
When responding to a new rhizome advertisement, do not try to HTTP
request a payload if filesize=0, just import the manifest directly
Various operations, eg "rhizome file add", do not report 'filehash'
fields where 'filesize' is zero
Do not delete rows from MANIFESTS table which have empty filehash
Various related bug fixes
2012-07-16 18:40:30 +09:30
Andrew Bettison
ff2c98afb3 Add "rhizome.fetch_interval_ms" config option 2012-07-11 16:51:30 +09:30
Jeremy Lakeman
495de9e0ec Add more profiling instrumentation 2012-07-03 15:50:04 +09:30
Jeremy Lakeman
29cba17891 Fix rhizome transfers 2012-06-27 16:54:42 +09:30
Andrew Bettison
208b9c15fd Factor out rhizome_hex_to_bytes() and chartonybl() 2012-06-25 14:51:21 +09:30
Andrew Bettison
0997909053 Fix import bundle test and bug
Add lots of debug to track down cause of rhizome list .selfsigned column bug
2012-06-15 18:10:10 +09:30
Andrew Bettison
8fca6b5c5e Rename "rhizome list" output column headers
_inserttime -> .inserttime
_selfsigned -> .selfsigned
2012-06-13 18:11:14 +09:30
Andrew Bettison
eeb8a22495 Fix rhizome list column count field 2012-06-12 18:13:21 +09:30
Andrew Bettison
56cb12f507 Refactor rhizome db creation and execution functions 2012-06-12 18:12:36 +09:30
Andrew Bettison
637ef02f30 Add _selfsigned column to "rhizome list" output
Add <pin,pin...> arg1 to "rhizome list" command, so it can open the keyring

Some tests in tests/rhizome fail
2012-06-08 18:25:43 +09:30
Andrew Bettison
7a71a521f8 Pass author SID parameters in binary, not hex 2012-06-08 15:27:15 +09:30
Andrew Bettison
86c7819f97 Remove spurious ERROR messsages
Tests assert that stderr contains no ERROR: lines after a successful exit

Rewrote sqlite_exec_int64() to separate error outcomes from legitimate
result values

Changed several WHY() calls to DEBUG()

Improved test framework
2012-06-08 13:13:26 +09:30
Andrew Bettison
b14db5c28b Improve "rhizome add file" for empty payloads
Accept the empty string for a payload pathname to mean a zero-length
payload.
2012-06-05 15:45:53 +09:30
Andrew Bettison
a9ad1b6afc Add <bsk> optional arg to "rhizome add file"
Improve regular expressions for common data types in test scripts

Revert column count field delimiter in "rhizome list" from ":" to "\n"

Add a few more test cases
2012-06-05 13:58:59 +09:30
gardners
cdf2753412 fixed filesize display bug for "rhizome list"
(%u is 32bit on android, but filesize is 64bit)
2012-05-28 13:51:24 +02:00
Jeremy Lakeman
0f214ed46f Adjust database schema 2012-05-28 19:24:02 +09:30
Jeremy Lakeman
36389d2b78 Insert manifest within a transaction 2012-05-28 14:08:01 +09:30
Jeremy Lakeman
0e75cbdcad Document recommended sql queries 2012-05-28 09:06:07 +09:30
gardners
4da9824ab0 fixed bug in storing file that could cause entertaining file hash
problems.
2012-05-27 14:44:53 +02:00
gardners
9d811876c7 added code to make sure that we only store the original number
of bytes when storing a growing journal.  Also make sure that
hash matches.
2012-05-27 14:24:55 +02:00
gardners
30e53f963c fixed various manifest synchronisation bugs, including probably
the main one that was stopping meshms (large manifests would
practically never get advertised under certain (common) conditions).
2012-05-27 08:31:48 +02:00
gardners
58cc45043b added code to encrypt files when storing into the database. 2012-05-26 11:36:19 +02:00
gardners
2de6bb025a rhizome_extract_file can now decrypt a file, and also uses
progressive blob operations so that we can extract files of
unlimited size.
2012-05-26 11:36:19 +02:00
gardners
2cd9aca1f8 fixed bugs related to dropping files from rhizome database. 2012-05-26 11:36:19 +02:00
gardners
96b13bdadc fixed error return codes for retrieve_file() 2012-05-26 11:36:18 +02:00
gardners
596e25c16e cleanups for extracting files from rhizome. 2012-05-26 11:36:18 +02:00
gardners
c2dfefe764 fixed bug in duplicate detection 2012-05-26 11:36:18 +02:00
gardners
04cc05d6e1 refactored manifest verification into a separate function
that is called only when needed, and marks a manifest as finalised
if the verifcation fails.  reading a manifest now never sets
finalised flag, as either _finalise() or _verify() must be called.
2012-05-26 11:36:18 +02:00
gardners
8161709c4e work on fixing handling of duplicate rhizome bundle debouncing.
(seems to work from command line, but not from the test framework).
2012-05-26 11:36:18 +02:00