Commit Graph

169 Commits

Author SHA1 Message Date
Jeremy Lakeman
90386ce1b1 Merge remote-tracking branch 'origin/development' into serial
Conflicts:
	rhizome.h
	rhizome_database.c
	rhizome_fetch.c
	rhizome_store.c
2013-02-20 17:23:03 +10:30
Jeremy Lakeman
73786bcb5d Refactor all rhizome reading and writing
- The API in rhizome_store.c is used for all reading and writing
- external storage is now usable for all transport options
2013-02-20 16:18:59 +10:30
Andrew Bettison
390655580d Add 'rhizome delete' and 'rhizome clean' commands
sqlite_void_exec() and its ilk now return the count of changed rows, not
just zero, on success

sqlite_exec_prepared() and its ilk now return the count of rows (number
of step results SQLITE_ROW), instead of just zero, on success

rhizome_clean() function now produces an optional report of its changes

rhizome_fail_write() and rhizome_finish_write() now log WARNings not
ERRORs if the SQL DELETE FROM FILES or DELETE FROM FILEBLOBS statements
fail

Refactor rhizome_open_read() to use sqlite_exec_int64()

Ensure that 'rhizome extract' and 'rhizome dump' commands return exit
status of 1 in "not found" conditions, not 255, which is reserved for
errors

Test cases for four new commands: 'rhizome delete bundle', 'rhizome
delete manifest', 'rhizome delete payload' and 'rhizome delete file'
(no test case for 'rhizome clean' yet)
2013-02-20 15:37:09 +10:30
gardners
641d749ab4 Create files for large rhizome bundles
- configurable size threashold
2013-02-20 15:36:32 +10:30
Jeremy Lakeman
e4d6520c46 Don't request manifests that are already in the fetch queue 2013-02-20 15:36:25 +10:30
Jeremy Lakeman
ba1800012e Only advertise BAR's on an alarm
- we no longer hit the database for every outgoing packet, attempting to announce bundles
- we no longer advertise manifests periodically
- when an interesting bar arrives, we ask for the manifest to be announced, which uses the existing packet format
2013-02-20 15:36:23 +10:30
Jeremy Lakeman
340003ed99 Don't keep sql blob open from http server 2013-01-30 11:46:10 +11:00
Jeremy Lakeman
3a93fee8a5 New JNI interface to better support cursor result sets 2013-01-23 14:43:50 +10:30
Jeremy Lakeman
ad01836d0e Use database query to respond to rhizome list, add name argument 2013-01-17 11:45:45 +10:30
Jeremy Lakeman
b25ff45438 Importing the same bundle twice returns duplicate flag 2013-01-15 10:32:48 +10:30
Jeremy Lakeman
5a82bfcf31 Add support for importing and exporting a file with an appended manifest 2013-01-11 14:30:52 +10:30
Jeremy Lakeman
2b480c1d1a Reinstate filesize and hash file extract output 2013-01-04 12:01:40 +10:30
Jeremy Lakeman
28a05baa54 Enable encryption of rhizome payloads 2013-01-03 16:12:24 +10:30
Jeremy Lakeman
dc43b83c80 Tidy up rhizome secret handling to improve reusability 2013-01-03 12:14:13 +10:30
Jeremy Lakeman
2967d1e00f Don't check for duplicates if an existing id has been supplied 2013-01-03 11:18:30 +10:30
Jeremy Lakeman
66deb575d3 Add offset to nonce instead of replacing byte values 2013-01-02 15:05:22 +10:30
Jeremy Lakeman
8b045dd1a6 Extract files based on the manifest id
Renamed rhizome extract file to rhizome dump file
Added rhizome extract file [manifest] [filepath] [pins]
Modified tests to use the appropriate command, assuming that MeshMS payloads will be encrypted
2013-01-02 12:58:05 +10:30
Jeremy Lakeman
6c7ba438a3 Refactor rhizome extract manifest into more useful methods 2013-01-02 11:15:03 +10:30
Jeremy Lakeman
a492c05d64 Re-add database cleanup 2012-12-31 15:09:12 +10:30
Jeremy Lakeman
57439b2162 Rhizome extract file will now log an error on hash mismatch 2012-12-28 11:34:22 +10:30
Jeremy Lakeman
72bc597e92 Add encryption support to writing of rhizome content 2012-12-27 09:39:10 +10:30
Jeremy Lakeman
f64de66b34 Refactor add via HTTP to be equivalent to command line add 2012-12-20 15:36:07 +10:30
Jeremy Lakeman
181d1363f1 Migrate command line import to new storage api 2012-12-19 16:16:49 +10:30
Jeremy Lakeman
c5d76a057f Add storage api for importing files 2012-12-18 10:51:12 +10:30
Jeremy Lakeman
b2a0f8caf5 Rhizome add streams directly into the database 2012-12-17 15:41:27 +10:30
Andrew Bettison
5985df751d Overhaul debug flags
Replace debugflags_t and DEBUG_XXX bit masks with config schema "debug.xxx"
entries.

No more support for "debug.all".
2012-12-11 15:59:46 +10:30
Andrew Bettison
8425882ffc Merge branch 'newconfig' into 'master'
Conflicts:
	commandline.c
	conf.h
	dataformats.c
	log.h
	overlay_address.c
	overlay_interface.c
	packetformats.c
	rhizome.c
	serval.h
	tests/directory_service
	vomp_console.c
2012-12-07 14:09:55 +10:30
Andrew Bettison
caa209fc1d Integrate new config into servald
Not passing any tests yet, but compiles and links and simple uses do not
SEGV.
2012-12-04 14:12:28 +10:30
gardners
e08593b466 stream manifest to buffer, and files to blob in database instead
of via files.
NOT COMPLETE - needs to calculate progressive hash and decide
if file is ok at end of import.
2012-12-03 16:52:08 +10:30
gardners
dbb1fe8d1e rhizome over mdp now writes content for in-order packets, and
import gets triggered, but file hash currently doesn't match.
2012-11-30 22:42:28 +10:30
gardners
79c90dde6e now rhizome mdp replies get parsed and the appropriate slot, if
any, is located.
2012-11-30 22:26:31 +10:30
gardners
b91e832ad7 added improved control of rhizome http and mdp servers via
separate config options with their own predicate functions.
2012-11-30 14:47:27 +10:30
gardners
faad1f26b1 Pass SID as well as ip:port through rhizome fetch request logic.
This is in preparation for rhizome over MDP.
2012-11-29 15:38:04 +10:30
Andrew Bettison
b66d8d6c5b Fix bug in 'rhizome direct pull'
Was not fetching payload of remote bundle, just manifest.  The problem was
caused by a change of logic recently to not activate any queued fetch
candidates immediately, but wait until the next fd_poll(), so that parsing a
single packetful of rhizome advertisments would start fetching the most
important one first, instead of the first one parsed.
2012-11-12 16:24:38 +10:30
gardners
b23b28b16d Changed rhizome manifest signature block type code semantics to
allow signature blocks of 256 bytes, and to allow multiple
signature types with same length. #34
2012-10-29 16:07:42 +10:30
Jeremy Lakeman
de95bb3971 Gracefully disable rhizome if the database cannot be opened
- close database after every command line operation
- don't cache rhizome enabled configuration
- don't send advertisements unless the database is open and the web server is running
- don't provess advertisements unless the database is open
2012-10-29 13:05:03 +10:30
Andrew Bettison
39fc4ce6de Issue #30, remove periodic Rhizome fetch alarm
Replace the main-loop scheduled periodic alarm with an "activate" alarm that is
scheduled whenever a fetch candidate is added to any queue, unless the alarm is
already scheduled.

Replace the "rhizome.fetch_interval_ms" config item with
"rhizome.fetch_delay_ms" [default 50], which is the number of milliseconds
between adding a fetch candidate and firing the "activate" alarm.  This allows
time for a few more Rhizome advertisment packets to arrive after the first one,
before deciding which fetches to start first.

Add new `is_scheduled()` alarm primitive.
2012-10-24 15:13:50 +10:30
Andrew Bettison
935a545ac7 Issue #30, fix Rhizome fetch slot allocation logic
Also improve the block comment documentation of many functions.
2012-10-24 12:08:41 +10:30
Andrew Bettison
a9d3a1387c Issue #30, rewrite Rhizome fetch queues and slots
Causes SEGV in rhizomestress test.
2012-10-23 18:10:20 +10:30
Jeremy Lakeman
f1b0374b97 Extract crypto signing primitives from mdp processing 2012-10-19 14:54:41 +10:30
Andrew Bettison
839de7557c Merge branch 'rhizomestress' into 'master' 2012-10-18 17:57:10 +10:30
Andrew Bettison
65ea612e27 Fix bugs revealed by Rhizome stress test
Overhauled the file fetch queue logic in rhizome_fetch.c.

Now the 'rhizomeprotocol' stress test passes in approximately 5 minutes on my
2009-vintage Dell laptop.

Added a call to rhizome_enqueue_suggestions() in rhizome_fetch_close() so that
a new Rhizome GET request is sent as soon as a fetch slot becomes free, instead
of waiting for the (default 5 second) timer to trigger the next GET.
2012-10-18 17:30:22 +10:30
gardners
499c4018bf Imported ref10/ implementation of crypto_sign from supercop-20120525
as recommended a while back by Dan Bernstein as offering the fastest
implementation of the crypto_sign() primitives for ARM.
Indeed this implementation IS faster. See comparison below for a
Rock 500 handset (800MHz(?) ARM6, no NEON):

Original ref/ implementation on an R500 stock rom (non-rooted)::
mean signature generation time = 96.80ms
mean signature verification time = 272.20ms

ref10/ implementations on an R500 stock rom (non-rooted):
mean signature generation time = 4.00ms
mean signature verification time = 13.00ms

Approximately 20x speed up, just like that :)
2012-10-18 17:26:48 +10:30
Andrew Bettison
259b9a9d24 Logging/debug improvements: __WHENCE__, __whence
Introduce __WHENCE__ macro and a block comment in log.h explaining it.

In "primitive" kinds of functions, rename 'whence' arguments to '__whence' and
use WHYF(), WARNF(), DEBUGF() macros instead of calling logMessage() directly.
2012-10-16 17:00:03 +10:30
Andrew Bettison
52ccd07020 Issue #17, improve speed of "rhizome extract manifest"
In the case that the MANIFESTS 'author' column is not NULL, do not perform a
full bundle secret verification in order to clear the '.readonly' flag, just
check whether the author's SID is present in the keyring with a proper-size
rhizome secret.
2012-10-15 18:33:44 +10:30
Andrew Bettison
3f1f495d68 Issue #17, overhaul manifest author crypto logic
Add test case for new feature of the "rhizome add" command: if the author SID
is not specified (empty arg) then it searches the keyring for the author.

Removed "authorSid" argument from several functions that also take a struct
rhizome_manifest * arg, since the author, if known, is now supplied in the
struct.

Improve return value handling and refactored some rhizome crypto code.
2012-10-11 17:58:24 +10:30
Andrew Bettison
2a8cfb0404 Quieten Rhizome advertisement debug logging
Introduce DEBUG_RHIZOME_ADS and sqlite_set_debugmask(), use in
rhizome_packetformats.c advertisement mustering and processing code.
2012-10-10 13:22:30 +10:30
Andrew Bettison
3678522872 Issue #17, add AUTHOR column to Rhizome MANIFESTS table
Replace ".selfsigned" column with ".author" and ".fromhere" columns in
output of "rhizome list" command.  (Note that a "sender" column is
already present.)

Add 'author' field to struct rhizome_manifest.

Log all fully rendered SQL statements on DEBUG_RHIZOME.

Update 'rhizomeops' test cases and improve the assert_rhizome_list()
test function to be able to assert authorship of files.
2012-10-09 17:48:06 +10:30
Andrew Bettison
89343c69cd Merge branch 'rhizomedirect' into 'master' 2012-10-05 17:45:30 +09:30
Andrew Bettison
d470ac730b Cosmetic improvement for better self-documentation 2012-10-04 18:11:16 +09:30