Commit Graph

181 Commits

Author SHA1 Message Date
Jeremy Lakeman
7d1714d398 Refactor database writes to reduce open/close calls 2013-07-18 13:52:42 +09:30
Jeremy Lakeman
a4bf4edbfc Write out of order packets with MDP transport
- add random order writing to rhizome store API
- remove rhizome flush, force all callers to provide buffer
2013-07-17 16:57:59 +09:30
Jeremy Lakeman
8ac359e550 Use consistent integer types for 64bit compilation 2013-07-13 14:47:06 +09:30
Jeremy Lakeman
a1ac7febe2 Add rhizome fetch status html page 2013-07-11 12:32:06 +09:30
Jeremy Lakeman
62a4cd407d Identify a journal bundle by the presense of a tail field 2013-07-08 14:57:47 +09:30
Jeremy Lakeman
e758e0130f Create journal append command 2013-07-08 14:29:08 +09:30
Jeremy Lakeman
53f3920b90 Move cli global state into a structure and pass it around 2013-07-03 16:51:27 +09:30
Jeremy Lakeman
8249f65f8f Don't create unused import folder 2013-07-03 10:54:05 +09:30
Jeremy Lakeman
adbbed5275 Add new rhizome sync protocol 2013-06-19 12:05:57 +09:30
Jeremy Lakeman
b67516558e Add clean command to verify all bundles 2013-06-14 11:58:19 +09:30
Jeremy Lakeman
96c0889f9a Don't ask for manifests if we don't have room in transfer queues 2013-04-11 15:24:41 +09:30
Andrew Bettison
6eb08ae805 Add new "config dump" command - not working yet
Add cf_fmt_xxx() functions for converting configuration structs into a
config object model tree.
2013-02-26 18:09:08 +10:30
Jeremy Lakeman
90386ce1b1 Merge remote-tracking branch 'origin/development' into serial
Conflicts:
	rhizome.h
	rhizome_database.c
	rhizome_fetch.c
	rhizome_store.c
2013-02-20 17:23:03 +10:30
Jeremy Lakeman
73786bcb5d Refactor all rhizome reading and writing
- The API in rhizome_store.c is used for all reading and writing
- external storage is now usable for all transport options
2013-02-20 16:18:59 +10:30
Andrew Bettison
390655580d Add 'rhizome delete' and 'rhizome clean' commands
sqlite_void_exec() and its ilk now return the count of changed rows, not
just zero, on success

sqlite_exec_prepared() and its ilk now return the count of rows (number
of step results SQLITE_ROW), instead of just zero, on success

rhizome_clean() function now produces an optional report of its changes

rhizome_fail_write() and rhizome_finish_write() now log WARNings not
ERRORs if the SQL DELETE FROM FILES or DELETE FROM FILEBLOBS statements
fail

Refactor rhizome_open_read() to use sqlite_exec_int64()

Ensure that 'rhizome extract' and 'rhizome dump' commands return exit
status of 1 in "not found" conditions, not 255, which is reserved for
errors

Test cases for four new commands: 'rhizome delete bundle', 'rhizome
delete manifest', 'rhizome delete payload' and 'rhizome delete file'
(no test case for 'rhizome clean' yet)
2013-02-20 15:37:09 +10:30
gardners
641d749ab4 Create files for large rhizome bundles
- configurable size threashold
2013-02-20 15:36:32 +10:30
Jeremy Lakeman
e4d6520c46 Don't request manifests that are already in the fetch queue 2013-02-20 15:36:25 +10:30
Jeremy Lakeman
ba1800012e Only advertise BAR's on an alarm
- we no longer hit the database for every outgoing packet, attempting to announce bundles
- we no longer advertise manifests periodically
- when an interesting bar arrives, we ask for the manifest to be announced, which uses the existing packet format
2013-02-20 15:36:23 +10:30
Jeremy Lakeman
340003ed99 Don't keep sql blob open from http server 2013-01-30 11:46:10 +11:00
Jeremy Lakeman
3a93fee8a5 New JNI interface to better support cursor result sets 2013-01-23 14:43:50 +10:30
Jeremy Lakeman
ad01836d0e Use database query to respond to rhizome list, add name argument 2013-01-17 11:45:45 +10:30
Jeremy Lakeman
b25ff45438 Importing the same bundle twice returns duplicate flag 2013-01-15 10:32:48 +10:30
Jeremy Lakeman
5a82bfcf31 Add support for importing and exporting a file with an appended manifest 2013-01-11 14:30:52 +10:30
Jeremy Lakeman
2b480c1d1a Reinstate filesize and hash file extract output 2013-01-04 12:01:40 +10:30
Jeremy Lakeman
28a05baa54 Enable encryption of rhizome payloads 2013-01-03 16:12:24 +10:30
Jeremy Lakeman
dc43b83c80 Tidy up rhizome secret handling to improve reusability 2013-01-03 12:14:13 +10:30
Jeremy Lakeman
2967d1e00f Don't check for duplicates if an existing id has been supplied 2013-01-03 11:18:30 +10:30
Jeremy Lakeman
66deb575d3 Add offset to nonce instead of replacing byte values 2013-01-02 15:05:22 +10:30
Jeremy Lakeman
8b045dd1a6 Extract files based on the manifest id
Renamed rhizome extract file to rhizome dump file
Added rhizome extract file [manifest] [filepath] [pins]
Modified tests to use the appropriate command, assuming that MeshMS payloads will be encrypted
2013-01-02 12:58:05 +10:30
Jeremy Lakeman
6c7ba438a3 Refactor rhizome extract manifest into more useful methods 2013-01-02 11:15:03 +10:30
Jeremy Lakeman
a492c05d64 Re-add database cleanup 2012-12-31 15:09:12 +10:30
Jeremy Lakeman
57439b2162 Rhizome extract file will now log an error on hash mismatch 2012-12-28 11:34:22 +10:30
Jeremy Lakeman
72bc597e92 Add encryption support to writing of rhizome content 2012-12-27 09:39:10 +10:30
Jeremy Lakeman
f64de66b34 Refactor add via HTTP to be equivalent to command line add 2012-12-20 15:36:07 +10:30
Jeremy Lakeman
181d1363f1 Migrate command line import to new storage api 2012-12-19 16:16:49 +10:30
Jeremy Lakeman
c5d76a057f Add storage api for importing files 2012-12-18 10:51:12 +10:30
Jeremy Lakeman
b2a0f8caf5 Rhizome add streams directly into the database 2012-12-17 15:41:27 +10:30
Andrew Bettison
5985df751d Overhaul debug flags
Replace debugflags_t and DEBUG_XXX bit masks with config schema "debug.xxx"
entries.

No more support for "debug.all".
2012-12-11 15:59:46 +10:30
Andrew Bettison
8425882ffc Merge branch 'newconfig' into 'master'
Conflicts:
	commandline.c
	conf.h
	dataformats.c
	log.h
	overlay_address.c
	overlay_interface.c
	packetformats.c
	rhizome.c
	serval.h
	tests/directory_service
	vomp_console.c
2012-12-07 14:09:55 +10:30
Andrew Bettison
caa209fc1d Integrate new config into servald
Not passing any tests yet, but compiles and links and simple uses do not
SEGV.
2012-12-04 14:12:28 +10:30
gardners
e08593b466 stream manifest to buffer, and files to blob in database instead
of via files.
NOT COMPLETE - needs to calculate progressive hash and decide
if file is ok at end of import.
2012-12-03 16:52:08 +10:30
gardners
dbb1fe8d1e rhizome over mdp now writes content for in-order packets, and
import gets triggered, but file hash currently doesn't match.
2012-11-30 22:42:28 +10:30
gardners
79c90dde6e now rhizome mdp replies get parsed and the appropriate slot, if
any, is located.
2012-11-30 22:26:31 +10:30
gardners
b91e832ad7 added improved control of rhizome http and mdp servers via
separate config options with their own predicate functions.
2012-11-30 14:47:27 +10:30
gardners
faad1f26b1 Pass SID as well as ip:port through rhizome fetch request logic.
This is in preparation for rhizome over MDP.
2012-11-29 15:38:04 +10:30
Andrew Bettison
b66d8d6c5b Fix bug in 'rhizome direct pull'
Was not fetching payload of remote bundle, just manifest.  The problem was
caused by a change of logic recently to not activate any queued fetch
candidates immediately, but wait until the next fd_poll(), so that parsing a
single packetful of rhizome advertisments would start fetching the most
important one first, instead of the first one parsed.
2012-11-12 16:24:38 +10:30
gardners
b23b28b16d Changed rhizome manifest signature block type code semantics to
allow signature blocks of 256 bytes, and to allow multiple
signature types with same length. #34
2012-10-29 16:07:42 +10:30
Jeremy Lakeman
de95bb3971 Gracefully disable rhizome if the database cannot be opened
- close database after every command line operation
- don't cache rhizome enabled configuration
- don't send advertisements unless the database is open and the web server is running
- don't provess advertisements unless the database is open
2012-10-29 13:05:03 +10:30
Andrew Bettison
39fc4ce6de Issue #30, remove periodic Rhizome fetch alarm
Replace the main-loop scheduled periodic alarm with an "activate" alarm that is
scheduled whenever a fetch candidate is added to any queue, unless the alarm is
already scheduled.

Replace the "rhizome.fetch_interval_ms" config item with
"rhizome.fetch_delay_ms" [default 50], which is the number of milliseconds
between adding a fetch candidate and firing the "activate" alarm.  This allows
time for a few more Rhizome advertisment packets to arrive after the first one,
before deciding which fetches to start first.

Add new `is_scheduled()` alarm primitive.
2012-10-24 15:13:50 +10:30
Andrew Bettison
935a545ac7 Issue #30, fix Rhizome fetch slot allocation logic
Also improve the block comment documentation of many functions.
2012-10-24 12:08:41 +10:30