Commit Graph

90 Commits

Author SHA1 Message Date
Jeremy Lakeman
2ef73884a1 Add configurable limit to ensure a minimum amount of free space is preserved
- default to 100MB of free space
- currently disabled on android due to missing sys/statvfs.h
2014-06-23 13:06:21 +09:30
Jeremy Lakeman
47f051917d Rhizome cleanup will now expire payloads and vacuum the db file. 2014-06-19 14:23:43 +09:30
Jeremy Lakeman
7ff89afcf4 Reinstate rhizome database storage limit
- old / large payloads should be evicted to fit more payloads
- if there isn't enough space, new payloads will not be added
2014-06-19 11:09:09 +09:30
Jeremy Lakeman
2a572d19bf Rework rhizome store to write content more lazily and cleanup more carefully 2014-06-04 17:13:08 +09:30
Jeremy Lakeman
dd9f5fb713 Only report that a payload exists if it actually exists 2014-04-07 13:22:27 +09:30
Andrew Bettison
76cce8fe7e Fix enum compiler warnings (clang -Wenum-conversion) 2014-04-03 10:36:38 +10:30
Andrew Bettison
53c1b1c04c Filesystem Heirarchy Standard (FHS) paths
If no instance directory specified, then use paths like
/etc/serval/serval.conf
/var/cache/serval
/var/log/serval
/var/run/serval
etc. for files, instead of all in a single directory.

Log all directory creation as INFO messages.

Interpretation of log.file.directory_path has changed slightly.

Updated servald configuration tech doc.
2014-03-26 15:35:43 +10:30
Jeremy Lakeman
ee68856878 Reduce log spam from debug.rhizome 2014-03-12 14:32:59 +10:30
Andrew Bettison
0bef066949 Store Rhizome external blob files in "blob" subdirectory
This will make issue #50 a bit easier to tackle
2014-03-07 16:55:38 +10:30
Jeremy Lakeman
974c87adf2 Remove orphan file rows if we notice the payload is missing 2014-03-01 05:45:32 +10:30
Jeremy Lakeman
1a543484bd Fix rhizome key generation for meshms 2014-01-22 15:51:06 +10:30
Jeremy Lakeman
cc51edd39d Fix compiler warnings 2014-01-06 12:36:00 +10:30
Andrew Bettison
ee9c96bb8c Implement HTTP POST /restful/rhizome/insert
Change HTTP request buffer pointers from (const char*) to (char*)
because some Rhizome operations can modify received data in-place, eg,
when decrypting it.
2013-12-30 18:09:56 +10:30
Andrew Bettison
464f0858f1 Refactor rhizome_write_file() to use fd not FILE* 2013-12-30 16:30:43 +10:30
Andrew Bettison
f81883758a Support streaming directly into Rhizome store
The rhizome_open_write() function now accepts RHIZOME_SIZE_UNSET as the
payload size, and streams into an external blob until finished, then
copies the external blob into the FILEBLOBS table if it is small enough.

Needed so that HTTP GET /restful/rhizome/insert can stream the payload
directly into the DB even if the supplied manifest does not specify
'filesize'.
2013-12-30 16:30:43 +10:30
Andrew Bettison
34188fa489 Replace config 'rhizome.external_blobs' with 'rhizome.max_blob_size' 2013-12-30 16:30:43 +10:30
Andrew Bettison
72040517e1 New enum rhizome_payload_status
Refactor a lot of Rhizome bundle storage code to use the new "enum
rhizome_payload_status" instead of mysterious int values to represent
the outcome of the operation.
2013-12-30 16:30:35 +10:30
Andrew Bettison
9ebef81a49 Formalise "rhizome add file" exit status
Formalise add-bundle result in "enum rhizome_bundle_status"

Rewrite rhizome_manifest_finalise(), rhizome_find_duplicate() and
rhizome_add_manifest() to return enum rhizome_bundle_status

New function rhizome_manifest_check_stored() that compares a manifest
with its stored counterpart and returns enum rhizome_bundle_status

Remove redundant rhizome_manifest_check_sanity(), consolidating all
manifest validation rules in rhizome_manifest_validate(), which now
checks the 'id' field is present, and that 'sender' and 'recipient' are
both present for MeshMS

Correct manifest finalisation logic: set the 'finalised' flag in
rhizome_manifest_validate(), not in rhizome_manifest_verify() (which
sets 'selfSigned'), and consistently clear 'finalised' flag in all
attribute setter functions

Remove manifest 'ttl' field and all references thereof (leaving unused
space in Rhizome BAR)

Rename some payload functions for clarity
2013-12-21 14:37:18 +10:30
Andrew Bettison
3b5c0d2092 More Rhizome external blob debug 2013-12-18 17:44:16 +10:30
Andrew Bettison
075f9c7c27 Use uint64_t for Rhizome manifest version
Instead of int64_t.  Fixes some -Wsign-compare warnings.

Replace sqlite_exec_int64() with sqlite_exec_uint64().

Also store rowid as uint64_t, and use 0 not -1 to indicate
unset.
2013-12-11 11:11:34 +10:30
Andrew Bettison
4428a76379 Use ssize_t for read(2) and write(2) return value
Test for errors using == -1 not < 0, and cast to (size_t) subsequently

Fixes some -Wsign-compare warnings
2013-12-10 17:21:35 +10:30
Andrew Bettison
6254709a84 Use RHIZOME_SIZE_UNSET instead of -1 for read_state.length
This may have been a reversion to older code caused by a careless
merge, because the change to RHIZOME_SIZE_UNSET was already made
when read_state.length was changed to unsigned.

Fixes some -Wsign-compare warnings
2013-12-10 17:17:18 +10:30
Andrew Bettison
4af6cf9d6a Fix -Wsign-compare warnings: use size_t or unsigned for byte counts 2013-12-10 16:52:53 +10:30
Andrew Bettison
9b64bb87e0 Uniform copyright notices
Added some missing copyright/license block comments
2013-12-04 17:15:36 +10:30
Andrew Bettison
42ab9aec4c Merge branch 'development' into 'naf4'
Remove two redundant calls to rhizome_retrieve_manifest() in meshms.c,
revealed by an assert() in the stricter manifest parsing code

Fix header files included by socket.h
2013-12-02 17:17:47 +10:30
Andrew Bettison
67a724b29d Fix Android compiler warnings ("%zd" and ssize_t)
Also squash some unconditional DEBUG statements
2013-11-21 17:32:58 +10:30
Andrew Bettison
7564d529a1 Add missing copyright/license comment blocks
Update a few existing copyright notices to reflect recent work
2013-11-21 16:05:18 +10:30
Andrew Bettison
45442d3eb4 Rewrite bundle author authentication
Replaced 'int has_author' manifest element with new 'enum authorship'
element to record the result of author authentication, to avoid
repeating expensive crypto operations.

Separated the handling of bundle secret arguments from author lookup and
authentication.  The new rhizome_apply_bundle_secret(m,bsk) is now
called at the top level to set the manifest secret key (if it
validates), and thereafter there is no need to pass the 'bsk' argument
to any other functions, as they can simply check the 'haveSecret' field
of the manifest.

Removed rhizome_extract_privatekey() which combined author lookup and
bundle secret validation, and replaced it with functions that only deal
with the author: rhizome_lookup_author() and rhizome_authenticate_author().

Renamed other functions to make their purpose and effect clearer.

Formalised the semantics of only storing AUTHENTICATED author SIDs in
the 'author' column of the MANIFESTS table, which necessitated a change
to a 'rhizomeops' test case: when adding a file using a BK-less
manifest, the author column is set to null, so the Rhizome list output
does not show the bundle as ".fromhere" and does not give an author for
that bundle.
2013-11-06 23:58:17 +10:30
Andrew Bettison
bacba19dc1 Refactor manifest: specific setter functions
Replace generic rhizome_manifest_set() and rhizome_manifest_set_ll()
with per-field setter functions, eg, rhizome_manifest_set_filesize().
Struct rhizome_manifest elements for all known fields, to replace the
use of rhizome_manifest_get() and rhizome_manifest_get_ll() everywhere:
sender, recipient, service, name, date, bundle_key.

Add boolean validity flags for binary blob types, to avoid having to compare
with many bytes of all-zero to detect presence, eg, has_sender, has_recipient,
has_author, has_bundle_key.  These maintained by the setter functions.

Rename existing manifest struct elements to be the same as their field
names: fileLength -> filesize, journalTail -> tail.

More use of unsigned int, size_t and uint64_t for payload sizes, offsets, byte
counts, etc. especially in rhizome_store.c and meshms.c.  More uniform use of
size_t to dimension memory buffers.  Fix some printf(3) style format strings
for 64-bit correctness on 32-bit systems.  Use new constant RHIZOME_SIZE_UNSET
instead of -1 to indicate unknown dimension, and explicitly assert its absence
before comparisons and arithmetic, for safety.

Replace some 'int' loop variables with 'unsigned' where appropriate.

Fix bugs discovered in MeshMS bundle private/public key generation and
bundle secret key handling for export/extract commands.

Instrument the first MeshMS test case to aid debugging.

New debug config flag: debug.manifest logs all modifications to all manifest
fields by setter functions.

Rename debug config flag: debug.rhizome_bind -> debug.rhizome_sql_bind.
2013-11-04 19:17:49 +10:30
Andrew Bettison
48921802f5 Use size_t in Rhizome store functions 2013-10-25 00:20:53 +10:30
Andrew Bettison
fa21bec880 Rewrite HTTP server 2013-10-25 00:19:37 +10:30
Andrew Bettison
974c7a56a0 Issue #11: Use rhizome_filehash_t everywhere 2013-10-11 15:35:27 +10:30
Andrew Bettison
ffafae86b1 Issue #11: replace long long with int64_t 2013-10-07 05:54:46 +10:30
Andrew Bettison
7b427d48ee A few more uses of rhizome_bid_t
Now the TEXT_TOUPPER SQL bind action is only used for filehash hex
strings; all Bundle ID parameters are bound using RHIZOME_BID_T.
2013-10-04 04:50:37 +09:30
Andrew Bettison
ab31420faf rhizome_bid_t
Consistent type for internal binary representation of Rhizome Bundle ID
(aka Manifest ID)
2013-10-03 23:16:45 +09:30
Andrew Bettison
7e3a552011 Issue #69: refactor all Rhizome SQL queries to use binding
Remove all uses of sprintf(3) and its ilk to form SQL query commands.
Use the new sqlite_bind() and sqlite_prepare_bind() functions instead.
2013-10-03 15:15:30 +09:30
Andrew Bettison
162108dd18 Fix duplicate payload logic
rhizome_finish_write() was treating a duplicate file hash as an error,
instead of dealing with it sensibly
2013-09-30 16:32:08 +09:30
Andrew Bettison
ae5ae2d394 Cosmetic changes to whitespace
Make it easier to edit in Vim
2013-09-30 16:06:23 +09:30
Jeremy Lakeman
ba3f77e67c Rename external payloads instead of linking and unlinking 2013-09-25 15:51:54 +09:30
Jeremy Lakeman
b38b4720cc Add html stats for http connection and mdp transfer counts 2013-09-24 14:34:51 +09:30
Jeremy Lakeman
8accabfcac Always store payloads with a temporary id 2013-08-27 16:15:51 +09:30
Jeremy Lakeman
ceb7f9d4c9 Fail writing content if it has already been closed 2013-08-26 17:43:06 +09:30
Jeremy Lakeman
9c49abd431 Store journal appends to sql blobs in one burst 2013-08-26 15:37:19 +09:30
Jeremy Lakeman
643b21d6a7 Fetch file payloads if they are missing, even if we already have the manifest 2013-08-22 15:14:21 +09:30
Jeremy Lakeman
f9b828c3dd Remove payload if hash doesn't match when reading back 2013-08-21 15:45:18 +09:30
Jeremy Lakeman
1468d2deb0 Pre-emptively send small payloads via MDP 2013-08-21 09:21:40 +09:30
Jeremy Lakeman
bf7d0d5b16 Cache rhizome read state when serving content via MDP 2013-08-16 14:57:28 +09:30
Jeremy Lakeman
e322114850 Only transfer new journal content via MDP 2013-08-15 14:36:56 +09:30
Jeremy Lakeman
91a600fba7 Add separate rows for delivered and read status
- add explicit transaction around blob writes so the commit can be retried
- remove status columns
- remove SID's from message log output to reduce size and complexity
2013-08-02 14:02:56 +09:30
Jeremy Lakeman
94381bab49 Fix non-journal, non-block based payload encryption 2013-07-26 15:13:22 +09:30