Commit Graph

117 Commits

Author SHA1 Message Date
Jeremy Lakeman
94cf79f0f3 Improve resilience to database locking issues 2016-07-25 16:54:11 +09:30
Jeremy Lakeman
dbded493c1 Refactor meshms ply handling into separate source file 2016-07-25 16:54:11 +09:30
Jeremy Lakeman
c56b733391 Fix free space measurement for OSX 2016-07-05 11:29:04 +09:30
Jeremy Lakeman
89ab832c21 Link multiple versions of a journal payload to the same file 2016-05-24 14:47:49 +09:30
Jeremy Lakeman
177f695671 Rework handling of race conditions when storing the same bundle (#109) 2016-05-23 15:38:40 +09:30
Jeremy Lakeman
92af9769ec Vacuum sqlite db pages if 1/4 of the db file is free space 2016-05-18 10:40:10 +09:30
Jeremy Lakeman
c625a598d9 Use malloc / free for manifest structs 2016-05-09 11:10:50 +09:30
Jeremy Lakeman
dd8f98a2ab Ensure we rollback if commit fails 2016-04-18 16:06:27 +09:30
Jeremy Lakeman
5636edfe20 Append manifest into zip file comment 2016-03-30 16:25:43 +10:30
Jeremy Lakeman
f2e2b4acb5 Replace nacl & sha methods with versions from libsodium to avoid undefined behaviour 2016-01-27 16:27:04 +10:30
Andrew Bettison
bd45186a6a Document REST POST /restful/rhizome/insert
Adds definitions for Rhizome concepts such as Bundle ID, Bundle Secret,
payload, etc. and a detailed step-by-step definition of the insertion
logic.
2015-12-07 22:39:19 +10:30
Andrew Bettison
419364b5a9 Improve REST HTTP response status codes
List all the HTTP status codes in the REST API tech doc.

Only use 403 Forbidden for requests originating from a disallowed
origin (ie, not localhost).

- Return 400 for missing, unknown, duplicate and out-of-order form
  parts in POST requests.
- Return 415 Unsupported Media Type for unsupported form part
  Content-Disposition and Content-Type (including unsupported
  charset).
- Return 414 Request-URI Too Long for any buffer exhaustion while
  parsing request.
- Return 419 Authentication Timeout for missing crypto secret.
2015-11-02 12:26:40 +10:30
Andrew Bettison
27d98a29fa Reimplement strlcpy() using strncpy_nul() 2015-11-02 10:11:41 +10:30
Andrew Bettison
6123503c15 Improve Rhizome HTTP status codes
Also improve some reason phrases and test them
2015-09-21 18:43:24 +09:30
Andrew Bettison
7d9a5faa4e Move if(config.debug.xxx) tests into DEBUGF()
Original DEBUG() and DEBUGF() macros renamed to _DEBUG() and _DEBUGF()
New DEBUG() and DEBUGF() macros, first argument is flagname
New DEBUGF2(foo, bar, ...) macro does if(config.debug.foo||config.debug.bar) test
Replace almost all config.debug.xxx references to IF_DEBUG(xxx)
2015-07-13 16:00:05 +09:30
Andrew Bettison
7734e24006 /restful/rhizome/append on existing journal
Refactor Rhizome insert/append logic into functions used by both CLI and
RESTful API.  Improve RESTful diagnostic messages.
2015-03-30 18:24:12 +10:30
Andrew Bettison
705dbeff09 Fix bug in rhizome_write_open_journal()
Was treating PAYLOAD_STATUS_NEW as an error
2015-03-23 17:40:08 +10:30
Andrew Bettison
67a6e1b9b6 Add debug logging to rhizome_write_open_journal() 2015-03-16 22:41:52 +10:30
Andrew Bettison
c9131f43a2 POST /restful/rhizome/append for journals 2015-03-02 18:52:12 +10:30
Jeremy Lakeman
2c72875508 Ensure sql lock failures are returned. 2014-09-12 15:24:23 +09:30
Andrew Bettison
606f087dd5 Merge latest Rhizome Java API into development 2014-07-11 12:29:24 +09:30
Andrew Bettison
e35bf77938 Rhizome Java API: insert/update empty bundle 2014-07-10 22:06:46 +09:30
Andrew Bettison
cf43635789 Improve Rhizome HTTP RESTful interface
Add RHIZOME_BUNDLE_STATUS_READONLY enum value

Tighten up switch statements on bundle and payload status enums (no
default labels)

Rename some recently added enum entries

Return bundle status and payload status in HTTP responses

Add test for failing to decrypt a foreign encrypted bundle payload,
fix bug that caused an assertion failure

Add tests for fetching a non-existent manifest and fetching bundles
whose payload blob is not in the store
2014-07-10 06:29:51 +09:30
Jeremy Lakeman
63226143ac Don't log an error when MDP fetch fails due to missing bundle 2014-07-07 11:31:09 +09:30
Jeremy Lakeman
096c9670ca Remember when we last verified each payload 2014-07-04 14:25:28 +09:30
Jeremy Lakeman
588b95ab20 Ensure temporary id is unique 2014-06-28 17:36:06 +09:30
Jeremy Lakeman
5df5a33721 Measure free space on Android using statfs() 2014-06-26 12:17:36 +09:30
Jeremy Lakeman
2ef73884a1 Add configurable limit to ensure a minimum amount of free space is preserved
- default to 100MB of free space
- currently disabled on android due to missing sys/statvfs.h
2014-06-23 13:06:21 +09:30
Jeremy Lakeman
47f051917d Rhizome cleanup will now expire payloads and vacuum the db file. 2014-06-19 14:23:43 +09:30
Jeremy Lakeman
7ff89afcf4 Reinstate rhizome database storage limit
- old / large payloads should be evicted to fit more payloads
- if there isn't enough space, new payloads will not be added
2014-06-19 11:09:09 +09:30
Jeremy Lakeman
2a572d19bf Rework rhizome store to write content more lazily and cleanup more carefully 2014-06-04 17:13:08 +09:30
Jeremy Lakeman
dd9f5fb713 Only report that a payload exists if it actually exists 2014-04-07 13:22:27 +09:30
Andrew Bettison
76cce8fe7e Fix enum compiler warnings (clang -Wenum-conversion) 2014-04-03 10:36:38 +10:30
Andrew Bettison
53c1b1c04c Filesystem Heirarchy Standard (FHS) paths
If no instance directory specified, then use paths like
/etc/serval/serval.conf
/var/cache/serval
/var/log/serval
/var/run/serval
etc. for files, instead of all in a single directory.

Log all directory creation as INFO messages.

Interpretation of log.file.directory_path has changed slightly.

Updated servald configuration tech doc.
2014-03-26 15:35:43 +10:30
Jeremy Lakeman
ee68856878 Reduce log spam from debug.rhizome 2014-03-12 14:32:59 +10:30
Andrew Bettison
0bef066949 Store Rhizome external blob files in "blob" subdirectory
This will make issue #50 a bit easier to tackle
2014-03-07 16:55:38 +10:30
Jeremy Lakeman
974c87adf2 Remove orphan file rows if we notice the payload is missing 2014-03-01 05:45:32 +10:30
Jeremy Lakeman
1a543484bd Fix rhizome key generation for meshms 2014-01-22 15:51:06 +10:30
Jeremy Lakeman
cc51edd39d Fix compiler warnings 2014-01-06 12:36:00 +10:30
Andrew Bettison
ee9c96bb8c Implement HTTP POST /restful/rhizome/insert
Change HTTP request buffer pointers from (const char*) to (char*)
because some Rhizome operations can modify received data in-place, eg,
when decrypting it.
2013-12-30 18:09:56 +10:30
Andrew Bettison
464f0858f1 Refactor rhizome_write_file() to use fd not FILE* 2013-12-30 16:30:43 +10:30
Andrew Bettison
f81883758a Support streaming directly into Rhizome store
The rhizome_open_write() function now accepts RHIZOME_SIZE_UNSET as the
payload size, and streams into an external blob until finished, then
copies the external blob into the FILEBLOBS table if it is small enough.

Needed so that HTTP GET /restful/rhizome/insert can stream the payload
directly into the DB even if the supplied manifest does not specify
'filesize'.
2013-12-30 16:30:43 +10:30
Andrew Bettison
34188fa489 Replace config 'rhizome.external_blobs' with 'rhizome.max_blob_size' 2013-12-30 16:30:43 +10:30
Andrew Bettison
72040517e1 New enum rhizome_payload_status
Refactor a lot of Rhizome bundle storage code to use the new "enum
rhizome_payload_status" instead of mysterious int values to represent
the outcome of the operation.
2013-12-30 16:30:35 +10:30
Andrew Bettison
9ebef81a49 Formalise "rhizome add file" exit status
Formalise add-bundle result in "enum rhizome_bundle_status"

Rewrite rhizome_manifest_finalise(), rhizome_find_duplicate() and
rhizome_add_manifest() to return enum rhizome_bundle_status

New function rhizome_manifest_check_stored() that compares a manifest
with its stored counterpart and returns enum rhizome_bundle_status

Remove redundant rhizome_manifest_check_sanity(), consolidating all
manifest validation rules in rhizome_manifest_validate(), which now
checks the 'id' field is present, and that 'sender' and 'recipient' are
both present for MeshMS

Correct manifest finalisation logic: set the 'finalised' flag in
rhizome_manifest_validate(), not in rhizome_manifest_verify() (which
sets 'selfSigned'), and consistently clear 'finalised' flag in all
attribute setter functions

Remove manifest 'ttl' field and all references thereof (leaving unused
space in Rhizome BAR)

Rename some payload functions for clarity
2013-12-21 14:37:18 +10:30
Andrew Bettison
3b5c0d2092 More Rhizome external blob debug 2013-12-18 17:44:16 +10:30
Andrew Bettison
075f9c7c27 Use uint64_t for Rhizome manifest version
Instead of int64_t.  Fixes some -Wsign-compare warnings.

Replace sqlite_exec_int64() with sqlite_exec_uint64().

Also store rowid as uint64_t, and use 0 not -1 to indicate
unset.
2013-12-11 11:11:34 +10:30
Andrew Bettison
4428a76379 Use ssize_t for read(2) and write(2) return value
Test for errors using == -1 not < 0, and cast to (size_t) subsequently

Fixes some -Wsign-compare warnings
2013-12-10 17:21:35 +10:30
Andrew Bettison
6254709a84 Use RHIZOME_SIZE_UNSET instead of -1 for read_state.length
This may have been a reversion to older code caused by a careless
merge, because the change to RHIZOME_SIZE_UNSET was already made
when read_state.length was changed to unsigned.

Fixes some -Wsign-compare warnings
2013-12-10 17:17:18 +10:30
Andrew Bettison
4af6cf9d6a Fix -Wsign-compare warnings: use size_t or unsigned for byte counts 2013-12-10 16:52:53 +10:30