Commit Graph

140 Commits

Author SHA1 Message Date
Jeremy Lakeman
4aeb6545ad Refactor storage space measurement to expose new status API
Change semantics of fake space limit to exclude used space, otherwise the result will keep changing on each call
2018-06-26 16:41:27 +09:30
Andrew Bettison
faa0f392c5 Improve Rhizome DB path configuration
Change the default 'rhizome.datastore_path' to "rhizome", ie, a sub-
directory of the instance directory.

Add a new erename() operation to "os.h".

Clean up the code around the fix 39f582cca (two commits ago) for a bug
introduced in 2014: the 'rhizome.db' file was always being created in
the instance directory, regardless of the setting of the
'rhizome.datastore_path' config option.

Whenever the Rhizome database is opened (by the server or by a CLI
invocation), if the database file does not exist but there is a
'rhizome.db' file in the legacy location, then move that file (and any
accompanying "blob" and "hash" sub-directories) to the configured
location.

Update copyright notices.
2018-04-17 12:01:02 +09:30
Jeremy Lakeman
39f582ccab Auto move rhizome store when config is first set 2018-04-10 16:50:09 +09:30
Jeremy Lakeman
59b6a70374 Use config to set and remember rhizome database location 2018-04-10 10:52:44 +09:30
Jeremy Lakeman
57767e444d Keep rhizome's external blobs in subfolders, migrating each file on first use 2018-04-10 10:52:44 +09:30
Andrew Bettison
c35af0617a Fix header files for Xcode iOS build
- make "debug.h" a private header so that the DEBUG() macro does not
  interfere with Xcode Debug builds, which set DEBUG=1
- move all #include "debug.h" from headers into .c files
- move 'struct idebug' into a new public header "idebug.h" so that
  "log.h" can be public
- move HTTPD port number defs from "httpd.h" to "constants.h", so that
  "httpd.h" can be private and "conf.h" can be public
- add missing nested includes so each header compiles stand-alone
  without error
- #include "sodium.h" instead of <sodium.h>
- #include "sqlite3.h" instead of <sqlite3.h>
- add header guard to "fifo.h"
- fix header guard in "sync_keys.h"
2018-03-06 15:29:29 +10:30
Jeremy Lakeman
1c543f3c41 Ensure space calculation is performed as a uint64_t (Fixes #125) 2017-09-06 13:01:58 +09:30
Jeremy Lakeman
a473304c06 Add restful import API 2017-05-17 10:16:46 +09:30
Jeremy Lakeman
940b8c2d4b Ack incoming feeds for threading 2017-04-19 14:46:36 +09:30
Jeremy Lakeman
1e70c8fe53 Assert if attempting to pipe a journal bundle after deriving a crypto key 2017-03-28 14:36:50 +10:30
Jeremy Lakeman
77198dd163 Read earlier journal payload out of store 2017-03-08 13:37:26 +10:30
Jeremy Lakeman
585e573ecf Refactor rhizome storage methods to return database busy status, and pause rhizome sync 2017-03-06 16:12:55 +10:30
Jeremy Lakeman
c739555e8c Ensure we call rhizome_fail_write when the store was busy 2017-03-01 12:44:47 +10:30
Jeremy Lakeman
4c538a7686 Return payload busy if write fails to flush, treat as an error in most cases 2017-02-28 13:13:09 +10:30
Jeremy Lakeman
f60704b3e1 Tweak caching edge case of reading message ply's backwards 2017-02-14 14:56:40 +10:30
Andrew Bettison
1efe6686fc Use config.h instead of -DHAVE_FOO=1
swiftc(1) requires every -D option to be prefixed with -Xcc, which
is not easy to achieve using autoconf.

Now instead of setting configuration macros like HAVE_LSEEK64 on the
command line using -D, they are defined in config.h, which is generated
by ./configure, ignored by Git, and included by all headers and source
files that use any configuration macro.

The dependency on Makefile has been replaced with a dependency on
Makefile.in and config.h, which helps avoid some redundant full
recompiles after running ./configure, because the configure script
does not overwrit config.h if its content does not change.
2016-10-19 16:27:20 +10:30
Jeremy Lakeman
b927af79a8 Introduce new typedef for signing keys to expose their internal structure 2016-09-27 10:28:46 +09:30
Andrew Bettison
e34141a2a5 Upgrade to SQLite 3.14.2
Ignores a variety of warnings when compiling sqlite3.c.

Uses the new sqlite3_trace_v2() function for logging SQL statements, now
that sqlite3_trace() and sqlite3_profile() are deprecated.

Fixes a rhizomeops test failure caused because SQLite 3.12 increased the
default block size from 1024 to 4096, causing the fixed overhead of an
empty Rhizome db to exceed 32KB.  This caused the test to fail because
the first "file add" failed.  Now the test fixture adapts to the fixed
overhead.

Deletes the SQLite 3.10.2 source amalgamation.
2016-09-21 22:47:35 +09:30
Andrew Bettison
6e65a2399f Add missing #include "str.h" 2016-09-21 18:47:49 +09:30
Jeremy Lakeman
05da181541 Return PAYLOAD_STATUS_NEW if file is now missing from the store 2016-09-05 15:13:09 +09:30
Jeremy Lakeman
e13b9c3c94 Reduce redundant reading of message ply's 2016-08-23 14:28:13 +09:30
Jeremy Lakeman
ce36e6f2c6 Fix assert on DB lock while flushing data 2016-08-22 11:13:29 +09:30
Jeremy Lakeman
c5d6579397 Add cli option to include text of last received message 2016-08-10 11:20:12 +09:30
Jeremy Lakeman
94cf79f0f3 Improve resilience to database locking issues 2016-07-25 16:54:11 +09:30
Jeremy Lakeman
dbded493c1 Refactor meshms ply handling into separate source file 2016-07-25 16:54:11 +09:30
Jeremy Lakeman
c56b733391 Fix free space measurement for OSX 2016-07-05 11:29:04 +09:30
Jeremy Lakeman
89ab832c21 Link multiple versions of a journal payload to the same file 2016-05-24 14:47:49 +09:30
Jeremy Lakeman
177f695671 Rework handling of race conditions when storing the same bundle (#109) 2016-05-23 15:38:40 +09:30
Jeremy Lakeman
92af9769ec Vacuum sqlite db pages if 1/4 of the db file is free space 2016-05-18 10:40:10 +09:30
Jeremy Lakeman
c625a598d9 Use malloc / free for manifest structs 2016-05-09 11:10:50 +09:30
Jeremy Lakeman
dd8f98a2ab Ensure we rollback if commit fails 2016-04-18 16:06:27 +09:30
Jeremy Lakeman
5636edfe20 Append manifest into zip file comment 2016-03-30 16:25:43 +10:30
Jeremy Lakeman
f2e2b4acb5 Replace nacl & sha methods with versions from libsodium to avoid undefined behaviour 2016-01-27 16:27:04 +10:30
Andrew Bettison
bd45186a6a Document REST POST /restful/rhizome/insert
Adds definitions for Rhizome concepts such as Bundle ID, Bundle Secret,
payload, etc. and a detailed step-by-step definition of the insertion
logic.
2015-12-07 22:39:19 +10:30
Andrew Bettison
419364b5a9 Improve REST HTTP response status codes
List all the HTTP status codes in the REST API tech doc.

Only use 403 Forbidden for requests originating from a disallowed
origin (ie, not localhost).

- Return 400 for missing, unknown, duplicate and out-of-order form
  parts in POST requests.
- Return 415 Unsupported Media Type for unsupported form part
  Content-Disposition and Content-Type (including unsupported
  charset).
- Return 414 Request-URI Too Long for any buffer exhaustion while
  parsing request.
- Return 419 Authentication Timeout for missing crypto secret.
2015-11-02 12:26:40 +10:30
Andrew Bettison
27d98a29fa Reimplement strlcpy() using strncpy_nul() 2015-11-02 10:11:41 +10:30
Andrew Bettison
6123503c15 Improve Rhizome HTTP status codes
Also improve some reason phrases and test them
2015-09-21 18:43:24 +09:30
Andrew Bettison
7d9a5faa4e Move if(config.debug.xxx) tests into DEBUGF()
Original DEBUG() and DEBUGF() macros renamed to _DEBUG() and _DEBUGF()
New DEBUG() and DEBUGF() macros, first argument is flagname
New DEBUGF2(foo, bar, ...) macro does if(config.debug.foo||config.debug.bar) test
Replace almost all config.debug.xxx references to IF_DEBUG(xxx)
2015-07-13 16:00:05 +09:30
Andrew Bettison
7734e24006 /restful/rhizome/append on existing journal
Refactor Rhizome insert/append logic into functions used by both CLI and
RESTful API.  Improve RESTful diagnostic messages.
2015-03-30 18:24:12 +10:30
Andrew Bettison
705dbeff09 Fix bug in rhizome_write_open_journal()
Was treating PAYLOAD_STATUS_NEW as an error
2015-03-23 17:40:08 +10:30
Andrew Bettison
67a6e1b9b6 Add debug logging to rhizome_write_open_journal() 2015-03-16 22:41:52 +10:30
Andrew Bettison
c9131f43a2 POST /restful/rhizome/append for journals 2015-03-02 18:52:12 +10:30
Jeremy Lakeman
2c72875508 Ensure sql lock failures are returned. 2014-09-12 15:24:23 +09:30
Andrew Bettison
606f087dd5 Merge latest Rhizome Java API into development 2014-07-11 12:29:24 +09:30
Andrew Bettison
e35bf77938 Rhizome Java API: insert/update empty bundle 2014-07-10 22:06:46 +09:30
Andrew Bettison
cf43635789 Improve Rhizome HTTP RESTful interface
Add RHIZOME_BUNDLE_STATUS_READONLY enum value

Tighten up switch statements on bundle and payload status enums (no
default labels)

Rename some recently added enum entries

Return bundle status and payload status in HTTP responses

Add test for failing to decrypt a foreign encrypted bundle payload,
fix bug that caused an assertion failure

Add tests for fetching a non-existent manifest and fetching bundles
whose payload blob is not in the store
2014-07-10 06:29:51 +09:30
Jeremy Lakeman
63226143ac Don't log an error when MDP fetch fails due to missing bundle 2014-07-07 11:31:09 +09:30
Jeremy Lakeman
096c9670ca Remember when we last verified each payload 2014-07-04 14:25:28 +09:30
Jeremy Lakeman
588b95ab20 Ensure temporary id is unique 2014-06-28 17:36:06 +09:30
Jeremy Lakeman
5df5a33721 Measure free space on Android using statfs() 2014-06-26 12:17:36 +09:30