Commit Graph

276 Commits

Author SHA1 Message Date
Jeremy Lakeman
47f051917d Rhizome cleanup will now expire payloads and vacuum the db file. 2014-06-19 14:23:43 +09:30
Jeremy Lakeman
7ff89afcf4 Reinstate rhizome database storage limit
- old / large payloads should be evicted to fit more payloads
- if there isn't enough space, new payloads will not be added
2014-06-19 11:09:09 +09:30
Jeremy Lakeman
80e139c2d9 Ensure orphan payloads or manifests are cleaned up 2014-06-16 15:56:31 +09:30
Jeremy Lakeman
f63e7fee19 Introduce typedef for handling BAR values 2014-06-16 14:38:14 +09:30
Jeremy Lakeman
2a572d19bf Rework rhizome store to write content more lazily and cleanup more carefully 2014-06-04 17:13:08 +09:30
Jeremy Lakeman
b6de3fcd34 Only send rhizome sync packets when we have neighbours 2014-05-30 15:03:22 +09:30
Andrew Bettison
21f51965c1 Overhaul Makefile and header files
Make .o files in separate subdirectories for servald and library
Factor struct __sourceloc and __WHENCE__ from "log.h" into "whence.h"
Factor sid_t etc from "serval.h" into "serval_types.h"
Factor rhizome_bid_t etc from "rhizome.h" into "rhizome_types.h"
Do not include "serval.h" in library sources
Add log_stderr.c and logMessage.c for stand-alone executables
2014-05-26 15:36:26 +09:30
Andrew Bettison
53c1b1c04c Filesystem Heirarchy Standard (FHS) paths
If no instance directory specified, then use paths like
/etc/serval/serval.conf
/var/cache/serval
/var/log/serval
/var/run/serval
etc. for files, instead of all in a single directory.

Log all directory creation as INFO messages.

Interpretation of log.file.directory_path has changed slightly.

Updated servald configuration tech doc.
2014-03-26 15:35:43 +10:30
Andrew Bettison
0bef066949 Store Rhizome external blob files in "blob" subdirectory
This will make issue #50 a bit easier to tackle
2014-03-07 16:55:38 +10:30
Jeremy Lakeman
b56f4c27d3 Replace internal usage of overlay_mdp_dispatch 2014-02-19 16:39:42 +10:30
Jeremy Lakeman
487df0408d Refactor mdp packet processing to avoid using struct overlay_mdp_frame 2014-02-19 16:34:11 +10:30
Andrew Bettison
a77b0702b1 Rename uuid_t --> serval_uuid_t
To avoid compile errors on OSX
2014-02-17 13:26:03 +10:30
Andrew Bettison
015ed0b181 Refactor to rename "Rhizome HTTP server" to "HTTPD server"
Move HTTPD start/stop/dispatch logic from rhizome_httpd.c into new
httpd.c

Rename config.debug.httpd -> config.debug.http_server
Rename config.debug.rhizome_httpd -> config.debug.httpd
Rename config.debug.rhizome_nohttptx -> config.debug.nohttptx
2014-02-03 15:56:07 +10:30
Andrew Bettison
94274ba5fa Refactor sqlite_retry_state into rhizome_list_cursor 2014-01-22 17:20:06 +10:30
Andrew Bettison
e3e3e1e046 rhizome_http.h 2014-01-22 15:51:59 +10:30
Jeremy Lakeman
5f2c6e364a Refactor interface addresses to use new socket_address struct 2014-01-09 15:01:38 +10:30
Jeremy Lakeman
64fe96ebfa Only drop BAR's from the sync if they are no longer interesting 2014-01-09 15:01:37 +10:30
Andrew Bettison
ee9c96bb8c Implement HTTP POST /restful/rhizome/insert
Change HTTP request buffer pointers from (const char*) to (char*)
because some Rhizome operations can modify received data in-place, eg,
when decrypting it.
2013-12-30 18:09:56 +10:30
Andrew Bettison
dd5048bd43 rhizome_manifest_set_name_from_path()
Refactor rhizome_fill_manifest() to use it
2013-12-30 16:30:43 +10:30
Andrew Bettison
7cecdf7a3a Add Rhizome db blob primitive functions
sqlite_blob_open_retry()
sqlite_blob_write_retry()
sqlite_blob_close()
2013-12-30 16:30:43 +10:30
Andrew Bettison
72040517e1 New enum rhizome_payload_status
Refactor a lot of Rhizome bundle storage code to use the new "enum
rhizome_payload_status" instead of mysterious int values to represent
the outcome of the operation.
2013-12-30 16:30:35 +10:30
Andrew Bettison
9ebef81a49 Formalise "rhizome add file" exit status
Formalise add-bundle result in "enum rhizome_bundle_status"

Rewrite rhizome_manifest_finalise(), rhizome_find_duplicate() and
rhizome_add_manifest() to return enum rhizome_bundle_status

New function rhizome_manifest_check_stored() that compares a manifest
with its stored counterpart and returns enum rhizome_bundle_status

Remove redundant rhizome_manifest_check_sanity(), consolidating all
manifest validation rules in rhizome_manifest_validate(), which now
checks the 'id' field is present, and that 'sender' and 'recipient' are
both present for MeshMS

Correct manifest finalisation logic: set the 'finalised' flag in
rhizome_manifest_validate(), not in rhizome_manifest_verify() (which
sets 'selfSigned'), and consistently clear 'finalised' flag in all
attribute setter functions

Remove manifest 'ttl' field and all references thereof (leaving unused
space in Rhizome BAR)

Rename some payload functions for clarity
2013-12-21 14:37:18 +10:30
Andrew Bettison
273c5f2689 Make rhizome_manifest_pack_variables() static 2013-12-20 11:29:26 +10:30
Andrew Bettison
d52ba4c871 Remove rhizome_read_manifest_file()
Replace with memcpy() followed by rhizome_manifest_parse()
for memory buffers

Replace with rhizome_read_manifest_from_file() for files
2013-12-20 11:29:17 +10:30
Andrew Bettison
250309f4d8 Fix strn_to_sid_t(), add strn_to_rhizome_bk_t()
Improve strn_to_rhizome_bid_t() and strn_to_rhizome_filehash_t() to
accept a NULL first arg -- useful for data validation
2013-12-18 17:44:16 +10:30
Andrew Bettison
6361bfd757 Implement HTTP GET /restful/rhizome/<BID>/raw.bin 2013-12-13 16:36:37 +10:30
Andrew Bettison
183cb46000 Add Rhizome HTTP response headers to /restful/rhizome/<BID>.rhm 2013-12-13 14:49:55 +10:30
Andrew Bettison
075f9c7c27 Use uint64_t for Rhizome manifest version
Instead of int64_t.  Fixes some -Wsign-compare warnings.

Replace sqlite_exec_int64() with sqlite_exec_uint64().

Also store rowid as uint64_t, and use 0 not -1 to indicate
unset.
2013-12-11 11:11:34 +10:30
Andrew Bettison
b7528412df Use UINT64_MAX instead of -1 for Rhizome HTTP response content_length
Fixes -Wsign-compare warnings
2013-12-10 17:06:45 +10:30
Andrew Bettison
4af6cf9d6a Fix -Wsign-compare warnings: use size_t or unsigned for byte counts 2013-12-10 16:52:53 +10:30
Andrew Bettison
30b2059608 Fix -Wunused-parameter warnings
New UNUSED(param) macro suppresses the warning for a specific parameter

In some cases, removed the unused parameters.  In others, used the
parameter in a DEBUGF() or assert() statement to document the intent.
2013-12-09 18:22:18 +10:30
Andrew Bettison
5226e47ef6 Uniform #ifdef __SERVAL_DNA__... for headers 2013-12-04 17:15:36 +10:30
Andrew Bettison
42ab9aec4c Merge branch 'development' into 'naf4'
Remove two redundant calls to rhizome_retrieve_manifest() in meshms.c,
revealed by an assert() in the stricter manifest parsing code

Fix header files included by socket.h
2013-12-02 17:17:47 +10:30
Andrew Bettison
8db5f9c14a Merge branch 'anyservice' into 'development'
Allows any valid "service" manifest field in the "rhizome add file"
command

Many improvements in Rhizome manifest parsing; stricter manifest syntax
rules (no comment or blank lines, field names must be alphanumeric
identifiers), faster preliminary manifest inspection when receiving
manifest advertisements or syncing manifests

The 'development' branch introduces "struct socket_address" which
coincidentally fixed the recently encountered Linux kernel 3.12
recvmsg(2) EINVAL problem, so that 'rhizomeprotocol' tests which fail on
the 'anyservice' branch will pass after this merge
2013-12-01 05:44:01 +10:30
Andrew Bettison
0dd1b302b5 Rewrite Rhizome manifest parsing
Move validation checks into new function rhizome_manifest_validate()

Remove rhizome_manifest 'errors' field

Replace rhizome_manifest 'warnings' with 'malformed'

Replace rhizome_manifest 'manifest_bytes' with 'manifest_body_bytes' and
refactor to use 'manifest_all_bytes' in all manifest i/o

Refactor rhizome_manifest_verify() and reverse sense of return value to
match rhizome_manifest_validate()

New function rhizome_manifest_inspect() -- lightweight manifest parser
used when receiving Rhizome advertisements

New 'rhizomeops' test case for invalid manifest "service" field values,
now passes
2013-12-01 05:13:34 +10:30
Jeremy Lakeman
99d5d9fa1f Refactor dataformats into separate header 2013-11-27 13:11:42 +10:30
Jeremy Lakeman
28fc1da2d2 Encapsulate rhizome fetch logging 2013-11-27 13:10:55 +10:30
Andrew Bettison
562e011847 Consolidate Rhizome string parsing functions
Remove unused rhizome_str[n]_is_...() functions in favour of
str[n]_to_rhizome_..._t() functions

Ensure that all str_to_..._t() functions accept a NULL 'dst' pointer
so they can be used easily to validate strings
2013-11-21 16:05:18 +10:30
Andrew Bettison
c2b9f75fcc Deprecate RHIZOME_MANIFEST_ID_... in favour of RHIZOME_BUNDLE_ID_... 2013-11-21 16:05:17 +10:30
Andrew Bettison
fce0893173 Implement HTTP /restful/rhizome/newsince/.../bundlelist.json
Now functions as per requirements, blocks for 60 seconds sending
new bundles as they appear, sends new token with each new bundle.
Tested manually.
2013-11-19 15:43:51 +10:30
Andrew Bettison
21b10b2cb7 Rhizome list cursor 'rowid_since' semantics
Setting 'rowid_since' now causes the cursor to iterate over bundles
in chronological order (oldest first) from that rowid upward.
2013-11-19 15:41:12 +10:30
Andrew Bettison
f8f7716544 Improve HTTP /restful/rhizome/bundlelist.json token
Now tokens are included in each row instead of a single token in the
enclosing JSON object.  Only puts tokens on the first row, and
thereafter all rows with a rowid higher than for the previous row that
had a token.
2013-11-18 15:10:28 +10:30
Andrew Bettison
4380fdcccd Token in /restful/rhizome/bundlelist.json output 2013-11-13 16:58:28 +10:30
Andrew Bettison
64db53a092 Add random UUID to Rhizome database 2013-11-13 12:44:31 +10:30
Andrew Bettison
13634f8748 Add ROWID field to struct rhizome_manifest
New ".rowid" output field from rhizome add, import, extract, export
operations.  (Also added missing ".inserttime" and "date" fields to
some operations.)

Use new "rhizome add file" .rowid output field to check output of of
/restful/rhizome/bundlelist.json
2013-11-11 18:18:08 +10:30
Andrew Bettison
d5b48f5a9e Improve HTTP server generated content logic
Content generator functions now take arguments describing the buffer
they are to fill, and respond with a struct containing the number of
bytes filled, and the number of free bytes needed before being called
again.

The HTTP response logic now fills the buffer as much as possible before
calling write(2) by topping it up instead of waiting for it to be
completely emptied before generating more content.
2013-11-09 11:31:21 +10:30
Andrew Bettison
a62b6f9250 Improve Rhizome list cursor
Separate function rhizome_list_commit() which, if not called, causes the
next list re-query to include the last row that was fetched with
rhizome_list_next(), rather than excluding it.
2013-11-08 18:29:49 +10:30
Andrew Bettison
1b906f3f11 Implement HTTP /restful/rhizome/bundlelist.json
Only tested for one bundle.
2013-11-07 23:40:56 +10:30
Andrew Bettison
051eca4775 Improve Rhizome list cursor
Remove 'rowcount' element

Order by descending ROWID, which is functionally the same as descending
inserttime but more reliable

Replace '_offset' cursor element with rowid first-last range to record
the expanding window of rows already returned, which allows release and
re-open of cursor mid-listing without missing rows or producing duplicates
2013-11-07 17:22:06 +10:30
Andrew Bettison
4f9cbeab20 Remove unused elements from struct rhizome_http_request
Also improve comments
2013-11-07 17:14:09 +10:30