Commit Graph

318 Commits

Author SHA1 Message Date
Andrew Bettison
0bef066949 Store Rhizome external blob files in "blob" subdirectory
This will make issue  a bit easier to tackle
2014-03-07 16:55:38 +10:30
Jeremy Lakeman
b56f4c27d3 Replace internal usage of overlay_mdp_dispatch 2014-02-19 16:39:42 +10:30
Jeremy Lakeman
487df0408d Refactor mdp packet processing to avoid using struct overlay_mdp_frame 2014-02-19 16:34:11 +10:30
Andrew Bettison
a77b0702b1 Rename uuid_t --> serval_uuid_t
To avoid compile errors on OSX
2014-02-17 13:26:03 +10:30
Andrew Bettison
015ed0b181 Refactor to rename "Rhizome HTTP server" to "HTTPD server"
Move HTTPD start/stop/dispatch logic from rhizome_httpd.c into new
httpd.c

Rename config.debug.httpd -> config.debug.http_server
Rename config.debug.rhizome_httpd -> config.debug.httpd
Rename config.debug.rhizome_nohttptx -> config.debug.nohttptx
2014-02-03 15:56:07 +10:30
Andrew Bettison
94274ba5fa Refactor sqlite_retry_state into rhizome_list_cursor 2014-01-22 17:20:06 +10:30
Andrew Bettison
e3e3e1e046 rhizome_http.h 2014-01-22 15:51:59 +10:30
Jeremy Lakeman
5f2c6e364a Refactor interface addresses to use new socket_address struct 2014-01-09 15:01:38 +10:30
Jeremy Lakeman
64fe96ebfa Only drop BAR's from the sync if they are no longer interesting 2014-01-09 15:01:37 +10:30
Andrew Bettison
ee9c96bb8c Implement HTTP POST /restful/rhizome/insert
Change HTTP request buffer pointers from (const char*) to (char*)
because some Rhizome operations can modify received data in-place, eg,
when decrypting it.
2013-12-30 18:09:56 +10:30
Andrew Bettison
dd5048bd43 rhizome_manifest_set_name_from_path()
Refactor rhizome_fill_manifest() to use it
2013-12-30 16:30:43 +10:30
Andrew Bettison
7cecdf7a3a Add Rhizome db blob primitive functions
sqlite_blob_open_retry()
sqlite_blob_write_retry()
sqlite_blob_close()
2013-12-30 16:30:43 +10:30
Andrew Bettison
72040517e1 New enum rhizome_payload_status
Refactor a lot of Rhizome bundle storage code to use the new "enum
rhizome_payload_status" instead of mysterious int values to represent
the outcome of the operation.
2013-12-30 16:30:35 +10:30
Andrew Bettison
9ebef81a49 Formalise "rhizome add file" exit status
Formalise add-bundle result in "enum rhizome_bundle_status"

Rewrite rhizome_manifest_finalise(), rhizome_find_duplicate() and
rhizome_add_manifest() to return enum rhizome_bundle_status

New function rhizome_manifest_check_stored() that compares a manifest
with its stored counterpart and returns enum rhizome_bundle_status

Remove redundant rhizome_manifest_check_sanity(), consolidating all
manifest validation rules in rhizome_manifest_validate(), which now
checks the 'id' field is present, and that 'sender' and 'recipient' are
both present for MeshMS

Correct manifest finalisation logic: set the 'finalised' flag in
rhizome_manifest_validate(), not in rhizome_manifest_verify() (which
sets 'selfSigned'), and consistently clear 'finalised' flag in all
attribute setter functions

Remove manifest 'ttl' field and all references thereof (leaving unused
space in Rhizome BAR)

Rename some payload functions for clarity
2013-12-21 14:37:18 +10:30
Andrew Bettison
273c5f2689 Make rhizome_manifest_pack_variables() static 2013-12-20 11:29:26 +10:30
Andrew Bettison
d52ba4c871 Remove rhizome_read_manifest_file()
Replace with memcpy() followed by rhizome_manifest_parse()
for memory buffers

Replace with rhizome_read_manifest_from_file() for files
2013-12-20 11:29:17 +10:30
Andrew Bettison
250309f4d8 Fix strn_to_sid_t(), add strn_to_rhizome_bk_t()
Improve strn_to_rhizome_bid_t() and strn_to_rhizome_filehash_t() to
accept a NULL first arg -- useful for data validation
2013-12-18 17:44:16 +10:30
Andrew Bettison
6361bfd757 Implement HTTP GET /restful/rhizome/<BID>/raw.bin 2013-12-13 16:36:37 +10:30
Andrew Bettison
183cb46000 Add Rhizome HTTP response headers to /restful/rhizome/<BID>.rhm 2013-12-13 14:49:55 +10:30
Andrew Bettison
075f9c7c27 Use uint64_t for Rhizome manifest version
Instead of int64_t.  Fixes some -Wsign-compare warnings.

Replace sqlite_exec_int64() with sqlite_exec_uint64().

Also store rowid as uint64_t, and use 0 not -1 to indicate
unset.
2013-12-11 11:11:34 +10:30
Andrew Bettison
b7528412df Use UINT64_MAX instead of -1 for Rhizome HTTP response content_length
Fixes -Wsign-compare warnings
2013-12-10 17:06:45 +10:30
Andrew Bettison
4af6cf9d6a Fix -Wsign-compare warnings: use size_t or unsigned for byte counts 2013-12-10 16:52:53 +10:30
Andrew Bettison
30b2059608 Fix -Wunused-parameter warnings
New UNUSED(param) macro suppresses the warning for a specific parameter

In some cases, removed the unused parameters.  In others, used the
parameter in a DEBUGF() or assert() statement to document the intent.
2013-12-09 18:22:18 +10:30
Andrew Bettison
5226e47ef6 Uniform #ifdef __SERVAL_DNA__... for headers 2013-12-04 17:15:36 +10:30
Andrew Bettison
42ab9aec4c Merge branch 'development' into 'naf4'
Remove two redundant calls to rhizome_retrieve_manifest() in meshms.c,
revealed by an assert() in the stricter manifest parsing code

Fix header files included by socket.h
2013-12-02 17:17:47 +10:30
Andrew Bettison
8db5f9c14a Merge branch 'anyservice' into 'development'
Allows any valid "service" manifest field in the "rhizome add file"
command

Many improvements in Rhizome manifest parsing; stricter manifest syntax
rules (no comment or blank lines, field names must be alphanumeric
identifiers), faster preliminary manifest inspection when receiving
manifest advertisements or syncing manifests

The 'development' branch introduces "struct socket_address" which
coincidentally fixed the recently encountered Linux kernel 3.12
recvmsg(2) EINVAL problem, so that 'rhizomeprotocol' tests which fail on
the 'anyservice' branch will pass after this merge
2013-12-01 05:44:01 +10:30
Andrew Bettison
0dd1b302b5 Rewrite Rhizome manifest parsing
Move validation checks into new function rhizome_manifest_validate()

Remove rhizome_manifest 'errors' field

Replace rhizome_manifest 'warnings' with 'malformed'

Replace rhizome_manifest 'manifest_bytes' with 'manifest_body_bytes' and
refactor to use 'manifest_all_bytes' in all manifest i/o

Refactor rhizome_manifest_verify() and reverse sense of return value to
match rhizome_manifest_validate()

New function rhizome_manifest_inspect() -- lightweight manifest parser
used when receiving Rhizome advertisements

New 'rhizomeops' test case for invalid manifest "service" field values,
now passes
2013-12-01 05:13:34 +10:30
Jeremy Lakeman
99d5d9fa1f Refactor dataformats into separate header 2013-11-27 13:11:42 +10:30
Jeremy Lakeman
28fc1da2d2 Encapsulate rhizome fetch logging 2013-11-27 13:10:55 +10:30
Andrew Bettison
562e011847 Consolidate Rhizome string parsing functions
Remove unused rhizome_str[n]_is_...() functions in favour of
str[n]_to_rhizome_..._t() functions

Ensure that all str_to_..._t() functions accept a NULL 'dst' pointer
so they can be used easily to validate strings
2013-11-21 16:05:18 +10:30
Andrew Bettison
c2b9f75fcc Deprecate RHIZOME_MANIFEST_ID_... in favour of RHIZOME_BUNDLE_ID_... 2013-11-21 16:05:17 +10:30
Andrew Bettison
fce0893173 Implement HTTP /restful/rhizome/newsince/.../bundlelist.json
Now functions as per requirements, blocks for 60 seconds sending
new bundles as they appear, sends new token with each new bundle.
Tested manually.
2013-11-19 15:43:51 +10:30
Andrew Bettison
21b10b2cb7 Rhizome list cursor 'rowid_since' semantics
Setting 'rowid_since' now causes the cursor to iterate over bundles
in chronological order (oldest first) from that rowid upward.
2013-11-19 15:41:12 +10:30
Andrew Bettison
f8f7716544 Improve HTTP /restful/rhizome/bundlelist.json token
Now tokens are included in each row instead of a single token in the
enclosing JSON object.  Only puts tokens on the first row, and
thereafter all rows with a rowid higher than for the previous row that
had a token.
2013-11-18 15:10:28 +10:30
Andrew Bettison
4380fdcccd Token in /restful/rhizome/bundlelist.json output 2013-11-13 16:58:28 +10:30
Andrew Bettison
64db53a092 Add random UUID to Rhizome database 2013-11-13 12:44:31 +10:30
Andrew Bettison
13634f8748 Add ROWID field to struct rhizome_manifest
New ".rowid" output field from rhizome add, import, extract, export
operations.  (Also added missing ".inserttime" and "date" fields to
some operations.)

Use new "rhizome add file" .rowid output field to check output of of
/restful/rhizome/bundlelist.json
2013-11-11 18:18:08 +10:30
Andrew Bettison
d5b48f5a9e Improve HTTP server generated content logic
Content generator functions now take arguments describing the buffer
they are to fill, and respond with a struct containing the number of
bytes filled, and the number of free bytes needed before being called
again.

The HTTP response logic now fills the buffer as much as possible before
calling write(2) by topping it up instead of waiting for it to be
completely emptied before generating more content.
2013-11-09 11:31:21 +10:30
Andrew Bettison
a62b6f9250 Improve Rhizome list cursor
Separate function rhizome_list_commit() which, if not called, causes the
next list re-query to include the last row that was fetched with
rhizome_list_next(), rather than excluding it.
2013-11-08 18:29:49 +10:30
Andrew Bettison
1b906f3f11 Implement HTTP /restful/rhizome/bundlelist.json
Only tested for one bundle.
2013-11-07 23:40:56 +10:30
Andrew Bettison
051eca4775 Improve Rhizome list cursor
Remove 'rowcount' element

Order by descending ROWID, which is functionally the same as descending
inserttime but more reliable

Replace '_offset' cursor element with rowid first-last range to record
the expanding window of rows already returned, which allows release and
re-open of cursor mid-listing without missing rows or producing duplicates
2013-11-07 17:22:06 +10:30
Andrew Bettison
4f9cbeab20 Remove unused elements from struct rhizome_http_request
Also improve comments
2013-11-07 17:14:09 +10:30
Andrew Bettison
9606b4b1b8 Refactor "rhizome list" command
Move output formatting into commandline.c, leave database query in
rhizome_database.c
2013-11-07 10:21:11 +10:30
Andrew Bettison
e00c945fd1 Merge branch 'naf4' into 'development'
Big refactor of Rhizome author and bundle secret handling.
2013-11-07 00:42:42 +10:30
Andrew Bettison
45442d3eb4 Rewrite bundle author authentication
Replaced 'int has_author' manifest element with new 'enum authorship'
element to record the result of author authentication, to avoid
repeating expensive crypto operations.

Separated the handling of bundle secret arguments from author lookup and
authentication.  The new rhizome_apply_bundle_secret(m,bsk) is now
called at the top level to set the manifest secret key (if it
validates), and thereafter there is no need to pass the 'bsk' argument
to any other functions, as they can simply check the 'haveSecret' field
of the manifest.

Removed rhizome_extract_privatekey() which combined author lookup and
bundle secret validation, and replaced it with functions that only deal
with the author: rhizome_lookup_author() and rhizome_authenticate_author().

Renamed other functions to make their purpose and effect clearer.

Formalised the semantics of only storing AUTHENTICATED author SIDs in
the 'author' column of the MANIFESTS table, which necessitated a change
to a 'rhizomeops' test case: when adding a file using a BK-less
manifest, the author column is set to null, so the Rhizome list output
does not show the bundle as ".fromhere" and does not give an author for
that bundle.
2013-11-06 23:58:17 +10:30
Jeremy Lakeman
b3ad074666 Fix compilation for 64-bit systems 2013-11-06 14:12:24 +10:30
Andrew Bettison
3aa24f7407 Refactor "rhizome list" main loop
Preparing for re-use in HTTP /restful/rhizome/bundlelist.json
2013-11-04 23:47:09 +10:30
Andrew Bettison
9625ba9734 Merge branch 'refactor-manifest' into 'development' 2013-11-04 19:41:00 +10:30
Andrew Bettison
bacba19dc1 Refactor manifest: specific setter functions
Replace generic rhizome_manifest_set() and rhizome_manifest_set_ll()
with per-field setter functions, eg, rhizome_manifest_set_filesize().
Struct rhizome_manifest elements for all known fields, to replace the
use of rhizome_manifest_get() and rhizome_manifest_get_ll() everywhere:
sender, recipient, service, name, date, bundle_key.

Add boolean validity flags for binary blob types, to avoid having to compare
with many bytes of all-zero to detect presence, eg, has_sender, has_recipient,
has_author, has_bundle_key.  These maintained by the setter functions.

Rename existing manifest struct elements to be the same as their field
names: fileLength -> filesize, journalTail -> tail.

More use of unsigned int, size_t and uint64_t for payload sizes, offsets, byte
counts, etc. especially in rhizome_store.c and meshms.c.  More uniform use of
size_t to dimension memory buffers.  Fix some printf(3) style format strings
for 64-bit correctness on 32-bit systems.  Use new constant RHIZOME_SIZE_UNSET
instead of -1 to indicate unknown dimension, and explicitly assert its absence
before comparisons and arithmetic, for safety.

Replace some 'int' loop variables with 'unsigned' where appropriate.

Fix bugs discovered in MeshMS bundle private/public key generation and
bundle secret key handling for export/extract commands.

Instrument the first MeshMS test case to aid debugging.

New debug config flag: debug.manifest logs all modifications to all manifest
fields by setter functions.

Rename debug config flag: debug.rhizome_bind -> debug.rhizome_sql_bind.
2013-11-04 19:17:49 +10:30
Jeremy Lakeman
5e756bec1e Fix types for 64-bit compilation 2013-10-30 17:23:44 +10:30
Andrew Bettison
c2b78f7b29 Refactor manifest: replace int with unsigned 2013-10-30 14:14:38 +10:30
Andrew Bettison
416b82b6c0 Refactor manifest: change haveSecret to enum 2013-10-30 13:14:26 +10:30
Andrew Bettison
0397a47753 Fix some test failures in new HTTP server code 2013-10-26 20:28:47 +10:30
Andrew Bettison
48921802f5 Use size_t in Rhizome store functions 2013-10-25 00:20:53 +10:30
Andrew Bettison
fa21bec880 Rewrite HTTP server 2013-10-25 00:19:37 +10:30
Andrew Bettison
00cf61721d Rename http_header_complete() to is_http_header_complete() 2013-10-16 11:22:02 +10:30
Andrew Bettison
97cbebc91e Issue : improve types in file i/o functions
Use open(2)/read(2)/write(2) instead of fopen(3)/fread(3)/fwrite(3) in
several places to avoid unnecessary buffering

Fix a bug in Rhizome HTTP add's handling of unconfigured manifest
template file

Improve some debug and error logging for file i/o
2013-10-11 16:24:51 +10:30
Andrew Bettison
974c7a56a0 Issue : Use rhizome_filehash_t everywhere 2013-10-11 15:35:27 +10:30
Andrew Bettison
c97bd4a69a Rewrite two major SQL queries using new bind syntax
Extend bind varargs syntax to handle optionally-NULL parameters.
2013-10-10 17:17:03 +10:30
Andrew Bettison
a95ef79139 Use sid_t everywhere
Remove stowSid() at last

Change API of tohex() and strbuf_tohex(), to pass string length instead of
binary byte count.  This allows odd numbers of hex digits to be produced.

Remove alloca_tohex_sid(); replace with alloca_tohex_sid_t()

New alloca_tohex_sid_t_trunc() macro
2013-10-09 18:54:21 +10:30
Andrew Bettison
7a8d750a0e Issue : uint16_t instead of short for TCP port number 2013-10-07 23:37:37 +10:30
Andrew Bettison
55edc74482 Fix intermittent 'rhizomeops' test failure 2013-10-07 04:47:14 +10:30
Andrew Bettison
7b427d48ee A few more uses of rhizome_bid_t
Now the TEXT_TOUPPER SQL bind action is only used for filehash hex
strings; all Bundle ID parameters are bound using RHIZOME_BID_T.
2013-10-04 04:50:37 +09:30
Andrew Bettison
ab31420faf rhizome_bid_t
Consistent type for internal binary representation of Rhizome Bundle ID
(aka Manifest ID)
2013-10-03 23:16:45 +09:30
Andrew Bettison
d18e48868d Issue : transient text SQL binding 2013-10-03 23:16:02 +09:30
Andrew Bettison
41e18e587d Issue : replace sqlite3_bind_xxx() calls
Use the new sqlite_bind() and sqlite_prepare_bind() calls instead.

Add cmp_sid_t() function, use it instead of memcmp().

Use alloca_tohex_sid_t() in preference to alloca_tohex_sid() when the
argument is a sid_t.
2013-10-03 16:44:06 +09:30
Andrew Bettison
7e3a552011 Issue : refactor all Rhizome SQL queries to use binding
Remove all uses of sprintf(3) and its ilk to form SQL query commands.
Use the new sqlite_bind() and sqlite_prepare_bind() functions instead.
2013-10-03 15:15:30 +09:30
Andrew Bettison
8f7d600216 Issue : continue work on multi-bind varargs
Revise sqlite_prepare() macro et al and underlying functions to use bind
varargs instead of sprintf(3)-style varargs.

Convert all SQL function calls in rhizome_database.c
2013-10-03 01:16:10 +09:30
Andrew Bettison
772e1bf9d6 Issue : start writing SQLite multi-bind function 2013-10-02 18:19:20 +09:30
Andrew Bettison
06d7620314 Add "rhizome add file" --force-new option
Overrides the default behaviour of de-duplicating added manifests

New 'rhizomeops' test case: AddForceDuplicate
2013-09-30 16:48:18 +09:30
Andrew Bettison
162108dd18 Fix duplicate payload logic
rhizome_finish_write() was treating a duplicate file hash as an error,
instead of dealing with it sensibly
2013-09-30 16:32:08 +09:30
Jeremy Lakeman
931ca60554 Refactor and simplify http server
- Define a function per page that is responsible for parsing requests
- Define a "generator" callback function for filling the buffer for a response
- Remove features that have never been used
- Remove functions that are no longer used
2013-09-24 14:34:50 +09:30
Jeremy Lakeman
38fe34e6e5 Bump rhizome idle timeout for congested links 2013-09-20 14:41:50 +09:30
Jeremy Lakeman
2a5ba97b48 Add rhizome test over fakeradio 2013-09-13 14:02:35 +09:30
Jeremy Lakeman
8accabfcac Always store payloads with a temporary id 2013-08-27 16:15:51 +09:30
Jeremy Lakeman
643b21d6a7 Fetch file payloads if they are missing, even if we already have the manifest 2013-08-22 15:14:21 +09:30
Jeremy Lakeman
f9b828c3dd Remove payload if hash doesn't match when reading back 2013-08-21 15:45:18 +09:30
Jeremy Lakeman
1468d2deb0 Pre-emptively send small payloads via MDP 2013-08-21 09:21:40 +09:30
Jeremy Lakeman
bf7d0d5b16 Cache rhizome read state when serving content via MDP 2013-08-16 14:57:28 +09:30
Jeremy Lakeman
771cb4151b Fetch journal range via HTTP 2013-08-15 17:00:43 +09:30
Jeremy Lakeman
21132466f1 Serve a range of a file payload via HTTP 2013-08-15 16:31:26 +09:30
Jeremy Lakeman
e322114850 Only transfer new journal content via MDP 2013-08-15 14:36:56 +09:30
Jeremy Lakeman
68bf04f69d Change manifest service to MeshMS2 2013-08-01 11:37:35 +09:30
Jeremy Lakeman
0c237db66a Add API to create bundles with deterministic ID's 2013-07-25 14:46:34 +09:30
Jeremy Lakeman
57fd0020c7 Add API for bufferred reading of rhizome bundles 2013-07-25 14:42:30 +09:30
Jeremy Lakeman
f9bc9d1913 Add support for encrypted journals with tail>0 2013-07-22 15:04:26 +09:30
Jeremy Lakeman
15ad333195 Implicitly buffer all writes to database blobs 2013-07-18 17:04:12 +09:30
Jeremy Lakeman
7d1714d398 Refactor database writes to reduce open/close calls 2013-07-18 13:52:42 +09:30
Jeremy Lakeman
a4bf4edbfc Write out of order packets with MDP transport
- add random order writing to rhizome store API
- remove rhizome flush, force all callers to provide buffer
2013-07-17 16:57:59 +09:30
Jeremy Lakeman
8ac359e550 Use consistent integer types for 64bit compilation 2013-07-13 14:47:06 +09:30
Jeremy Lakeman
a1ac7febe2 Add rhizome fetch status html page 2013-07-11 12:32:06 +09:30
Jeremy Lakeman
62a4cd407d Identify a journal bundle by the presense of a tail field 2013-07-08 14:57:47 +09:30
Jeremy Lakeman
e758e0130f Create journal append command 2013-07-08 14:29:08 +09:30
Jeremy Lakeman
53f3920b90 Move cli global state into a structure and pass it around 2013-07-03 16:51:27 +09:30
Jeremy Lakeman
8249f65f8f Don't create unused import folder 2013-07-03 10:54:05 +09:30
Jeremy Lakeman
adbbed5275 Add new rhizome sync protocol 2013-06-19 12:05:57 +09:30
Jeremy Lakeman
b67516558e Add clean command to verify all bundles 2013-06-14 11:58:19 +09:30
Jeremy Lakeman
96c0889f9a Don't ask for manifests if we don't have room in transfer queues 2013-04-11 15:24:41 +09:30
Andrew Bettison
6eb08ae805 Add new "config dump" command - not working yet
Add cf_fmt_xxx() functions for converting configuration structs into a
config object model tree.
2013-02-26 18:09:08 +10:30
Jeremy Lakeman
90386ce1b1 Merge remote-tracking branch 'origin/development' into serial
Conflicts:
	rhizome.h
	rhizome_database.c
	rhizome_fetch.c
	rhizome_store.c
2013-02-20 17:23:03 +10:30