Commit Graph

215 Commits

Author SHA1 Message Date
Andrew Bettison
051eca4775 Improve Rhizome list cursor
Remove 'rowcount' element

Order by descending ROWID, which is functionally the same as descending
inserttime but more reliable

Replace '_offset' cursor element with rowid first-last range to record
the expanding window of rows already returned, which allows release and
re-open of cursor mid-listing without missing rows or producing duplicates
2013-11-07 17:22:06 +10:30
Andrew Bettison
9606b4b1b8 Refactor "rhizome list" command
Move output formatting into commandline.c, leave database query in
rhizome_database.c
2013-11-07 10:21:11 +10:30
Andrew Bettison
45442d3eb4 Rewrite bundle author authentication
Replaced 'int has_author' manifest element with new 'enum authorship'
element to record the result of author authentication, to avoid
repeating expensive crypto operations.

Separated the handling of bundle secret arguments from author lookup and
authentication.  The new rhizome_apply_bundle_secret(m,bsk) is now
called at the top level to set the manifest secret key (if it
validates), and thereafter there is no need to pass the 'bsk' argument
to any other functions, as they can simply check the 'haveSecret' field
of the manifest.

Removed rhizome_extract_privatekey() which combined author lookup and
bundle secret validation, and replaced it with functions that only deal
with the author: rhizome_lookup_author() and rhizome_authenticate_author().

Renamed other functions to make their purpose and effect clearer.

Formalised the semantics of only storing AUTHENTICATED author SIDs in
the 'author' column of the MANIFESTS table, which necessitated a change
to a 'rhizomeops' test case: when adding a file using a BK-less
manifest, the author column is set to null, so the Rhizome list output
does not show the bundle as ".fromhere" and does not give an author for
that bundle.
2013-11-06 23:58:17 +10:30
Andrew Bettison
3aa24f7407 Refactor "rhizome list" main loop
Preparing for re-use in HTTP /restful/rhizome/bundlelist.json
2013-11-04 23:47:09 +10:30
Andrew Bettison
bacba19dc1 Refactor manifest: specific setter functions
Replace generic rhizome_manifest_set() and rhizome_manifest_set_ll()
with per-field setter functions, eg, rhizome_manifest_set_filesize().
Struct rhizome_manifest elements for all known fields, to replace the
use of rhizome_manifest_get() and rhizome_manifest_get_ll() everywhere:
sender, recipient, service, name, date, bundle_key.

Add boolean validity flags for binary blob types, to avoid having to compare
with many bytes of all-zero to detect presence, eg, has_sender, has_recipient,
has_author, has_bundle_key.  These maintained by the setter functions.

Rename existing manifest struct elements to be the same as their field
names: fileLength -> filesize, journalTail -> tail.

More use of unsigned int, size_t and uint64_t for payload sizes, offsets, byte
counts, etc. especially in rhizome_store.c and meshms.c.  More uniform use of
size_t to dimension memory buffers.  Fix some printf(3) style format strings
for 64-bit correctness on 32-bit systems.  Use new constant RHIZOME_SIZE_UNSET
instead of -1 to indicate unknown dimension, and explicitly assert its absence
before comparisons and arithmetic, for safety.

Replace some 'int' loop variables with 'unsigned' where appropriate.

Fix bugs discovered in MeshMS bundle private/public key generation and
bundle secret key handling for export/extract commands.

Instrument the first MeshMS test case to aid debugging.

New debug config flag: debug.manifest logs all modifications to all manifest
fields by setter functions.

Rename debug config flag: debug.rhizome_bind -> debug.rhizome_sql_bind.
2013-11-04 19:17:49 +10:30
Andrew Bettison
c2b78f7b29 Refactor manifest: replace int with unsigned 2013-10-30 14:14:38 +10:30
Andrew Bettison
878ddb135d Fix sprintf(3) formats - change some %d to %u 2013-10-15 11:15:31 +10:30
Jeremy Lakeman
c7fa395968 Fix format patterns for 64-bit compilation 2013-10-14 14:45:43 +10:30
Andrew Bettison
97cbebc91e Issue #11: improve types in file i/o functions
Use open(2)/read(2)/write(2) instead of fopen(3)/fread(3)/fwrite(3) in
several places to avoid unnecessary buffering

Fix a bug in Rhizome HTTP add's handling of unconfigured manifest
template file

Improve some debug and error logging for file i/o
2013-10-11 16:24:51 +10:30
Andrew Bettison
974c7a56a0 Issue #11: Use rhizome_filehash_t everywhere 2013-10-11 15:35:27 +10:30
Andrew Bettison
c97bd4a69a Rewrite two major SQL queries using new bind syntax
Extend bind varargs syntax to handle optionally-NULL parameters.
2013-10-10 17:17:03 +10:30
Andrew Bettison
a95ef79139 Use sid_t everywhere
Remove stowSid() at last

Change API of tohex() and strbuf_tohex(), to pass string length instead of
binary byte count.  This allows odd numbers of hex digits to be produced.

Remove alloca_tohex_sid(); replace with alloca_tohex_sid_t()

New alloca_tohex_sid_t_trunc() macro
2013-10-09 18:54:21 +10:30
Jeremy Lakeman
60e3f5a3fc Fix printf formats for 64bit compilation 2013-10-09 10:31:52 +10:30
Andrew Bettison
ffafae86b1 Issue #11: replace long long with int64_t 2013-10-07 05:54:46 +10:30
Andrew Bettison
55edc74482 Fix intermittent 'rhizomeops' test failure 2013-10-07 04:47:14 +10:30
Andrew Bettison
7b427d48ee A few more uses of rhizome_bid_t
Now the TEXT_TOUPPER SQL bind action is only used for filehash hex
strings; all Bundle ID parameters are bound using RHIZOME_BID_T.
2013-10-04 04:50:37 +09:30
Andrew Bettison
e3f64516d5 Add config option 'debug.rhizome_bind' 2013-10-04 03:46:52 +09:30
Andrew Bettison
f5e7db089c Fix bugs revealed by test failures
Introduced by the recent change to rhizome_bid_t everywhere
2013-10-04 03:12:52 +09:30
Andrew Bettison
ab31420faf rhizome_bid_t
Consistent type for internal binary representation of Rhizome Bundle ID
(aka Manifest ID)
2013-10-03 23:16:45 +09:30
Andrew Bettison
d18e48868d Issue #69: transient text SQL binding 2013-10-03 23:16:02 +09:30
Andrew Bettison
41e18e587d Issue #69: replace sqlite3_bind_xxx() calls
Use the new sqlite_bind() and sqlite_prepare_bind() calls instead.

Add cmp_sid_t() function, use it instead of memcmp().

Use alloca_tohex_sid_t() in preference to alloca_tohex_sid() when the
argument is a sid_t.
2013-10-03 16:44:06 +09:30
Andrew Bettison
7e3a552011 Issue #69: refactor all Rhizome SQL queries to use binding
Remove all uses of sprintf(3) and its ilk to form SQL query commands.
Use the new sqlite_bind() and sqlite_prepare_bind() functions instead.
2013-10-03 15:15:30 +09:30
Andrew Bettison
8f7d600216 Issue #69: continue work on multi-bind varargs
Revise sqlite_prepare() macro et al and underlying functions to use bind
varargs instead of sprintf(3)-style varargs.

Convert all SQL function calls in rhizome_database.c
2013-10-03 01:16:10 +09:30
Andrew Bettison
772e1bf9d6 Issue #69: start writing SQLite multi-bind function 2013-10-02 18:19:20 +09:30
Andrew Bettison
162108dd18 Fix duplicate payload logic
rhizome_finish_write() was treating a duplicate file hash as an error,
instead of dealing with it sensibly
2013-09-30 16:32:08 +09:30
Andrew Bettison
01216f06c2 Refactor rhizome_cleanup()
Compute time horizons once only, not once per query

Factor query that deletes orphan FILEBLOBS rows into its own function
2013-09-30 16:20:50 +09:30
Andrew Bettison
99a0d36812 Improve Rhizome debugging
Log the integer result of an int64 query
2013-09-30 16:16:44 +09:30
Andrew Bettison
c91ea7f8ac Make some Rhizome db functions static 2013-09-30 16:15:59 +09:30
Andrew Bettison
a04fc263f9 Improve comments on Rhizome db functions 2013-09-30 16:15:01 +09:30
Jeremy Lakeman
504aa92711 Send a sync immediately on importing a new bundle 2013-09-20 14:41:50 +09:30
Jeremy Lakeman
b916363b40 Treat empty filehash the same as NULL 2013-08-28 16:15:28 +09:30
Jeremy Lakeman
44ab51e2ab Don't clean the database for every command line operation by default 2013-08-27 13:44:05 +09:30
Jeremy Lakeman
643b21d6a7 Fetch file payloads if they are missing, even if we already have the manifest 2013-08-22 15:14:21 +09:30
Jeremy Lakeman
f9b828c3dd Remove payload if hash doesn't match when reading back 2013-08-21 15:45:18 +09:30
Jeremy Lakeman
bf7d0d5b16 Cache rhizome read state when serving content via MDP 2013-08-16 14:57:28 +09:30
Jeremy Lakeman
68bf04f69d Change manifest service to MeshMS2 2013-08-01 11:37:35 +09:30
Jeremy Lakeman
f9bc9d1913 Add support for encrypted journals with tail>0 2013-07-22 15:04:26 +09:30
Jeremy Lakeman
3855ff35c4 Clean up string formatting for 64bit compilation 2013-07-15 09:59:24 +09:30
Jeremy Lakeman
8ac359e550 Use consistent integer types for 64bit compilation 2013-07-13 14:47:06 +09:30
Jeremy Lakeman
53f3920b90 Move cli global state into a structure and pass it around 2013-07-03 16:51:27 +09:30
Jeremy Lakeman
8249f65f8f Don't create unused import folder 2013-07-03 10:54:05 +09:30
Jeremy Lakeman
adbbed5275 Add new rhizome sync protocol 2013-06-19 12:05:57 +09:30
Jeremy Lakeman
a18230f591 Reduce latency impact of BAR lookups based on aggregate time 2013-06-14 14:09:58 +09:30
Jeremy Lakeman
b67516558e Add clean command to verify all bundles 2013-06-14 11:58:19 +09:30
Jeremy Lakeman
96c0889f9a Don't ask for manifests if we don't have room in transfer queues 2013-04-11 15:24:41 +09:30
Andrew Bettison
65f6e88e67 Add emkdirs() and emkdirsn() 2013-04-04 17:37:49 +10:30
Jeremy Lakeman
90386ce1b1 Merge remote-tracking branch 'origin/development' into serial
Conflicts:
	rhizome.h
	rhizome_database.c
	rhizome_fetch.c
	rhizome_store.c
2013-02-20 17:23:03 +10:30
Jeremy Lakeman
73786bcb5d Refactor all rhizome reading and writing
- The API in rhizome_store.c is used for all reading and writing
- external storage is now usable for all transport options
2013-02-20 16:18:59 +10:30
gardners
944486c048 simplify external/internal blob storage to a boolean flag instead
of a threshold.
2013-02-20 16:18:59 +10:30
gardners
9e9f93b079 make external blob output conditional on debug flag. 2013-02-20 16:18:58 +10:30