Commit Graph

2684 Commits

Author SHA1 Message Date
Andrew Bettison
d5b48f5a9e Improve HTTP server generated content logic
Content generator functions now take arguments describing the buffer
they are to fill, and respond with a struct containing the number of
bytes filled, and the number of free bytes needed before being called
again.

The HTTP response logic now fills the buffer as much as possible before
calling write(2) by topping it up instead of waiting for it to be
completely emptied before generating more content.
2013-11-09 11:31:21 +10:30
Andrew Bettison
a62b6f9250 Improve Rhizome list cursor
Separate function rhizome_list_commit() which, if not called, causes the
next list re-query to include the last row that was fetched with
rhizome_list_next(), rather than excluding it.
2013-11-08 18:29:49 +10:30
Andrew Bettison
32ce7f5ed9 Do not catch SIGQUIT
So that it can be used to make a core dump during manual debugging
2013-11-08 18:25:33 +10:30
Andrew Bettison
1b906f3f11 Implement HTTP /restful/rhizome/bundlelist.json
Only tested for one bundle.
2013-11-07 23:40:56 +10:30
Andrew Bettison
9ecf9891fb Add strbuf JSON helper functions 2013-11-07 23:39:53 +10:30
Andrew Bettison
d337542067 Improve HTTP server generated content logic
Support generated content with an unspecified Content-Length.  Generator
functions return 1 if there is more content to come, and 0 if they have
just produced the last piece of content.
2013-11-07 23:39:24 +10:30
Andrew Bettison
051eca4775 Improve Rhizome list cursor
Remove 'rowcount' element

Order by descending ROWID, which is functionally the same as descending
inserttime but more reliable

Replace '_offset' cursor element with rowid first-last range to record
the expanding window of rows already returned, which allows release and
re-open of cursor mid-listing without missing rows or producing duplicates
2013-11-07 17:22:06 +10:30
Andrew Bettison
47988bfe0f Make Content-Length optional in HTTP responses
To eventually support the /restful/rhizome/bundlelist.json query which
will not compute in advance the length of its response.
2013-11-07 17:18:38 +10:30
Andrew Bettison
4f9cbeab20 Remove unused elements from struct rhizome_http_request
Also improve comments
2013-11-07 17:14:09 +10:30
Andrew Bettison
9606b4b1b8 Refactor "rhizome list" command
Move output formatting into commandline.c, leave database query in
rhizome_database.c
2013-11-07 10:21:11 +10:30
Andrew Bettison
e00c945fd1 Merge branch 'naf4' into 'development'
Big refactor of Rhizome author and bundle secret handling.
2013-11-07 00:42:42 +10:30
Andrew Bettison
45442d3eb4 Rewrite bundle author authentication
Replaced 'int has_author' manifest element with new 'enum authorship'
element to record the result of author authentication, to avoid
repeating expensive crypto operations.

Separated the handling of bundle secret arguments from author lookup and
authentication.  The new rhizome_apply_bundle_secret(m,bsk) is now
called at the top level to set the manifest secret key (if it
validates), and thereafter there is no need to pass the 'bsk' argument
to any other functions, as they can simply check the 'haveSecret' field
of the manifest.

Removed rhizome_extract_privatekey() which combined author lookup and
bundle secret validation, and replaced it with functions that only deal
with the author: rhizome_lookup_author() and rhizome_authenticate_author().

Renamed other functions to make their purpose and effect clearer.

Formalised the semantics of only storing AUTHENTICATED author SIDs in
the 'author' column of the MANIFESTS table, which necessitated a change
to a 'rhizomeops' test case: when adding a file using a BK-less
manifest, the author column is set to null, so the Rhizome list output
does not show the bundle as ".fromhere" and does not give an author for
that bundle.
2013-11-06 23:58:17 +10:30
Andrew Bettison
4aea05f445 Fix prototype of bcopy()
Add const to void * src argument
2013-11-06 23:56:33 +10:30
Andrew Bettison
a90eced640 Fix test framework: bug in assertStdoutIs() et al
Could overwrite the latest stdout or stderr file produced by execute()
2013-11-06 23:56:33 +10:30
Jeremy Lakeman
7629f24d3f Free subscriber entries after all commands 2013-11-06 15:36:21 +10:30
Jeremy Lakeman
b84925f4ed Free the keyring once it's no longer needed 2013-11-06 15:35:44 +10:30
Dan Staples
f1d817463e fixed keyring-related memory leaks 2013-11-06 14:30:35 +10:30
Jeremy Lakeman
b3ad074666 Fix compilation for 64-bit systems 2013-11-06 14:12:24 +10:30
Andrew Bettison
3aa24f7407 Refactor "rhizome list" main loop
Preparing for re-use in HTTP /restful/rhizome/bundlelist.json
2013-11-04 23:47:09 +10:30
Andrew Bettison
016fbe0244 Merge branch 'development' into 'naf4' 2013-11-04 20:01:58 +10:30
Andrew Bettison
9625ba9734 Merge branch 'refactor-manifest' into 'development' 2013-11-04 19:41:00 +10:30
Andrew Bettison
e61466b815 Fix printf format string for 32-bit systems
The C standard defines the difference between two (char*) pointers is
defined to be either short, int or long.  Not long long (64 bits).  On
64-bit systems, logging "%"PRIhttp_size_t (int64_t) for pointer
difference works by coincidence because a long or int is 64 bits, but
causes SEGV on 32 bit platforms.

The portable solution is to log pointer differences using "%d" and
explicitly cast the pointer difference to (int) in the arg list, or
"%ld" and cast it to (long).
2013-11-04 19:33:15 +10:30
Andrew Bettison
bacba19dc1 Refactor manifest: specific setter functions
Replace generic rhizome_manifest_set() and rhizome_manifest_set_ll()
with per-field setter functions, eg, rhizome_manifest_set_filesize().
Struct rhizome_manifest elements for all known fields, to replace the
use of rhizome_manifest_get() and rhizome_manifest_get_ll() everywhere:
sender, recipient, service, name, date, bundle_key.

Add boolean validity flags for binary blob types, to avoid having to compare
with many bytes of all-zero to detect presence, eg, has_sender, has_recipient,
has_author, has_bundle_key.  These maintained by the setter functions.

Rename existing manifest struct elements to be the same as their field
names: fileLength -> filesize, journalTail -> tail.

More use of unsigned int, size_t and uint64_t for payload sizes, offsets, byte
counts, etc. especially in rhizome_store.c and meshms.c.  More uniform use of
size_t to dimension memory buffers.  Fix some printf(3) style format strings
for 64-bit correctness on 32-bit systems.  Use new constant RHIZOME_SIZE_UNSET
instead of -1 to indicate unknown dimension, and explicitly assert its absence
before comparisons and arithmetic, for safety.

Replace some 'int' loop variables with 'unsigned' where appropriate.

Fix bugs discovered in MeshMS bundle private/public key generation and
bundle secret key handling for export/extract commands.

Instrument the first MeshMS test case to aid debugging.

New debug config flag: debug.manifest logs all modifications to all manifest
fields by setter functions.

Rename debug config flag: debug.rhizome_bind -> debug.rhizome_sql_bind.
2013-11-04 19:17:49 +10:30
Andrew Bettison
675e2021cc Add ".secret" output field to all rhizome export/extract commands 2013-11-04 17:42:22 +10:30
Andrew Bettison
4d54f55906 Improve Rhizome testdefs: add rhizome_list_unpack()
Makes all the output of a "rhizome list" command available in shell
variables
2013-11-04 17:40:02 +10:30
Andrew Bettison
7caebb97be Cosmetic changes to test framework
Re-order function definitions to group them more logically, and add a
few more comments.  No functional change whatsoever.
2013-11-04 17:40:02 +10:30
Andrew Bettison
34c5522e48 Improve test framework: tfw_cat --hexdump, TFWSTDOUT
Add -h|--hexdump option to tfw_cat() and fix faulty logic in tfw_cat()
-v option wof files not ending in \n (newline)

Add TFWSTDOUT and TFWSTDERR env vars set by execute() with absolute
path names of files containing stdout and stderr respectively (as an
alternative to using replayStdout() and replayStderr()).
2013-11-04 17:40:02 +10:30
Andrew Bettison
b1b5a79f6f Improve test framework: timeout on executing commands
Default timeout is 60 seconds, can be overridden by --timeout=N on command-line
or by TFW_EXECUTE_TIMEOUT var or by --timeout option to execute() primitive

Cull timeout sleep processes created by wait_until() primitive
2013-11-04 17:40:01 +10:30
Jeremy Lakeman
5e756bec1e Fix types for 64-bit compilation 2013-10-30 17:23:44 +10:30
Andrew Bettison
c2b78f7b29 Refactor manifest: replace int with unsigned 2013-10-30 14:14:38 +10:30
Andrew Bettison
416b82b6c0 Refactor manifest: change haveSecret to enum 2013-10-30 13:14:26 +10:30
Andrew Bettison
2ac1bc3240 Get HTTP Basic authentication working
Second 'rhizomehttp' test now passes
2013-10-30 11:07:45 +10:30
Andrew Bettison
21fe12859f Implement HTTP basic authentication
Use it in /restful/rhizome/bundlelist.json -- first 'rhizomehttp' test
passes
2013-10-29 17:32:04 +10:30
Andrew Bettison
b9faf54c91 Support more HTTP multipart inner headers
Now Content-Length and Content-Type are parsed as well as
Content-Disposition
2013-10-28 22:27:27 +10:30
Andrew Bettison
9d54c629b2 Merge branch 'naf4' into 'development'
Rewritten HTTP request parsing and response buffering
2013-10-28 18:20:40 +10:30
Andrew Bettison
3cac1b51a1 Fix subtle MIME parsing bug 2013-10-28 17:50:49 +10:30
Andrew Bettison
6ee691b161 Fix bugs in new HTTP server MIME body parsing code
Fixes all remaining 'rhizomeprotocol' test failures.
2013-10-28 14:25:16 +10:30
Andrew Bettison
8f60a4ceb5 Fix bugs in new HTTP server MIME body parsing code
Fixes 'rhizomeprotocol' test 24 HttpAddLocal.  Four tests still fail.
2013-10-28 12:08:57 +10:30
Andrew Bettison
2a9329c0c8 Fix bugs in new HTTP server MIME body parsing code
Fixes 'rhizomeprotocol' test 24 HttpImport.  Five tests still fail.
2013-10-27 13:19:23 +10:30
Andrew Bettison
0397a47753 Fix some test failures in new HTTP server code 2013-10-26 20:28:47 +10:30
Andrew Bettison
5ff5a02bb9 Fix a couple more bugs in new HTTP server code 2013-10-25 23:57:23 +10:30
Andrew Bettison
6488f7ad65 Fix basic bugs in new HTTP server code
All 'rhizomeprotocol' HTTP tests still fail
2013-10-25 17:38:51 +10:30
Andrew Bettison
04efb92ff6 Make logging D and T macros into expressions
So they can be used in-line in multi-line logical expressions.
2013-10-25 17:38:28 +10:30
Andrew Bettison
4de6c77d45 Add alloca_strdup() macro 2013-10-25 00:20:53 +10:30
Andrew Bettison
1a413b72db Add str_tolower_inplace() function 2013-10-25 00:20:53 +10:30
Andrew Bettison
95e45f452e Improve str_to_int() et al
Always set *afterp to point to the first invalid character
even if conversion fails
2013-10-25 00:20:53 +10:30
Andrew Bettison
291a631095 New header file "fdqueue.h"
So that "http_server.h" does not have to include "serval.h" which
creates a circular dependency.

Remove the __SERVALDNA__HTTP_SERVER_IMPLEMENTATION hack from
"http_server.h"
2013-10-25 00:20:53 +10:30
Andrew Bettison
48921802f5 Use size_t in Rhizome store functions 2013-10-25 00:20:53 +10:30
Andrew Bettison
05d4215752 Add 'strlen' argument to strn_fromprint() 2013-10-25 00:19:58 +10:30
Andrew Bettison
e82d22d765 Add alloca_poll_events() macro 2013-10-25 00:19:58 +10:30