Commit Graph

760 Commits

Author SHA1 Message Date
Jeremy Lakeman
44e73b969e Add new rhizome sync status API 2018-06-26 16:41:27 +09:30
Jeremy Lakeman
18bc162361 Add Java API for restful storage information request 2018-06-26 16:41:27 +09:30
Jeremy Lakeman
4aeb6545ad Refactor storage space measurement to expose new status API
Change semantics of fake space limit to exclude used space, otherwise the result will keep changing on each call
2018-06-26 16:41:27 +09:30
Jeremy Lakeman
5ec74ea307 Rewrite json parser to encapsulate object and array parsing 2018-06-19 14:25:29 +09:30
Jeremy Lakeman
0c5a84c305 Fallback to streaming bytes when rename() fails due to crossing filesystems 2018-05-22 16:05:27 +09:30
Andrew Bettison
9e89a6ddcf Fix response of PUT /restful/keyring/SID/lock (fixes #139)
To be consistent with documentation, the response must contain the
details of the locked identity.  Added relevant assertions to the
lock/unlock test case.
2018-05-17 11:39:37 +09:30
Jeremy Lakeman
d02bf3b7b9 Don't release keyring entries just because a user requested them (Fixes #137) 2018-05-16 11:47:02 +09:30
Andrew Bettison
7ab9bc0a10 Replace netcat6 with OpenBSD netcat
Netcat6 has been discontinued in favour of OpenBSD netcat.
2018-04-27 17:29:02 +09:30
Jeremy Lakeman
2f57fe3a47 Fix msp tests to use unused file descriptor 2018-04-24 12:53:56 +09:30
Andrew Bettison
f2e8198dba Fix tests broken by new Rhizome store directory
The recent faa0f39 did not update these tests that look in the "blob"
subdirectory.
2018-04-23 13:26:11 +09:30
Andrew Bettison
faa0f392c5 Improve Rhizome DB path configuration
Change the default 'rhizome.datastore_path' to "rhizome", ie, a sub-
directory of the instance directory.

Add a new erename() operation to "os.h".

Clean up the code around the fix 39f582cca (two commits ago) for a bug
introduced in 2014: the 'rhizome.db' file was always being created in
the instance directory, regardless of the setting of the
'rhizome.datastore_path' config option.

Whenever the Rhizome database is opened (by the server or by a CLI
invocation), if the database file does not exist but there is a
'rhizome.db' file in the legacy location, then move that file (and any
accompanying "blob" and "hash" sub-directories) to the configured
location.

Update copyright notices.
2018-04-17 12:01:02 +09:30
Jeremy Lakeman
39f582ccab Auto move rhizome store when config is first set 2018-04-10 16:50:09 +09:30
Jeremy Lakeman
5e0e59e8b8 Fix tests/rhizomeall 2018-04-10 10:52:44 +09:30
Jeremy Lakeman
59b6a70374 Use config to set and remember rhizome database location 2018-04-10 10:52:44 +09:30
Jeremy Lakeman
57767e444d Keep rhizome's external blobs in subfolders, migrating each file on first use 2018-04-10 10:52:44 +09:30
Andrew Bettison
a691a39870 Add Route Swift API
Uses the Route REST API internally.

Create the 'allswift' test script that aggregates the Keyring and Route
Swift API tests.
2018-04-09 14:43:37 +09:30
Andrew Bettison
dce8b378d2 Refactor routing and Swift test set-up
Add some servald config set-up functions to 'testdefs_routing.sh' and
'testdefs_swift.sh' that can be re-used in the Swift Routing API tests.

Improve some commenting on existing functions, for consistency.
2018-04-09 09:22:36 +09:30
Andrew Bettison
12bc773d4d Refactor Keyring Swift API
Factor JSON table parsing into a separate method of ServalRestfulClient
so it can be re-used by other APIs.
2018-04-09 07:50:42 +09:30
Andrew Bettison
bc637e1a5c Update Swift API to use new Keyring REST API
Now uses the correct, documented REST requests (not the legacy ones) and
includes a single-identity query.
2018-04-06 08:51:31 +09:30
Andrew Bettison
189485a513 Add Route Java API 2018-04-05 18:12:59 +09:30
Andrew Bettison
8345d896a6 Add Route REST API (fixes #96)
Consists of a single, one-shot request, GET /restful/route/all.json, for
the time being.  A "newsince" request can be added later.

Add a Markdown tech doc that specifies the new Route REST API and update
the MDP tech doc a little.

Add the 'routerestful' test script, which uses some test utility
fuctions that have been factored out of the 'routing' test script into
the new testdefs_routing.sh.

Add the new 'allrestful' test script.
2018-03-29 15:31:03 +10:30
Andrew Bettison
2ef315b692 Fix tfw_createfile utility
Was not omitting characters given by the --omit option.
2018-03-29 15:19:54 +10:30
Andrew Bettison
2e0cb4e6dc Improve test framework error handling
If runTests() terminated with a FATAL condition, the cause was not
always apparent, because the error message usually got overwritten by
the test progress output on the user's terminal.  To fix this, the main
loop's standard error is now collected in a temporary file which is sent
to standard error just before exit.

The test framework was not always handling its internal FATAL error code
(255) correctly, so this has been fixed.

One cause of a FATAL termination was if a test's unique temporary
directory, which was based on its Process ID, already existed.  Anything
that left a temporary directory behind increased the likelihood of a
FATAL in a subsequent test run.  (For example, one keyring test case was
not killing its Serval DNA daemon, and the daemon was re-creating its
instance directory before eventually terminating itself.  This test case
has been fixed.)

The test framework now allocates its temporary directory by re-trying
different random numbers until mkdir(1) succeeds, and only fataling
after 20 failures.
2018-03-29 15:09:45 +10:30
Andrew Bettison
4f5567e2c1 Use rigorous quoting throughout test scripts
This makes it much easier to refactor test fixtures and utilities.
2018-03-27 17:29:21 +10:30
Andrew Bettison
1936015ac7 Fix Rhizome protocol "ManyFiles" test
Increase the timeout from 30 to 240 seconds.  Reduce debug logging.  Fix
bundle authors.
2018-03-27 17:29:21 +10:30
Andrew Bettison
bc7f7aeaaf Refactor all REST API tests
Rename 'testdefs_json.sh' to 'testdefs_rest.sh'; add setup functions and
rest_request() function, and roll out through existing test scripts.
2018-03-27 17:29:21 +10:30
Andrew Bettison
9ef7d33d69 Log more debug in MDP_ROUTE_TABLE request 2018-03-27 09:06:41 +10:30
Andrew Bettison
007a7af122 Update Java API to use new Keyring REST API
The ServalDCient class now requires a "verb" parameter as well as the
URL parameter.

Add Java API method to query a single keyring identity.
2018-03-19 18:07:33 +10:30
Andrew Bettison
98ec1c9608 Redesign the Keyring REST API (fixes #132)
The new API follows REST rules by using the proper request verbs:
POST, PUT, PATCH and DELETE, instead of just GET.

The legacy GET-only API is still supported for backward compatibility,
but not longer tested or documented.

Add a new query-single-identity operation.

Implement the lock-single-identity operation, which until now had been
documented but not yet implemented.  Whenever a single identity is
locked (released), any other unlocked identities with the same PIN are
flagged to indicate that the PIN is not "fully" unlocked, so that the
next time the PIN is entered, the slot decryption is re-tried for
non-loaded identities, and the locked identity will be unlocked again.

Update the 'keyring' and 'keyringrestful' test scripts:
- refactored to reduce curl command-line clutter in test cases
- now tests the redesigned request verbs and paths
- added a test for GET /restful/keyring/SID
- added a test for PUT /restful/keyring/SID/lock
2018-03-19 18:06:23 +10:30
Andrew Bettison
fad187046e Add missing tests/alljava script 2018-03-06 18:31:26 +10:30
Andrew Bettison
3ffa4b10af Add Swift keyring client API with tests
Add a swift-client-api subdirectory containing a Swift source package
and a Makefile.in that compiles it into the "ServalClient" Swift module
using the Swift package manager.  The Swift API contains the following
classes:

- ServalKeyring provides the operations: add, remove, set, list
- AbstractId and its specialisation SubscriberId, already in near-final
  form, are data types for SID and the like
- ServalRestfulClient (internal) uses an HTTP client to access the
  Serval DNA RESTful interface

Improve the REST /keyring/set operation to only alter the DID or Name if
the corresponding query parameter is supplied.  Modify the internal
keyring_set_did() function to only assign the DID or Name if the
corresponding parameter is not a null pointer.

The configure script ensures that the Swift build target version is
10.10 or later when compiling for Mac OS-X, so that the package manager
will succeed.

Add autoconf macros for the Swift package manager.
2018-03-06 15:29:17 +10:30
Andrew Bettison
e9d91d04ca Fix failing Rhizome Java API tests
Update the Java API to be consistent with recent changes to the Rhizome
REST API was that regularised the MIME types used in Content-Type
headers.

Add some missing CONTENT_TYPE_ constants to "httpd.h", and ensure they
are consistent with the Java getMimeType() methods in subclasses of
AbstractId.

Add the --stderr option to all executeJavaOk statements in the
'rhizomejava' test script, to reveal the exception message in the case
of failure.
2018-03-06 15:16:56 +10:30
Andrew Bettison
16d4a6d18e Fix the $JAVAC conditional in tests/all
Add a test script for all Java tests, called "alljava".

Include testconfig.sh in testdefs.sh, not testdefs_java.sh, so that the
tests/all script can successfully test the $JAVAC variable.
2018-03-06 15:16:56 +10:30
Andrew Bettison
92fa6c196a Rewrite logging system
Rename the logging primitive functions and utility functions, prefixing
all with 'serval_log', eg: logMessage() -> serval_logf() etc.

Add an XPRINTF xhexdump() function and use it to implement the
serval_log_hexdump() utility, renamed from dump().  Add macros
WHY_dump(), WARN_dump(), HINT_dump() and DEBUG_dump(), and use them
everywhere.

Remove the 'log.console.dump_config' and 'log.file.dump_config'
configuration options; configuration is now dumped in every log prolog.

The logging system now constructs the log prolog by invoking the new
'log_prolog' trigger, so that it no longer depends on the version string
and configuration system.  Any system that wants to present a message in
the log prolog can define its own trigger, which calls standard log
primitives to print the message.

Split the logging system into a front-end (log.c) that provides the
logging primitives and is independent of the configuration system, and a
set of back-end "outputters" (log_output_console.c, log_output_file.c,
log_output_android.c) that may depend on the configuration system and
are decoupled from the front-end using the 'logoutput' link section.

These log outputters are explicitly linked into executables by the
Makefile rules, but could also be linked in using USE_FEATURE().  The
USE_FEATURE() calls have _not_ been added to servald_features.c, so that
different daemon executables can be built with the same feature set but
different log outputs.
2018-03-06 15:16:56 +10:30
Andrew Bettison
cc016f7bf7 Fix two broken rhizomerestful tests
On Debian testing, it appears that curl(1) no longer sends an Expect:
100-continue header by default, so the test script must explicitly
supply it.
2017-11-21 15:11:48 +10:30
Jeremy Lakeman
ce2c276570 Detect Wifi and cabled ethernet on linux systems, allow matching by type in config 2017-11-07 16:53:19 +10:30
Andrew Bettison
68deae8393 Improve out-of-tree build support
Use the conventional autoconf @srcdir@ substitution instead of requiring
the SOURCE_PREFIX variable to be passed to the Makefile.

Also support running test scripts in an out-of-tree build directory.
2017-10-23 17:51:42 +10:30
Andrew Bettison
34e2e8d4bc Improve and document HTTP REST Rhizome import
The REST Rhizome import request now requires the 'id' and 'version'
query parameters to either both be supplied or neither, and fails if
they do not match the manifest that is supplied in the request body.
Added a test case for this.

Added a test case to ensure that if the 'id' and 'version' query
parameters cause a hit (already in store) then the response is sent
immediately without reading the request body.

Improve the documentation for the REST Rhizome import request.
2017-10-20 15:50:18 +10:30
Andrew Bettison
fc478d07ff Preserve curl trace files in all rhizomerestful test cases
These trace files make it easier to see the innards of REST
API requests and responses, which improves the value of tests
as API specification/documentation.
2017-10-20 09:47:34 +10:30
Jeremy Lakeman
3333d2faa2 Compare MeshMS sender first, so you can talk to yourself (Fixes #124) 2017-09-06 12:38:53 +09:30
Jeremy Lakeman
d1a1a41f08 Escape codepoints U+10000 to U+10FFFF in json output as surrogate pairs 2017-06-29 19:23:51 +09:30
Jeremy Lakeman
65932e05db Improve reliability of last routing test 2017-06-20 16:32:12 +09:30
Jeremy Lakeman
9cb561229a Allow following a feed that isn't in rhizome yet, and override the displayed name 2017-06-20 12:17:57 +09:30
Jeremy Lakeman
508e95436a Add API's for marking a feed as blocked 2017-06-07 13:52:44 +09:30
Jeremy Lakeman
af2d32c25b Add Java API for importing bundles with manifests in zip comments 2017-05-24 13:33:03 +09:30
Jeremy Lakeman
2db8c24e15 Use chunked transfer encoding and expect header in Java API 2017-05-23 11:37:26 +09:30
Jeremy Lakeman
77b603942f Support "Transfer-Encoding: chunked" for POST'ed content 2017-05-22 16:17:17 +09:30
Jeremy Lakeman
87d171cdf7 Reject rhizome import before sending manifest & payload 2017-05-17 14:55:56 +09:30
Jeremy Lakeman
e376c1cb80 Add proper server support for http/1.1 Expect header 2017-05-17 14:24:28 +09:30
Jeremy Lakeman
a473304c06 Add restful import API 2017-05-17 10:16:46 +09:30