ExternalVendorCode/serval-dna
1
0
Fork 0
mirror of https://github.com/servalproject/serval-dna.git synced 2025-03-22 20:05:18 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Improve REST API technical documentation (fixes #118)

Browse Source 
Users and contributors have had difficulty working out how to use the
more complex requests in the REST interface, particularly POST
/restful/rhizome/insert.

Improve the Markdown documentation to provide concrete examples of POST
requests and provide much more extensive description of how to use the
"multipart/form-data" Content-Type.
This commit is contained in:
Andrew Bettison 2017-10-20 10:03:18 +10:30
parent 9e32c01e61
commit 5eb19f1a16
2 changed files with 483 additions and 267 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

419
doc/REST-API-Rhizome.md
View File

@ -1,6 +1,6 @@
Rhizome REST API
================
[Serval Project][], February 2016
[Serval Project][], October 2017
Introduction
------------
@ -74,9 +74,6 @@ Every [Bundle](#bundle) in Rhizome is identified by its *Bundle ID*
256-bit public key in the [Curve25519][] key space, generated from the random
[Bundle Secret](#bundle-secret) when the the bundle is first created.
[BID]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:bid
[Curve25519]: https://en.wikipedia.org/wiki/Curve25519
### Bundle version
A Bundle's *version* is a 64-bit unsigned integer chosen by the bundle's
@ -134,13 +131,38 @@ lost, then the bundle becomes immutable.
### Manifest
A Rhizome bundle's *manifest* consists of two parts: a meta-data section and a
signature section.
signature section, separated by a NUL (zero) byte:
The meta-data section is a set of key-value *fields*. A field key consists of
up to 80 alphanumeric ASCII characters, and the first character must be
alphabetic. A field's value consists of zero or more bytes that may have any
value except ASCII NUL (0), CR (13) and NL (10). Conventionally, numeric
values are represented using their decimal ASCII representation.
MANIFEST = METADATA NUL SIGNATURE
If the NUL byte is missing, then the manifest is *unsigned*.
The meta-data section consists of a set of key-value *fields* in arbitrary
order, conforming to the following grammar:
METADATA = ( KEY "=" VALUE LF ){0..*}
KEY = ALPHA ( ALPHANUM ){0..79}
VALUE = ( VALUECHAR ){0..*}
ALPHA = octet in set ASCII A..F or a..f
ALPHANUM = ALPHA or octet in set ASCII 0..9
VALUECHAR = any ASCII octet except NUL CR or LF
ASCII = any octet in range 0..127
NUL = octet with value 0
LF = octet with value 10
CR = octet with value 13
The signature section uses a binary format, and consists of one or more
concatenated signature blocks. Each block begins with a single *type* byte,
followed by the bytes of the signature itself. The length of the signature is
computed as `type × 4 + 4`, not counting the type byte:
SIGNATURE = ( BLOCK ){1..*}
BLOCK = TYPE ( ANY ){TYPE * 4 + 4}
TYPE = octet with value 23
ANY = any octet in range 0..255
The only supported signature type is 23 (hex 17), which is a 96-byte signature
that is verified using [Curve25519][].
Every manifest must contain the following *core* fields, or it is *partial*:
@ -292,31 +314,66 @@ journals by discarding bytes over which the tail has advanced.
Rhizome REST API common features
--------------------------------
### text+binarysig manifest format
### Rhizome HTTP content types
The Rhizome REST API accepts and returns [manifest](#manifest)s in only one
format, denoted **text+binarysig**. The *Content-Type* for this format is
**rhizome/manifest; format=text+binarysig**.
#### rhizome/manifest
Serval DNA [POST][] requests that take [Rhizome manifest](#manifest) parameters
use the non-standard **rhizome/manifest** content type in the parameter's [form
part][]. See the [insert request](#post-restfulrhizomeinsert) for an example.
Serval DNA also uses this content type when it returns a manifest in an HTTP
response. See the [get manifest request](#get-restfulrhizomebidrhm) for an
example.
Currently only one format is supported, denoted **text+binarysig**, which must
be explicitly specified, so the correct [Content-Type][] header is:
Content-Type: rhizome/manifest; format=text+binarysig
This format is described in detail in the [manifest](#manifest) section.
In future, other formats may be supported, for example, all-binary or all-text.
The TEXT part of this format lists key-value fields in arbitrary order, using
the following grammar:
#### rhizome/bid
TEXT = ( KEY "=" VALUE "\n" ){0..*}
KEY = ALPHA ( ALPHANUM ){0..79}
VALUE = ( VALUECHAR ){0..*}
VALUECHAR = any ASCII except NUL "\r" "\n"
Serval DNA [POST](#post) requests that take [Bundle ID](#bundle-id) parameters
use the non-standard **rhizome/bid** content type in the parameter's [form
part](#multipart-form-data). See the [insert request](#post-restfulrhizomeinsert)
for an example.
Following the text is a single NUL byte, followed by the signature section in a
binary format. If the NUL byte is missing, then the manifest is *unsigned*.
At present only the **hex** format is supported, and must be explicitly
specified. A missing or different format will cause a [415 Unsupported Media
Type][415] response. The correct [Content-Type][] header is:
The signature section consists of one or more concatenated signature blocks.
Each block begins with a single *type* byte, followed by the bytes of the
signature itself. The length of the signature is computed as `type × 4 + 4`.
Content-Type: rhizome/bid; format=hex
The only supported signature type is 23 (hex 17), which is a 96-byte signature
that is verified using [Curve25519][].
Hex format parameter values may only contain ASCII characters from the set
`0123456789ABCDEFabcdef`; any other character (such as a trailing newline) will
cause a [400 Bad Request][400] response.
In future other formats may be supported, such as Base-64, 7-bit binary, or
8-bit binary.
#### rhizome/bundlesecret
Serval DNA [POST](#post) requests that take [bundle secret](#bundle-secret)
parameters use the non-standard **rhizome/bundlesecret; format=hex** content
type in the parameter's [form part](#multipart-form-data). See the [insert
request](#post-restfulrhizomeinsert) for an example.
At present only the **hex** format is supported, and must be explicitly
specified. A missing or different format will cause a [415 Unsupported Media
Type][415] response. The correct [Content-Type][] header is:
Content-Type: rhizome/bundlesecret; format=hex
Hex format parameter values may only contain ASCII characters from the set
`0123456789ABCDEFabcdef`; any other character (such as a trailing newline) will
cause a [400 Bad Request][400] response.
In future other formats may be supported, such as Base-64, 7-bit binary, or
8-bit binary.
### Rhizome HTTP response headers
@ -597,11 +654,10 @@ be [200 OK][200] and:
* the [Rhizome response bundle headers](#rhizome-response-bundle-headers) give
information about the found bundle, some of which is duplicated from the
manifest
* the response's Content-Type is **rhizome/manifest; format=text+binarysig**
* the response's Content-Length is the size, in bytes, of the manifest with
its binary signature appended
* the response's content is the Rhizome manifest in [text+binarysig
format](#textbinarysig-manifest-format)
* the response's Content-Type is [rhizome/manifest](#rihzomemanifest)
* the response's Content-Length is the size, in bytes, of the entire manifest,
including its binary signature
* the response's content is the Rhizome [manifest](#manifest)
If the **manifest is not found** in the local Rhizome store, then the response
will be [404 Not Found][404] and:
@ -664,55 +720,102 @@ eg:
/restful/rhizome/1702BD647D614DB72C36BD634B6870CA31040C2EEC5069AEC0C0841D0CC671BE/decrypted.bin
The responses are identical to those for [GET /restful/rhizome/BID/raw.bin](get-restfulrhizomebidrawbin),
with the following additional case:
If the **manifest and payload are both found** and the payload is **encrypted**
(the manifest's `crypt` field is 1), but the **payload secret is not known**,
then:
(the manifest's `crypt` field is 1), then the *payload secret* is determined as
follows:
* the [bundle status code](#bundle-status-code) will be 0
* the [payload status code](#payload-status-code) will be 5
* if the manifest has both *sender* and *recipient* [SID][]s:
* if the recipient's identity is found (unlocked) in the keyring, then the
secret is derived from the recipient's [Serval ID](#serval-id) secret;
otherwise
* if the recipient's identity is not found in the keyring (locked or missing)
but the sender's identity is found (unlocked) in the keyring, then the
secret is derived from the sender's [Serval ID](#serval-id) secret;
otherwise
* if neither identity is found in the keyring (both are locked or missing),
then the the payload secret is unknown.
* otherwise, the payload secret is derived directly from the [Bundle
Secret](#bundle-secret), which in turn is deduced from the `BK` [Bundle
Key](#bundle-key) field in the manifest, if present, as long as the bundle's
author can be found (unlocked) in the keyring. If there is no `BK` field,
or if no unlocked identity in the keyring can provide the necessary [Rhizome
Secret](#rhizome-secret), then the payload secret is unknown.
The responses are identical to [GET /restful/rhizome/BID/raw.bin](get-restfulrhizomebidrawbin),
with the following variations:
If the **payload is encrypted** and the **payload secret is known**, then
the response will be [200 OK][200] and:
* the [bundle status code](#bundle-status-code) will be 1
* the [payload status code](#payload-status-code) will be 0 if the decrypted
payload has zero length, otherwise 2
* the [Rhizome response bundle headers](#rhizome-response-bundle-headers) give
information about the found bundle, some of which is duplicated from the
manifest
* the response's Content-Type is **application/octet-stream**
* the response's Content-Length is the size, in bytes, of the decrypted
payload
* the response's content is the bundle's decrypted payload
If the **payload is encrypted** and the **payload secret is not known** then:
* the request will fail with status [419 Authentication Timeout][419]
* the [Rhizome response bundle headers](#rhizome-response-bundle-headers) give
information about the found manifest
* the response's content is the [Rhizome JSON result](#rhizome-json-result)
object
For a bundle that has a *sender* and a *recipient*, the payload secret is
determined as follows:
* if the recipient's identity is found (unlocked) in the keyring, then the
secret is derived from the recipient's [Serval ID](#serval-id) secret;
otherwise
* if the recipient's identity is not found in the keyring (locked or missing)
but the sender's identity is found (unlocked) in the keyring, then the
secret is derived from the sender's [Serval ID](#serval-id) secret;
otherwise
* neither identity is found in the keyring (both are locked or missing), so
the payload secret is unknown.
For all other bundles, the payload secret is derived directly from the [Bundle
Secret](#bundle-secret), whether supplied as a query parameter or deduced from
the bundle's [Bundle Key](#bundle-key). If the Bundle Secret is unknown, then
the payload secret is unknown.
* the response body is a [Rhizome JSON result](#rhizome-json-result) object,
in which:
* the [bundle status code](#bundle-status-code) is 0
* the [payload status code](#payload-status-code) is 5
### POST /restful/rhizome/insert
This request allows a client to add a new bundle to the Rhizome store, or
update an existing bundle in the store. This request cannot be used to create
or update [journals](#journal); use the [append](#post-restfulrhizomeappend)
request instead.
The Rhizome insert [POST][] request allows a client to add a new bundle to the
Rhizome store, or update an existing bundle in the store. This request cannot
be used to create or update [journals](#journal); use the
[append](#post-restfulrhizomeappend) request instead.
Takes the following parameters, all optional under various conditions:
This request does not accept any [query parameters][] in the *path*, but does
accept parameters using a [Content-Type][] of [multipart/form-data][], in which
each parameter has its own content type. For example:
* **bundle-id** The [Bundle ID](#bundle-id) of an existing bundle to update;
64 hexadecimal digits. If the bundle currently exists in the Rhizome store
then a copy of its manifest is used as the basis of the new bundle, omitting
its `version`, `filesize`, `filehash` fields (which must be supplied or
inferred anew).
POST /restful/rhizome/insert HTTP/1.0
Content-Type: multipart/form-data;boundary=OoOoOoOo
--OoOoOoOo
Content-Disposition: form-data; name=bundle-author
Content-Type: serval/sid;format=hex
3DA7BA5E97DF4918DB5528450875EC9F788F0C37BC2603FD1BA7FF276C575018
--OoOoOoOo
Content-Disposition: form-data; name=manifest
Content-Type: rhizome/manifest;format=text+binarysig
service=file
name=helloworld.txt
--OoOoOoOo
Content-Disposition: form-data; name=payload; filename="helloworld.txt"
Content-Type: application/octet-stream
Hello world!
--OoOoOoOo--
* **bundle-author** The [SID][] of the bundle's [author](#bundle-author):
The parameters are all optional under various conditions:
* **bundle-id** = the [Bundle ID](#bundle-id) of an existing bundle to update:
* 64 hexadecimal digits
* [Content-Type][] must be [rhizome/bid](#rhizomebid)
* if the bundle currently exists in the Rhizome store then a copy of its
manifest is used as the basis of the new bundle, omitting its `version`,
`filesize`, `filehash` fields (which must be supplied or inferred anew).
* **bundle-author** = the [SID][] of the bundle's [author](#bundle-author):
* 64 hexadecimal digits;
* [Content-Type][] must be [serval/sid][]
* the bundle author sets (or removes) the bundle's `BK` field, overriding
any `BK` field in the partial manifest supplied in the *manifest*
parameter or in the existing bundle nominated by the *bundle-id*
@ -723,22 +826,24 @@ Takes the following parameters, all optional under various conditions:
request fails with status [400 Bad Request][400] and the message
‘Spurious "bundle-id" form part’.
* **bundle-secret** The [Bundle Secret](#bundle-secret); 64 hexadecimal
digits. This is needed in order to create a bundle with a specific [Bundle
ID](#bundle-id) (supplied in the **manifest** parameter), or to update an
existing bundle that is anonymous or the author is not a currently unlocked
identity in the keyring.
* **bundle-secret** = the [Bundle Secret](#bundle-secret):
* 64 hexadecimal digits
* [Content-Type][] must be [rhizome/bundlesecret](#rhizomebundlesecret)
* is needed in order to create a bundle with a specific [Bundle ID](#bundle-id)
(supplied in the *manifest* parameter), or to update an existing bundle
that is anonymous or the author is not a currently unlocked identity in
the keyring.
* **manifest** A partial, unsigned manifest in [text+binarysig
format](#textbinarysig-manifest-format), with a correct *Content-Type*
header. The fields in this manifest are used to form the new bundle's
manifest, overwriting the fields of any existing manifest specified by the
*bundle-id* parameter, if given.
* **manifest** = a partial, unsigned [manifest](#manifest):
* [Content-Type][] must be [rhizome/manifest](#rhizomemanifest)
* the fields in this manifest are used to form the new bundle's manifest,
overwriting the fields of any existing manifest specified by the
*bundle-id* parameter, if given.
* **payload** The content of the new bundle's payload:
* the form part's *Content-Type* header is currently ignored, but in future
it may be used to determine the default values of some manifest fields;
* this parameter must occur after the *manifest* parameter, otherwise the
* **payload** = the content of the new bundle's payload:
* [Content-Type][] is currently ignored, but in future it may be used to
determine the default values of some manifest fields;
* this parameter must come after the *manifest* parameter, otherwise the
request fails with status [400 Bad Request][400] and the message ‘Missing
"manifest" form part’;
* the *payload* parameter must not be supplied if the `filesize` field in
@ -746,27 +851,26 @@ Takes the following parameters, all optional under various conditions:
The insertion logic proceeds in the following steps:
1. If the partial manifest supplied in the *manifest* parameter is malformed
(syntax error) or contains a core field with an invalid value, then the
request fails with status [422 Unprocessable Entity][422] and the [bundle
status code](#bundle-status-code) for “invalid”.
2. If a *bundle-id* parameter was supplied and the given bundle exists in the
1. If a *bundle-id* parameter was supplied and the given bundle exists in the
Rhizome store, then the new bundle's manifest is initialised by copying all
the fields from the existing manifest.
3. If a partial manifest was supplied in the *manifest* parameter, then its
2. If a partial manifest was supplied in the *manifest* parameter, then its
fields are copied into the new manifest, overwriting any that were copied
in step 2.
in step 1. If the partial manifest is malformed (syntax error) or contains
a core field with an invalid value, then the request fails with status [422
Unprocessable Entity][422] and the [bundle status
code](#bundle-status-code) for “invalid”.
4. If the `tail` field is present in the new manifest then the new bundle is a
3. If the `tail` field is present in the new manifest then the new bundle is a
[journal](#journal), so the request fails with status [422 Unprocessable
Entity][422] and the [bundle status code](#bundle-status-code) for
“invalid”. Journals can only be created and updated using the
[append](#post-restfulrhizomeappend) request.
5. If the *bundle-secret* parameter was supplied, then a public key ([Bundle
ID](#bundle-id)) is derived from the [Bundle Secret](#bundle-secret), and:
4. If the *bundle-secret* parameter was supplied, then a public key
([Bundle ID](#bundle-id)) is derived from the [Bundle Secret](#bundle-secret),
and:
* if the new manifest has no `id` field, then the `id` field is set to the
derived public key;
@ -778,16 +882,15 @@ The insertion logic proceeds in the following steps:
Otherwise, if no *bundle-secret* parameter was supplied:
* if the new manifest has no `id` field, then a new [Bundle
Secret](#bundle-secret) is generated randomly, the [Bundle
ID](#bundle-id) is derived from the new Bundle Secret, and the `id` field
set to that Bundle ID;
* if the new manifest has no `id` field, then a new [Bundle Secret](#bundle-secret)
is generated randomly, the [Bundle ID](#bundle-id) is derived from the
new Bundle Secret, and the `id` field set to that Bundle ID;
* if the new manifest already has an `id` field but no `BK` field ([Bundle
Key](#bundle-key)) (ie, the bundle is *anonymous*), then the [Bundle
Secret](#bundle-secret) cannot be discovered, so the request fails with
status [419 Authentication Timeout][419] and the [bundle status
code](#bundle-status-code) for “readonly”.
* if the new manifest already has an `id` field but no `BK` field
([Bundle Key](#bundle-key)) (ie, the bundle is *anonymous*), then the
[Bundle Secret](#bundle-secret) cannot be discovered, so the request
fails with status [419 Authentication Timeout][419] and the [bundle
status code](#bundle-status-code) for “readonly”.
* otherwise, if the *bundle-author* parameter was given, then that [SID][]
is looked up in the keyring. If the identity is found, then the [Bundle
@ -799,17 +902,21 @@ The insertion logic proceeds in the following steps:
Authentication Timeout][419] and the [bundle status
code](#bundle-status-code) for “readonly”.
* otherwise, if no *bundle-author* parameter was given, then the keyring is
searched for an identity whose [Rhizome Secret](#rhizome-secret) combined
with the `BK` field ([Bundle Key](#bundle-key)) produces a [Bundle
Secret](#bundle-secret) whose derived [Bundle ID](#bundle-id) matches the
`id` field. The search starts with the identity given by the `sender`
field, if present. If none is found, then the request fails with status
[419 Authentication Timeout][419] and the [bundle status
code](#bundle-status-code) for “readonly”, otherwise the author is
deduced to be the found identity.
* otherwise, if no *bundle-author* parameter was given but the manifest has
a `BK` field, then the keyring is searched for an identity whose [Rhizome
Secret](#rhizome-secret) combined with the `BK` field ([Bundle
Key](#bundle-key)) produces a [Bundle Secret](#bundle-secret) whose
derived [Bundle ID](#bundle-id) matches the `id` field. The search
starts with the identity given by the `sender` field, if present. If
none is found, then the request fails with status [419 Authentication
Timeout][419] and the [bundle status code](#bundle-status-code) for
“readonly”, otherwise the author is deduced to be the found identity.
6. If the *bundle-author* parameter was given and step 5 set the `id` field
* otherwise, if no *bundle-author* parameter was given and the manifest has
no `BK` field, then an *anonymous* bundle is produced, ie, with no `BK`
key.
5. If the *bundle-author* parameter was given and step 4 set the `id` field
(either derived from the *bundle-secret* parameter or randomly generated),
then the *bundle-author* [SID][] is looked up in the keyring. If not
found, then the request fails with status [419 Authentication Timeout][419]
@ -817,14 +924,14 @@ The insertion logic proceeds in the following steps:
found, then the author's [Rhizome Secret](#rhizome-secret) is used to
calculate the [Bundle Key](#bundle-key) and set the `BK` field.
7. The following fields are initialised if they are missing:
6. The following fields are initialised if they are missing:
* `service` to the value `file`
* `version` to the current [Unix time][] in milliseconds since the epoch
* `date` to the current [Unix time][] in milliseconds since the epoch
* `crypt` to `1` if the `sender` and `recipient` fields are both set
8. If the *payload* parameter is given and is non-empty, then its value is
7. If the *payload* parameter is given and is non-empty, then its value is
stored in the store, and its size and [SHA-512][] digest computed. If the
manifest is missing either or both of the `filesize` and `filehash` fields,
then the missing ones are filled in from the computed values. If the
@ -833,7 +940,7 @@ The insertion logic proceeds in the following steps:
Entity][422] and the [bundle status code](#bundle-status-code) for
“inconsistent”.
9. The manifest is *validated* to ensure that:
8. The manifest is *validated* to ensure that:
* the `id` field is present
* the `version` field is present
@ -850,7 +957,7 @@ The insertion logic proceeds in the following steps:
Entity][422] and the [bundle status code](#bundle-status-code) for
“invalid”.
10. If step 5 set the `id` field (either derived from the *bundle-secret*
9. If step 4 set the `id` field (either derived from the *bundle-secret*
parameter or randomly generated) and the bundle is a *duplicate* of a
bundle that is already in the store, then the request finishes with status
[200 OK][200] and the [bundle status code](#bundle-status-code) for
@ -862,15 +969,15 @@ The insertion logic proceeds in the following steps:
* the same `sender` field, and
* the same `recipient` field.
11. The manifest is signed using the [Bundle Secret](#bundle-secret), and the
10. The manifest is signed using the [Bundle Secret](#bundle-secret), and the
signature appended to the manifest after a single ASCII NUL (0) separator
byte. If the result exceeds the maximum manifest size (8 KiB) then the
request fails with status [422 Unprocessable Entity][422] and the [bundle
status code](#bundle-status-code) for “manifest too big”.
12. If the Rhizome store already contains a manifest with the same [Bundle
ID](#bundle-id), then its version is compared with the new manifest's
version.
11. If the Rhizome store already contains a manifest with the same
[Bundle ID](#bundle-id), then its version is compared with the new
manifest's version.
* If they have the same version, then the new manifest is not stored, and
the request returns status [200 OK][200] and the [bundle status
@ -881,7 +988,7 @@ The insertion logic proceeds in the following steps:
Accepted][202] and the [bundle status code](#bundle-status-code) for
“old”.
13. The new manifest is stored in the Rhizome store, replacing any existing
12. The new manifest is stored in the Rhizome store, replacing any existing
manifest with the same [Bundle ID](#bundle-id). The request returns status
[201 Created][201] and the [bundle status code](#bundle-status-code) for
“new”.
@ -895,34 +1002,35 @@ identical in all respects except as follows:
The steps of the insertion logic have these variations:
1. The validity checks on any partial manifest given in the *manifest*
parameter will also fail if the partial manifest contains a `version`,
`filesize` or `filehash` field.
2. If the *bundle-id* parameter specifies an existing manifest, then the
1. If the *bundle-id* parameter specifies an existing manifest, then the
`version`, `filesize` and `filehash` fields are not copied from the
existing manifest to the new manifest.
3. After the partial manifest has been copied into the new manifest, if the
*bundle-id* parameter was not given or specified a bundle that was not
found in the store (step 2), then the `filesize` and `tail` fields are
initialised to zero (0) if they are missing.
2. The validity checks on any partial manifest given in the *manifest*
parameter will also fail if the partial manifest contains a `version`,
`filesize` or `filehash` field. After the partial manifest has been copied
into the new manifest, if the *bundle-id* parameter was not given or
specified a bundle that was not found in the store (step 1), then the
`filesize` and `tail` fields are initialised to zero (0) if they are
missing.
4. If the `tail` field is missing from the new manifest then the bundle is not
3. If the `tail` field is missing from the new manifest then the bundle is not
a [journal](#journal), so the request fails with status [422 Unprocessable
Entity][422] and the [bundle status code](#bundle-status-code) for
“invalid”.
4. No change.
5. No change.
6. No change.
7. No change.
8. After the payload has been stored, the `filesize` and `filehash` fields are
7. After the payload has been stored, the `filesize` and `filehash` fields are
always set, overriding any that were already present. Also, the `version`
is always set to `tail + filesize`.
8. No change.
9. No change.
10. No change.
@ -931,28 +1039,34 @@ The steps of the insertion logic have these variations:
12. No change.
13. No change.
### POST /restful/rhizome/import
This request allows the client to store a valid manifest and payload that have been
obtained through some other means.
The import [POST](#post) request allows the client to store a valid bundle
(manifest and payload) in the store that may have been obtained through some
other means, such as exporting from another store using the [manifest
request](#get-restfulrhizomebidrhm) and the [raw payload
request](#get-restfulrhizomebidrawbin).
* **manifest** A signed manifest in [text+binarysig format](#textbinarysig-manifest-format),
with a correct *Content-Type* header.
This request accepts the following parameters using a [Content-Type][] of
[multipart/form-data][], in which each parameter has its own content type:
* **payload** The content of the bundle's payload:
* the form part's *Content-Type* header is currently ignored, but in future
it may be used to determine the default values of some manifest fields;
* this parameter must occur after the *manifest* parameter, otherwise the
* **manifest** (required) = a signed [manifest](#manifest):
* [Content-Type][] must be [rhizome/manifest](#rhizomemanifest)
* **payload** = the content of the bundle's payload:
* [Content-Type][] is currently ignored, but in future it may be used to
determine the default values of some manifest fields;
* this parameter must come after the *manifest* parameter, otherwise the
request fails with status [400 Bad Request][400] and the message ‘Missing
"manifest" form part’;
* the *payload* parameter must not be supplied if the `filesize` field in
the *manifest* parameter is zero.
the *manifest* parameter is zero;
* if the bundle is encrypted (the manifest `crypt` field is 1) then the
payload must be in encrypted form, not plain text.
-----
**Copyright 2015 Serval Project Inc.**
**Copyright 2015-2017 Serval Project Inc.**
![CC-BY-4.0](./cc-by-4.0.png)
Available under the [Creative Commons Attribution 4.0 International licence][CC BY 4.0].
@ -960,9 +1074,12 @@ Available under the [Creative Commons Attribution 4.0 International licence][CC
[Serval Project]: http://www.servalproject.org/
[CC BY 4.0]: ../LICENSE-DOCUMENTATION.md
[Rhizome]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:rhizome
[BID]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:bid
[Serval Mesh network]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:mesh_network
[Serval DNA]: ../README.md
[REST-API]: ./REST-API.md
[form part]: ./REST-API.md#multipart-form-data
[POST]: ./REST-API.md#post
[JSON result]: ./REST-API.md#json-result
[store and forward]: https://en.wikipedia.org/wiki/Store_and_forward
[SID]: ./REST-API-Keyring.md#serval-id
@ -970,13 +1087,19 @@ Available under the [Creative Commons Attribution 4.0 International licence][CC
[MeshMS]: ./REST-API-MeshMS.md
[MeshMS conversations]: ./REST-API-MeshMS.md#conversation
[JSON table]: ./REST-API.md#json-table
[Curve25519]: https://en.wikipedia.org/wiki/Curve25519
[Unix time]: https://en.wikipedia.org/wiki/Unix_time
[Y2038 problem]: https://en.wikipedia.org/wiki/Year_2038_problem
[query parameters]: https://en.wikipedia.org/wiki/Query_string
[Content-Type]: ./REST-API.md#content-type-header
[multipart/form-data]: ./REST-API.md#multipartform-data
[serval/sid]: ./REST-API.md#servalsid
[200]: ./REST-API.md#200-ok
[201]: ./REST-API.md#201-created
[202]: ./REST-API.md#202-accepted
[400]: ./REST-API.md#400-bad-request
[404]: ./REST-API.md#404-not-found
[415]: ./REST-API.md#415-unsupported-media-type
[419]: ./REST-API.md#419-authentication-timeout
[422]: ./REST-API.md#422-unprocessable-entity
[423]: ./REST-API.md#423-locked

331
doc/REST-API.md
View File

@ -1,6 +1,6 @@
REST API
========
[Serval Project][], February 2016
[Serval Project][], October 2017
Introduction
------------
@ -27,14 +27,16 @@ gives applications access to the network through two main classes of [API][]:
This document describes the features in common to all the [HTTP REST][] APIs.
### Protocol and port
Protocol and port
-----------------
The Serval DNA [HTTP REST][] API is an [HTTP 1.0][] server that only accepts
requests on the loopback interface (IPv4 address 127.0.0.1), TCP port 4110. It
rejects requests that do not originate on the local host, by replying
[403 Forbidden](#403-forbidden).
### Security
Security
--------
The REST API uses plain HTTP *without* encryption. REST requests and responses
are not carried over any physical network link, only local (“logical”) links
@ -50,7 +52,8 @@ Serval DNA and its clients, much more effective than intercepting their
communications, so encrypting client-server communications would offer no
protection whatsoever.
### Authentication
Authentication
--------------
Clients of the HTTP REST API must authenticate themselves using [Basic
Authentication][]. This narrows the window for opportunistic attacks on the
@ -77,11 +80,12 @@ PASSWORD is a cleartext secret, so the Serval DNA configuration file must be
protected from unauthorised access or modification by other apps. That makes
this mechanism unsuitable for general use.
### Request
Request
-------
An HTTP REST request is a normal [HTTP 1.0][] [GET](#get) or [POST](#post):
#### GET
### GET
A **GET** request consists of an initial "GET" line containing the *path* and
*HTTP version*, followed by zero or more header lines, followed by a blank
@ -94,98 +98,153 @@ For example:
Accept: */*
GET requests only accept parameters as [query parameters][] in the *path*.
GET requests only accept parameters as [percent encoded][] [query parameters][]
in the *path*.
[query parameters]: http://tools.ietf.org/html/rfc3986#section-3.4
#### POST
### POST
A **POST** request has a similar structure to a GET request except that the first word
of the first line is "POST".
A **POST** request has a similar structure to a GET request except that the
first word of the first line is "POST", and there may be a body of content
following the blank line that ends the header.
POST requests accept parameters as [query parameters][] in the *path* and also
as a request body with a [Content-Type](#request-content-type) of
[multipart/form-data][]. These two kinds of parameters are not exclusive; a
request may contain a mixture of both.
POST requests accept parameters as [percent encoded][] [query parameters][] in
the *path* and also as a request body with a [Content-Type](#content-type-header)
of [multipart/form-data][]. These two kinds of parameters are not exclusive; a
POST request may contain a mixture of both.
A POST request may also include a [Content-Length](#request-content-length)
or [Transfer-Encoding](#request-transfer-encoding) header;
a [Content-Type](#request-content-type); and an [Expect](#request-expect) header as described below.
A POST request may also include the following headers as described below:
* [Content-Length](#content-length-header)
* [Content-Type](#content-type-header)
* [Range](#range-header)
* [Transfer-Encoding](#transfer-encoding-header)
* [Expect](#expect-header)
#### Request Transfer-Encoding
### Content-Length header
In a request, a **Transfer-Encoding** header of "chunked", indicates that the
client can generate and transmit the request body without pre-calculating the
final length. No other transfer encodings are currently supported.
In a [POST](#post) request, the **Content-Length** header gives the exact
number of bytes (octets) in the request's body, which must be correct. Serval
DNA will not process a request until it receives Content-Length bytes, so if
Content-Length is too large, the request will suspend and eventually time out.
Serval DNA will ignore any bytes received after it has read Content-Length
bytes, so if Content-Length is too small, the request body will be malformed.
Each chunk is generated as the size of the chunk in hex, followed
by '\r\n', followed by the request bytes, followed by another '\r\n'.
Serval DNA treats a missing Content-Length header the same as a Content-Length
of zero; it will not attempt to read the request body so any body content will
be ignored if sent.
The end of the input is indicated with a chunk of zero length.
### Content-Type header
#### Request Expect
In a [POST](#post) request, the **Content-Type** header gives the [Internet
Media Type][] of the body.
The presense of an **Expect** header of "100-Continue" indicates that the server
should respond with an intermediate response of "HTTP/1.1 100 Continue" before the
client begins to transmit a request body.
If for any reason the server determines that the request body is not needed, or
the request is invalid, the server will generate the response immediately without
reading the contents of the request body.
#### Request Content-Length
In a request, the **Content-Length** header gives the exact number of bytes
(octets) in the request's body, which must be correct. Serval DNA will not
process a request until it receives Content-Length bytes, so if Content-Length
is too large, the request will suspend and eventually time out. Serval DNA
will ignore any bytes received after it has read Content-Length bytes, so if
Content-Length is too small, the request body will be malformed.
A missing Content-Length header will be treated the same as a Content-Length of zero.
#### Request Content-Type
In a request, the **Content-Type** header gives the [Internet Media Type][] of
the body. Serval DNA currently supports the following media types in requests:
* **[multipart/form-data][]** is used to send parameters in [POST](#post)
requests. The **boundary** parameter must specify a string that does not
occur anywhere within the content of any form part.
* **text/plain; charset=utf-8** is used for [MeshMS][] message form parts.
The only supported charset is utf-8; a missing or different charset will
cause a [415 Unsupported Media Type](#415-unsupported-media-type) response.
* **rhizome/manifest; format=text+binarysig** is used for [Rhizome
manifest][]s in [text+binarysig format][].
A missing Content-Type header in a `POST` request will cause a [400 Bad
A missing Content-Type header in a [POST](#post) request will cause a [400 Bad
Request](#400-bad-request) response. An unsupported content type will cause a
[415 Unsupported Media Type](#415-unsupported-media-type) response.
The following media types are *not supported*:
#### multipart/form-data
* [application/x-www-form-urlencoded][] is commonly used to send parameters in
[POST](#post) requests, and is the predecessor web standard to
[multipart/form-data][]. It has the benefit of being simpler than
[multipart/form-data][] for requests that take short, mainly textual
parameters, but is very inefficient for encoding large binary values and
does not provide any means to associate metadata such as content-type and
encoding with individual parameters. In future, some REST API requests may
support [application/x-www-form-urlencoded][].
Serval DNA [POST](#post) requests that take their parameters in the request
body accept the **[multipart/form-data][]** content type.
[multipart/form-data]: https://www.ietf.org/rfc/rfc2388.txt
[application/x-www-form-urlencoded]: https://tools.ietf.org/html/rfc1866#section-8.2.1
The multipart **boundary** must specify a string consisting of at most 70 ASCII
characters that does not occur anywhere within the content of any part, and is
used to delimit the parts of the request body. Typically this string is
generated using a large random number encoded into printable ASCII, eg:
#### Request Range
Content-Type: multipart/form-data;boundary=081d31d4c3d23014
[HTTP 1.1 Range][] retrieval is partially supported. In a request, the
**Range** header gives the start and end, in byte offsets, of the resource to
be returned. The server may respond with exactly the range requested, in which
case the response status code will be [206 Partial Content](#206-partial-content),
or it may ignore the Range header and respond with the entire requested
resource.
Each form part introduces one named parameter, and consists of its own header
section, followed by a blank line (a CR-LF immediately following the CR-LF that
terminates the last header), followed by a body that contains the parameter's
content, followed by a CR-LF. Every part must have a [Content-Disposition][]
header that gives the parameter's name, and may have an optional
[Content-Type][] header.
The following example passes two parameters to a request, called `first` and
`second`, the first is a single line of plain text terminated by a LF, the
second is an HTML document with no terminating LF:
POST /an/example/request HTTP/1.0
Content-Type: multipart/form-data;boundary=4aceafdc5cc7295d
--4aceafdc5cc7295d
Content-Disposition: form-data; name=first
Content-Type: text/plain; charset=utf-8
Hello, world!
--4aceafdc5cc7295d
Content-Disposition: form-data; name=second; filename="hello_world.html"
Content-Type: text/html; charset=utf-8
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>The second parameter</title>
</head>
<body>
<h1>Hello world!</h1>
</body>
</html>
--4aceafdc5cc7295d--
#### application/x-www-form-urlencoded
Serval DNA [POST](#post) requests *do not support* the
[application/x-www-form-urlencoded][] content type, although some may do so in
future.
[application/x-www-form-urlencoded][] is the predecessor web standard to
[multipart/form-data](#multipart-form-data). It has the benefit of being
simpler for requests that take short, mainly textual parameters, but is very
inefficient for encoding large binary values and does not provide any means to
associate metadata such as content-type, encoding and file name with individual
parameters.
#### text/plain
Serval DNA [POST](#post) requests that take plain text parameters use the
**text/plain** content type in the parameter's [form part](#multipart-form-data).
See the [MeshMS send request][] for an example.
The only supported charset is **utf-8**; a missing or different charset will
cause a [415 Unsupported Media Type](#415-unsupported-media-type) response.
The correct [Content-Type](#content-type-header) header is:
Content-Type: text/plain; charset=utf-8
#### serval/sid
Serval DNA [POST](#post) requests that take [SID][] parameters use the
non-standard **serval/sid** content type within the parameter's [form
part](#multipart-form-data). See the [Rhizome insert request][] for an
example.
At present only the **hex** format is supported, and must be explicitly
specified. A missing or different format will cause a [415 Unsupported Media
Type](#415-unsupported-media-type). The correct [Content-Type](#content-type-header)
header is:
Content-Type: serval/sid; format=hex
Hex format parameter values may only contain ASCII characters from the set
`0123456789ABCDEFabcdef`; any other character (such as a trailing newline) will
cause a [400 Bad Request](#400-bad-request) response.
In future other formats may be supported, such as Base-64, 7-bit binary, or
8-bit binary.
### Range header
[HTTP 1.1 Range][] retrieval is partially supported. In a [POST](#post)
request, the **Range** header gives the start and end, in byte offsets, of the
resource to be returned. The server may respond with exactly the range
requested, in which case the response status code will be [206 Partial
Content](#206-partial-content), or it may ignore the Range header and respond
with the entire requested resource.
For example, the following header asks that the server omit the first 64 bytes
and send only the next 64 bytes (note that ranges are inclusive of their end
@ -200,9 +259,33 @@ response may be the entire content or [501 Not Implemented](#501-not-implemented
[HTTP 1.1 Range]: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35
### Responses
### Transfer-Encoding header
An HTTP REST response is a normal [HTTP 1.0][] response consisting of a header
In a [POST](#post) request, a **Transfer-Encoding** header of "chunked",
indicates that the client can generate and transmit the request body without
pre-calculating the final length. No other transfer encodings are currently
supported.
Each chunk is generated as the size of the chunk in hex, followed by CR-LF,
followed by the request bytes, followed by another CR-LF.
The end of the input is indicated with a chunk of zero length.
### Expect header
In a [POST](#post) request, the presense of an **Expect** header of
"100-Continue" indicates that the server should respond with an intermediate
response of "HTTP/1.1 100 Continue" before the client begins to transmit a
request body.
If for any reason the server determines that the request body is not needed, or
the request is invalid, the server will generate the response immediately without
reading the contents of the request body.
Responses
---------
An HTTP REST *response* is a normal [HTTP 1.0][] response consisting of a header
block, a blank line, and an optional body, for example: As usual, all lines are
terminated by an ASCII CR-LF sequence. For example:
@ -224,14 +307,15 @@ return to the client; for example, [Rhizome response headers](#rhizome-response-
[application/json]: https://tools.ietf.org/html/rfc4627
### Response status code
Response status code
--------------------
The HTTP REST API response uses the [HTTP status code][] to indicate the
outcome of the request as follows:
[HTTP status code]: http://www.w3.org/Protocols/HTTP/1.0/spec.html#Status-Codes
#### 200 OK
### 200 OK
The operation was successful and no new entity was created. Most requests
return this code to indicate success. Requests that create a new entity only
@ -245,13 +329,13 @@ entity twice yields the same state as creating it once. This is an important
property for a purely distributed network that has no central arbiter to
enforce sequencing of operations.)
#### 201 Created
### 201 Created
The operation was successful and the entity was created. This code is only
returned by requests that create new entities, in the case that the entity did
not exist beforehand and has been created successfully.
#### 202 Accepted
### 202 Accepted
The operation was successful but the entity was not created. This code is only
returned by requests that create new entities, in the case that the request was
@ -261,23 +345,23 @@ this code when inserting a bundle to a full Rhizome store if the new bundle's
rank falls below all other bundles, so the new bundle itself would be evicted
to make room.
#### 206 Partial Content
### 206 Partial Content
The operation was successful and the response contains part of the requested
content. This code is only returned by requests that fetch an entity (the
fetched entity forms the body of the response) if the request supplied a
[Range](#request-range) header that specified less than the entire entity.
[Range](#range-header) header that specified less than the entire entity.
#### 400 Bad Request
### 400 Bad Request
The HTTP request was malformed, and should not be repeated without
modifications. This could be for several reasons:
- invalid syntax in the request header block
- a `POST` request MIME part is missing, duplicated or out of order
- a `POST` request was given an unsupported MIME part
- a `POST` request MIME part has missing or malformed content
- a [POST](#post) request parameter is missing, duplicated or out of order
- a [POST](#post) request was given an unsupported parameter
- a [POST](#post) request parameter has missing or malformed content
#### 401 Unauthorized
### 401 Unauthorized
The request did not supply an "Authorization" header with a recognised
credential. This response contains a "WWW-Authenticate" header that describes
@ -293,12 +377,12 @@ the missing credential:
"http_status_message": "Unauthorized"
}
#### 403 Forbidden
### 403 Forbidden
The request failed because the server does not accept requests from the
originating host.
#### 404 Not Found
### 404 Not Found
The request failed because the [HTTP request URI][] does not exist. This could
be for several reasons:
@ -311,7 +395,7 @@ be for several reasons:
[HTTP request URI]: http://www.w3.org/Protocols/HTTP/1.0/spec.html#Request-URI
#### 405 Method Not Allowed
### 405 Method Not Allowed
The request failed because the [HTTP request method][] is not supported for the
given path. Usually this means that a [GET](#get) request was attempted on a
@ -319,23 +403,23 @@ path that only supports [POST](#post), or vice versa.
[HTTP request method]: http://www.w3.org/Protocols/HTTP/1.0/spec.html#Method
#### 411 Length Required
### 411 Length Required
A `POST` request did not supply either a [Content-Length](#request-content-length)
or [Transfer-Encoding](#request-transfer-encoding) header.
A [POST](#post) request did not supply either a [Content-Length](#content-length-header)
or [Transfer-Encoding](#transfer-encoding-header) header.
#### 414 Request-URI Too Long
### 414 Request-URI Too Long
The request failed because the [HTTP request URI][] was too long. The server
persists the path and a few other pieces of the request in a fixed size request
buffer, and this response is triggered if the collective size of these does not
leave enough buffer for receiving the remainder of the request.
#### 415 Unsupported Media Type
### 415 Unsupported Media Type
A `POST` request failed because of an unsupported content type, which could be
for several reasons:
- the request's [Content-Type](#request-content-type) header specified an
A [POST](#post) request failed because of an unsupported content type, which
could be for several reasons:
- the request's [Content-Type](#content-type-header) header specified an
unsupported media type
- a part of a [multipart/form-data][] request body has:
- a missing `Content-Disposition` header, or
@ -343,27 +427,27 @@ for several reasons:
- a missing or unsupported `Content-Type` header (including a missing or
unsupported `charset` parameter)
#### 416 Requested Range Not Satisfiable
### 416 Requested Range Not Satisfiable
The [Range](#request-range) header specified a range whose start position falls
The [Range](#range-header) header specified a range whose start position falls
outside the size of the requested entity.
#### 419 Authentication Timeout
### 419 Authentication Timeout
The request failed because the server does not possess and cannot derive the
necessary cryptographic secret or credential. For example, updating a [Rhizome
bundle][] without providing the [bundle secret][]. This code is not part of
the HTTP standard.
#### 422 Unprocessable Entity
### 422 Unprocessable Entity
A `POST` request supplied data that was inconsistent or violates semantic
A [POST](#post) request supplied data that was inconsistent or violates semantic
constraints, so cannot be processed. For example, the [Rhizome
insert](./REST-API-Rhizome.md#post-restfulrhizomeinsert) operation responds
with 422 if the manifest *filesize* and *filehash* fields do not match the
supplied payload.
#### 423 Locked
### 423 Locked
The request cannot be performed because a necessary resource is busy for
reasons outside the control of the requester and server.
@ -375,7 +459,7 @@ from directly accessing the Rhizome database. Once these improvements are
done, this code should no longer occur except during unusual testing and
development situations.
#### 429 Too Many Requests
### 429 Too Many Requests
The request cannot be performed because a necessary resource is temporarily
unavailable due to a high volume of concurrent requests.
@ -390,7 +474,7 @@ resources. For example, if [Serval DNA][] is ever limited to service only a
few HTTP requests at a time, then this code will be returned to new requests
that would exceed the limit.
#### 431 Request Header Fields Too Large
### 431 Request Header Fields Too Large
The request header block was too long.
@ -399,7 +483,7 @@ buffer memory for each [request](#request), and the HTTP server read each
header line entirely into that buffer before parsing it. If a single header
exceeded the size of this buffer, then the 431 response was returned.
#### 500 Internal Server Error
### 500 Internal Server Error
The request failed because of an internal error in [Serval DNA][], not an error
in the request itself. This could be for several reasons:
@ -412,14 +496,15 @@ re-tried, but in general they will persist because the cause is not transient.
Temporary failures that can be resolved by re-trying the request are generally
indicated by other status codes, such as [423 Locked](#423-locked).
#### 501 Not Implemented
### 501 Not Implemented
The requested operation is valid but not yet implemented. This is used for the
following cases:
- a request [Range](#request-range) header specifies a multi range
- a [POST](#post) request [Range](#range-header) header specifies a multi range
#### Cross-Origin Resource Sharing (CORS)
Cross-Origin Resource Sharing (CORS)
------------------------------------
To support client-side JavaScript applications, Serval DNA has a limited
implementation of [Cross-Origin Resource Sharing][CORS]. If a request contains
@ -445,7 +530,8 @@ Serval DNA will respond:
[CORS]: http://www.w3.org/TR/cors/
#### JSON result
JSON result
-----------
All responses that convey no special content return the following *JSON result*
object:
@ -468,7 +554,8 @@ found”, “Payload not found”, etc.
Some responses augment the *JSON result* object with extra fields; for example,
[Rhizome JSON result][] and [Keyring JSON result][].
### JSON table
JSON table
----------
Many HTTP REST responses that return a list of regular objects (eg, [GET
/restful/rhizome/bundlelist.json](./REST-API-Rhizome.md#get-restfulrhizomebundlelistjson))
@ -517,7 +604,7 @@ expression to perform the transformation:
]
-----
**Copyright 2015 Serval Project Inc.**
**Copyright 2015-2017 Serval Project Inc.**
![CC-BY-4.0](./cc-by-4.0.png)
Available under the [Creative Commons Attribution 4.0 International licence][CC BY 4.0].
@ -546,9 +633,15 @@ Available under the [Creative Commons Attribution 4.0 International licence][CC
[Rhizome bundle]: ./REST-API-Rhizome.md#bundle
[Rhizome manifest]: ./REST-API-Rhizome.md#manifest
[Rhizome JSON result]: ./REST-API-Rhizome.md#rhizome-json-result
[Rhizome insert request]: ./REST-API-Rhizome.md#post-restfulrhizomeinsert
[MeshMS send request]: ./REST-API-MeshMS.md#post-restfulmeshmssendersidrecipientsidsendmessage
[Keyring JSON result]: ./REST-API-Keyring.md#keyring-json-result
[bundle secret]: ./REST-API-Rhizome.md#bundle-secret
[text+binarysig format]: ./REST-API-Rhizome.md#textbinarysig-manifest-format
[multipart/form-data]: https://www.ietf.org/rfc/rfc7578
[Content-Disposition]: https://tools.ietf.org/html/rfc7578#section-4.2
[Content-Type]: https://tools.ietf.org/html/rfc7578#section-4.4
[application/x-www-form-urlencoded]: https://tools.ietf.org/html/rfc1866#section-8.2.1
[percent encoded]: https://en.wikipedia.org/wiki/Percent-encoding
[JSON]: https://en.wikipedia.org/wiki/JSON
[UTF-8]: https://en.wikipedia.org/wiki/UTF-8
[jq(1)]: https://stedolan.github.io/jq/