From 5eb19f1a16394c581e286804d602f9ea32eff19c Mon Sep 17 00:00:00 2001 From: Andrew Bettison Date: Fri, 20 Oct 2017 10:03:18 +1030 Subject: [PATCH] Improve REST API technical documentation (fixes #118) Users and contributors have had difficulty working out how to use the more complex requests in the REST interface, particularly POST /restful/rhizome/insert. Improve the Markdown documentation to provide concrete examples of POST requests and provide much more extensive description of how to use the "multipart/form-data" Content-Type. --- doc/REST-API-Rhizome.md | 419 ++++++++++++++++++++++++++-------------- doc/REST-API.md | 331 +++++++++++++++++++------------ 2 files changed, 483 insertions(+), 267 deletions(-) diff --git a/doc/REST-API-Rhizome.md b/doc/REST-API-Rhizome.md index d5ad60c0..f991bc37 100644 --- a/doc/REST-API-Rhizome.md +++ b/doc/REST-API-Rhizome.md @@ -1,6 +1,6 @@ Rhizome REST API ================ -[Serval Project][], February 2016 +[Serval Project][], October 2017 Introduction ------------ @@ -74,9 +74,6 @@ Every [Bundle](#bundle) in Rhizome is identified by its *Bundle ID* 256-bit public key in the [Curve25519][] key space, generated from the random [Bundle Secret](#bundle-secret) when the the bundle is first created. -[BID]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:bid -[Curve25519]: https://en.wikipedia.org/wiki/Curve25519 - ### Bundle version A Bundle's *version* is a 64-bit unsigned integer chosen by the bundle's @@ -134,13 +131,38 @@ lost, then the bundle becomes immutable. ### Manifest A Rhizome bundle's *manifest* consists of two parts: a meta-data section and a -signature section. +signature section, separated by a NUL (zero) byte: -The meta-data section is a set of key-value *fields*. A field key consists of -up to 80 alphanumeric ASCII characters, and the first character must be -alphabetic. A field's value consists of zero or more bytes that may have any -value except ASCII NUL (0), CR (13) and NL (10). Conventionally, numeric -values are represented using their decimal ASCII representation. + MANIFEST = METADATA NUL SIGNATURE + +If the NUL byte is missing, then the manifest is *unsigned*. + +The meta-data section consists of a set of key-value *fields* in arbitrary +order, conforming to the following grammar: + + METADATA = ( KEY "=" VALUE LF ){0..*} + KEY = ALPHA ( ALPHANUM ){0..79} + VALUE = ( VALUECHAR ){0..*} + ALPHA = octet in set ASCII A..F or a..f + ALPHANUM = ALPHA or octet in set ASCII 0..9 + VALUECHAR = any ASCII octet except NUL CR or LF + ASCII = any octet in range 0..127 + NUL = octet with value 0 + LF = octet with value 10 + CR = octet with value 13 + +The signature section uses a binary format, and consists of one or more +concatenated signature blocks. Each block begins with a single *type* byte, +followed by the bytes of the signature itself. The length of the signature is +computed as `type × 4 + 4`, not counting the type byte: + + SIGNATURE = ( BLOCK ){1..*} + BLOCK = TYPE ( ANY ){TYPE * 4 + 4} + TYPE = octet with value 23 + ANY = any octet in range 0..255 + +The only supported signature type is 23 (hex 17), which is a 96-byte signature +that is verified using [Curve25519][]. Every manifest must contain the following *core* fields, or it is *partial*: @@ -292,31 +314,66 @@ journals by discarding bytes over which the tail has advanced. Rhizome REST API common features -------------------------------- -### text+binarysig manifest format +### Rhizome HTTP content types -The Rhizome REST API accepts and returns [manifest](#manifest)s in only one -format, denoted **text+binarysig**. The *Content-Type* for this format is -**rhizome/manifest; format=text+binarysig**. +#### rhizome/manifest + +Serval DNA [POST][] requests that take [Rhizome manifest](#manifest) parameters +use the non-standard **rhizome/manifest** content type in the parameter's [form +part][]. See the [insert request](#post-restfulrhizomeinsert) for an example. + +Serval DNA also uses this content type when it returns a manifest in an HTTP +response. See the [get manifest request](#get-restfulrhizomebidrhm) for an +example. + +Currently only one format is supported, denoted **text+binarysig**, which must +be explicitly specified, so the correct [Content-Type][] header is: + + Content-Type: rhizome/manifest; format=text+binarysig + +This format is described in detail in the [manifest](#manifest) section. In future, other formats may be supported, for example, all-binary or all-text. -The TEXT part of this format lists key-value fields in arbitrary order, using -the following grammar: +#### rhizome/bid - TEXT = ( KEY "=" VALUE "\n" ){0..*} - KEY = ALPHA ( ALPHANUM ){0..79} - VALUE = ( VALUECHAR ){0..*} - VALUECHAR = any ASCII except NUL "\r" "\n" +Serval DNA [POST](#post) requests that take [Bundle ID](#bundle-id) parameters +use the non-standard **rhizome/bid** content type in the parameter's [form +part](#multipart-form-data). See the [insert request](#post-restfulrhizomeinsert) +for an example. -Following the text is a single NUL byte, followed by the signature section in a -binary format. If the NUL byte is missing, then the manifest is *unsigned*. +At present only the **hex** format is supported, and must be explicitly +specified. A missing or different format will cause a [415 Unsupported Media +Type][415] response. The correct [Content-Type][] header is: -The signature section consists of one or more concatenated signature blocks. -Each block begins with a single *type* byte, followed by the bytes of the -signature itself. The length of the signature is computed as `type × 4 + 4`. + Content-Type: rhizome/bid; format=hex -The only supported signature type is 23 (hex 17), which is a 96-byte signature -that is verified using [Curve25519][]. +Hex format parameter values may only contain ASCII characters from the set +`0123456789ABCDEFabcdef`; any other character (such as a trailing newline) will +cause a [400 Bad Request][400] response. + +In future other formats may be supported, such as Base-64, 7-bit binary, or +8-bit binary. + +#### rhizome/bundlesecret + +Serval DNA [POST](#post) requests that take [bundle secret](#bundle-secret) +parameters use the non-standard **rhizome/bundlesecret; format=hex** content +type in the parameter's [form part](#multipart-form-data). See the [insert +request](#post-restfulrhizomeinsert) for an example. + +At present only the **hex** format is supported, and must be explicitly +specified. A missing or different format will cause a [415 Unsupported Media +Type][415] response. The correct [Content-Type][] header is: + + Content-Type: rhizome/bundlesecret; format=hex + +Hex format parameter values may only contain ASCII characters from the set +`0123456789ABCDEFabcdef`; any other character (such as a trailing newline) will +cause a [400 Bad Request][400] response. + +In future other formats may be supported, such as Base-64, 7-bit binary, or +8-bit binary. ### Rhizome HTTP response headers @@ -597,11 +654,10 @@ be [200 OK][200] and: * the [Rhizome response bundle headers](#rhizome-response-bundle-headers) give information about the found bundle, some of which is duplicated from the manifest -* the response's Content-Type is **rhizome/manifest; format=text+binarysig** -* the response's Content-Length is the size, in bytes, of the manifest with - its binary signature appended -* the response's content is the Rhizome manifest in [text+binarysig - format](#textbinarysig-manifest-format) +* the response's Content-Type is [rhizome/manifest](#rihzomemanifest) +* the response's Content-Length is the size, in bytes, of the entire manifest, + including its binary signature +* the response's content is the Rhizome [manifest](#manifest) If the **manifest is not found** in the local Rhizome store, then the response will be [404 Not Found][404] and: @@ -664,55 +720,102 @@ eg: /restful/rhizome/1702BD647D614DB72C36BD634B6870CA31040C2EEC5069AEC0C0841D0CC671BE/decrypted.bin -The responses are identical to those for [GET /restful/rhizome/BID/raw.bin](get-restfulrhizomebidrawbin), -with the following additional case: - If the **manifest and payload are both found** and the payload is **encrypted** -(the manifest's `crypt` field is 1), but the **payload secret is not known**, -then: +(the manifest's `crypt` field is 1), then the *payload secret* is determined as +follows: -* the [bundle status code](#bundle-status-code) will be 0 -* the [payload status code](#payload-status-code) will be 5 +* if the manifest has both *sender* and *recipient* [SID][]s: + + * if the recipient's identity is found (unlocked) in the keyring, then the + secret is derived from the recipient's [Serval ID](#serval-id) secret; + otherwise + * if the recipient's identity is not found in the keyring (locked or missing) + but the sender's identity is found (unlocked) in the keyring, then the + secret is derived from the sender's [Serval ID](#serval-id) secret; + otherwise + * if neither identity is found in the keyring (both are locked or missing), + then the the payload secret is unknown. + +* otherwise, the payload secret is derived directly from the [Bundle + Secret](#bundle-secret), which in turn is deduced from the `BK` [Bundle + Key](#bundle-key) field in the manifest, if present, as long as the bundle's + author can be found (unlocked) in the keyring. If there is no `BK` field, + or if no unlocked identity in the keyring can provide the necessary [Rhizome + Secret](#rhizome-secret), then the payload secret is unknown. + +The responses are identical to [GET /restful/rhizome/BID/raw.bin](get-restfulrhizomebidrawbin), +with the following variations: + +If the **payload is encrypted** and the **payload secret is known**, then +the response will be [200 OK][200] and: + +* the [bundle status code](#bundle-status-code) will be 1 +* the [payload status code](#payload-status-code) will be 0 if the decrypted + payload has zero length, otherwise 2 +* the [Rhizome response bundle headers](#rhizome-response-bundle-headers) give + information about the found bundle, some of which is duplicated from the + manifest +* the response's Content-Type is **application/octet-stream** +* the response's Content-Length is the size, in bytes, of the decrypted + payload +* the response's content is the bundle's decrypted payload + +If the **payload is encrypted** and the **payload secret is not known** then: + +* the request will fail with status [419 Authentication Timeout][419] * the [Rhizome response bundle headers](#rhizome-response-bundle-headers) give information about the found manifest -* the response's content is the [Rhizome JSON result](#rhizome-json-result) - object - -For a bundle that has a *sender* and a *recipient*, the payload secret is -determined as follows: - -* if the recipient's identity is found (unlocked) in the keyring, then the - secret is derived from the recipient's [Serval ID](#serval-id) secret; - otherwise -* if the recipient's identity is not found in the keyring (locked or missing) - but the sender's identity is found (unlocked) in the keyring, then the - secret is derived from the sender's [Serval ID](#serval-id) secret; - otherwise -* neither identity is found in the keyring (both are locked or missing), so - the payload secret is unknown. - -For all other bundles, the payload secret is derived directly from the [Bundle -Secret](#bundle-secret), whether supplied as a query parameter or deduced from -the bundle's [Bundle Key](#bundle-key). If the Bundle Secret is unknown, then -the payload secret is unknown. +* the response body is a [Rhizome JSON result](#rhizome-json-result) object, + in which: + * the [bundle status code](#bundle-status-code) is 0 + * the [payload status code](#payload-status-code) is 5 ### POST /restful/rhizome/insert -This request allows a client to add a new bundle to the Rhizome store, or -update an existing bundle in the store. This request cannot be used to create -or update [journals](#journal); use the [append](#post-restfulrhizomeappend) -request instead. +The Rhizome insert [POST][] request allows a client to add a new bundle to the +Rhizome store, or update an existing bundle in the store. This request cannot +be used to create or update [journals](#journal); use the +[append](#post-restfulrhizomeappend) request instead. -Takes the following parameters, all optional under various conditions: +This request does not accept any [query parameters][] in the *path*, but does +accept parameters using a [Content-Type][] of [multipart/form-data][], in which +each parameter has its own content type. For example: -* **bundle-id** The [Bundle ID](#bundle-id) of an existing bundle to update; - 64 hexadecimal digits. If the bundle currently exists in the Rhizome store - then a copy of its manifest is used as the basis of the new bundle, omitting - its `version`, `filesize`, `filehash` fields (which must be supplied or - inferred anew). + POST /restful/rhizome/insert HTTP/1.0 + Content-Type: multipart/form-data;boundary=OoOoOoOo + + --OoOoOoOo + Content-Disposition: form-data; name=bundle-author + Content-Type: serval/sid;format=hex + + 3DA7BA5E97DF4918DB5528450875EC9F788F0C37BC2603FD1BA7FF276C575018 + --OoOoOoOo + Content-Disposition: form-data; name=manifest + Content-Type: rhizome/manifest;format=text+binarysig + + service=file + name=helloworld.txt + + --OoOoOoOo + Content-Disposition: form-data; name=payload; filename="helloworld.txt" + Content-Type: application/octet-stream + + Hello world! + + --OoOoOoOo-- -* **bundle-author** The [SID][] of the bundle's [author](#bundle-author): +The parameters are all optional under various conditions: + +* **bundle-id** = the [Bundle ID](#bundle-id) of an existing bundle to update: + * 64 hexadecimal digits + * [Content-Type][] must be [rhizome/bid](#rhizomebid) + * if the bundle currently exists in the Rhizome store then a copy of its + manifest is used as the basis of the new bundle, omitting its `version`, + `filesize`, `filehash` fields (which must be supplied or inferred anew). + +* **bundle-author** = the [SID][] of the bundle's [author](#bundle-author): * 64 hexadecimal digits; + * [Content-Type][] must be [serval/sid][] * the bundle author sets (or removes) the bundle's `BK` field, overriding any `BK` field in the partial manifest supplied in the *manifest* parameter or in the existing bundle nominated by the *bundle-id* @@ -723,22 +826,24 @@ Takes the following parameters, all optional under various conditions: request fails with status [400 Bad Request][400] and the message ‘Spurious "bundle-id" form part’. -* **bundle-secret** The [Bundle Secret](#bundle-secret); 64 hexadecimal - digits. This is needed in order to create a bundle with a specific [Bundle - ID](#bundle-id) (supplied in the **manifest** parameter), or to update an - existing bundle that is anonymous or the author is not a currently unlocked - identity in the keyring. +* **bundle-secret** = the [Bundle Secret](#bundle-secret): + * 64 hexadecimal digits + * [Content-Type][] must be [rhizome/bundlesecret](#rhizomebundlesecret) + * is needed in order to create a bundle with a specific [Bundle ID](#bundle-id) + (supplied in the *manifest* parameter), or to update an existing bundle + that is anonymous or the author is not a currently unlocked identity in + the keyring. -* **manifest** A partial, unsigned manifest in [text+binarysig - format](#textbinarysig-manifest-format), with a correct *Content-Type* - header. The fields in this manifest are used to form the new bundle's - manifest, overwriting the fields of any existing manifest specified by the - *bundle-id* parameter, if given. +* **manifest** = a partial, unsigned [manifest](#manifest): + * [Content-Type][] must be [rhizome/manifest](#rhizomemanifest) + * the fields in this manifest are used to form the new bundle's manifest, + overwriting the fields of any existing manifest specified by the + *bundle-id* parameter, if given. -* **payload** The content of the new bundle's payload: - * the form part's *Content-Type* header is currently ignored, but in future - it may be used to determine the default values of some manifest fields; - * this parameter must occur after the *manifest* parameter, otherwise the +* **payload** = the content of the new bundle's payload: + * [Content-Type][] is currently ignored, but in future it may be used to + determine the default values of some manifest fields; + * this parameter must come after the *manifest* parameter, otherwise the request fails with status [400 Bad Request][400] and the message ‘Missing "manifest" form part’; * the *payload* parameter must not be supplied if the `filesize` field in @@ -746,27 +851,26 @@ Takes the following parameters, all optional under various conditions: The insertion logic proceeds in the following steps: -1. If the partial manifest supplied in the *manifest* parameter is malformed - (syntax error) or contains a core field with an invalid value, then the - request fails with status [422 Unprocessable Entity][422] and the [bundle - status code](#bundle-status-code) for “invalid”. - -2. If a *bundle-id* parameter was supplied and the given bundle exists in the +1. If a *bundle-id* parameter was supplied and the given bundle exists in the Rhizome store, then the new bundle's manifest is initialised by copying all the fields from the existing manifest. -3. If a partial manifest was supplied in the *manifest* parameter, then its +2. If a partial manifest was supplied in the *manifest* parameter, then its fields are copied into the new manifest, overwriting any that were copied - in step 2. + in step 1. If the partial manifest is malformed (syntax error) or contains + a core field with an invalid value, then the request fails with status [422 + Unprocessable Entity][422] and the [bundle status + code](#bundle-status-code) for “invalid”. -4. If the `tail` field is present in the new manifest then the new bundle is a +3. If the `tail` field is present in the new manifest then the new bundle is a [journal](#journal), so the request fails with status [422 Unprocessable Entity][422] and the [bundle status code](#bundle-status-code) for “invalid”. Journals can only be created and updated using the [append](#post-restfulrhizomeappend) request. -5. If the *bundle-secret* parameter was supplied, then a public key ([Bundle - ID](#bundle-id)) is derived from the [Bundle Secret](#bundle-secret), and: +4. If the *bundle-secret* parameter was supplied, then a public key + ([Bundle ID](#bundle-id)) is derived from the [Bundle Secret](#bundle-secret), + and: * if the new manifest has no `id` field, then the `id` field is set to the derived public key; @@ -778,16 +882,15 @@ The insertion logic proceeds in the following steps: Otherwise, if no *bundle-secret* parameter was supplied: - * if the new manifest has no `id` field, then a new [Bundle - Secret](#bundle-secret) is generated randomly, the [Bundle - ID](#bundle-id) is derived from the new Bundle Secret, and the `id` field - set to that Bundle ID; + * if the new manifest has no `id` field, then a new [Bundle Secret](#bundle-secret) + is generated randomly, the [Bundle ID](#bundle-id) is derived from the + new Bundle Secret, and the `id` field set to that Bundle ID; - * if the new manifest already has an `id` field but no `BK` field ([Bundle - Key](#bundle-key)) (ie, the bundle is *anonymous*), then the [Bundle - Secret](#bundle-secret) cannot be discovered, so the request fails with - status [419 Authentication Timeout][419] and the [bundle status - code](#bundle-status-code) for “readonly”. + * if the new manifest already has an `id` field but no `BK` field + ([Bundle Key](#bundle-key)) (ie, the bundle is *anonymous*), then the + [Bundle Secret](#bundle-secret) cannot be discovered, so the request + fails with status [419 Authentication Timeout][419] and the [bundle + status code](#bundle-status-code) for “readonly”. * otherwise, if the *bundle-author* parameter was given, then that [SID][] is looked up in the keyring. If the identity is found, then the [Bundle @@ -799,17 +902,21 @@ The insertion logic proceeds in the following steps: Authentication Timeout][419] and the [bundle status code](#bundle-status-code) for “readonly”. - * otherwise, if no *bundle-author* parameter was given, then the keyring is - searched for an identity whose [Rhizome Secret](#rhizome-secret) combined - with the `BK` field ([Bundle Key](#bundle-key)) produces a [Bundle - Secret](#bundle-secret) whose derived [Bundle ID](#bundle-id) matches the - `id` field. The search starts with the identity given by the `sender` - field, if present. If none is found, then the request fails with status - [419 Authentication Timeout][419] and the [bundle status - code](#bundle-status-code) for “readonly”, otherwise the author is - deduced to be the found identity. + * otherwise, if no *bundle-author* parameter was given but the manifest has + a `BK` field, then the keyring is searched for an identity whose [Rhizome + Secret](#rhizome-secret) combined with the `BK` field ([Bundle + Key](#bundle-key)) produces a [Bundle Secret](#bundle-secret) whose + derived [Bundle ID](#bundle-id) matches the `id` field. The search + starts with the identity given by the `sender` field, if present. If + none is found, then the request fails with status [419 Authentication + Timeout][419] and the [bundle status code](#bundle-status-code) for + “readonly”, otherwise the author is deduced to be the found identity. -6. If the *bundle-author* parameter was given and step 5 set the `id` field + * otherwise, if no *bundle-author* parameter was given and the manifest has + no `BK` field, then an *anonymous* bundle is produced, ie, with no `BK` + key. + +5. If the *bundle-author* parameter was given and step 4 set the `id` field (either derived from the *bundle-secret* parameter or randomly generated), then the *bundle-author* [SID][] is looked up in the keyring. If not found, then the request fails with status [419 Authentication Timeout][419] @@ -817,14 +924,14 @@ The insertion logic proceeds in the following steps: found, then the author's [Rhizome Secret](#rhizome-secret) is used to calculate the [Bundle Key](#bundle-key) and set the `BK` field. -7. The following fields are initialised if they are missing: +6. The following fields are initialised if they are missing: * `service` to the value `file` * `version` to the current [Unix time][] in milliseconds since the epoch * `date` to the current [Unix time][] in milliseconds since the epoch * `crypt` to `1` if the `sender` and `recipient` fields are both set -8. If the *payload* parameter is given and is non-empty, then its value is +7. If the *payload* parameter is given and is non-empty, then its value is stored in the store, and its size and [SHA-512][] digest computed. If the manifest is missing either or both of the `filesize` and `filehash` fields, then the missing ones are filled in from the computed values. If the @@ -833,7 +940,7 @@ The insertion logic proceeds in the following steps: Entity][422] and the [bundle status code](#bundle-status-code) for “inconsistent”. -9. The manifest is *validated* to ensure that: +8. The manifest is *validated* to ensure that: * the `id` field is present * the `version` field is present @@ -850,7 +957,7 @@ The insertion logic proceeds in the following steps: Entity][422] and the [bundle status code](#bundle-status-code) for “invalid”. -10. If step 5 set the `id` field (either derived from the *bundle-secret* +9. If step 4 set the `id` field (either derived from the *bundle-secret* parameter or randomly generated) and the bundle is a *duplicate* of a bundle that is already in the store, then the request finishes with status [200 OK][200] and the [bundle status code](#bundle-status-code) for @@ -862,15 +969,15 @@ The insertion logic proceeds in the following steps: * the same `sender` field, and * the same `recipient` field. -11. The manifest is signed using the [Bundle Secret](#bundle-secret), and the +10. The manifest is signed using the [Bundle Secret](#bundle-secret), and the signature appended to the manifest after a single ASCII NUL (0) separator byte. If the result exceeds the maximum manifest size (8 KiB) then the request fails with status [422 Unprocessable Entity][422] and the [bundle status code](#bundle-status-code) for “manifest too big”. -12. If the Rhizome store already contains a manifest with the same [Bundle - ID](#bundle-id), then its version is compared with the new manifest's - version. +11. If the Rhizome store already contains a manifest with the same + [Bundle ID](#bundle-id), then its version is compared with the new + manifest's version. * If they have the same version, then the new manifest is not stored, and the request returns status [200 OK][200] and the [bundle status @@ -881,7 +988,7 @@ The insertion logic proceeds in the following steps: Accepted][202] and the [bundle status code](#bundle-status-code) for “old”. -13. The new manifest is stored in the Rhizome store, replacing any existing +12. The new manifest is stored in the Rhizome store, replacing any existing manifest with the same [Bundle ID](#bundle-id). The request returns status [201 Created][201] and the [bundle status code](#bundle-status-code) for “new”. @@ -895,34 +1002,35 @@ identical in all respects except as follows: The steps of the insertion logic have these variations: -1. The validity checks on any partial manifest given in the *manifest* - parameter will also fail if the partial manifest contains a `version`, - `filesize` or `filehash` field. - -2. If the *bundle-id* parameter specifies an existing manifest, then the +1. If the *bundle-id* parameter specifies an existing manifest, then the `version`, `filesize` and `filehash` fields are not copied from the existing manifest to the new manifest. -3. After the partial manifest has been copied into the new manifest, if the - *bundle-id* parameter was not given or specified a bundle that was not - found in the store (step 2), then the `filesize` and `tail` fields are - initialised to zero (0) if they are missing. +2. The validity checks on any partial manifest given in the *manifest* + parameter will also fail if the partial manifest contains a `version`, + `filesize` or `filehash` field. After the partial manifest has been copied + into the new manifest, if the *bundle-id* parameter was not given or + specified a bundle that was not found in the store (step 1), then the + `filesize` and `tail` fields are initialised to zero (0) if they are + missing. -4. If the `tail` field is missing from the new manifest then the bundle is not +3. If the `tail` field is missing from the new manifest then the bundle is not a [journal](#journal), so the request fails with status [422 Unprocessable Entity][422] and the [bundle status code](#bundle-status-code) for “invalid”. +4. No change. + 5. No change. 6. No change. -7. No change. - -8. After the payload has been stored, the `filesize` and `filehash` fields are +7. After the payload has been stored, the `filesize` and `filehash` fields are always set, overriding any that were already present. Also, the `version` is always set to `tail + filesize`. +8. No change. + 9. No change. 10. No change. @@ -931,28 +1039,34 @@ The steps of the insertion logic have these variations: 12. No change. -13. No change. - ### POST /restful/rhizome/import -This request allows the client to store a valid manifest and payload that have been -obtained through some other means. +The import [POST](#post) request allows the client to store a valid bundle +(manifest and payload) in the store that may have been obtained through some +other means, such as exporting from another store using the [manifest +request](#get-restfulrhizomebidrhm) and the [raw payload +request](#get-restfulrhizomebidrawbin). -* **manifest** A signed manifest in [text+binarysig format](#textbinarysig-manifest-format), - with a correct *Content-Type* header. +This request accepts the following parameters using a [Content-Type][] of +[multipart/form-data][], in which each parameter has its own content type: -* **payload** The content of the bundle's payload: - * the form part's *Content-Type* header is currently ignored, but in future - it may be used to determine the default values of some manifest fields; - * this parameter must occur after the *manifest* parameter, otherwise the +* **manifest** (required) = a signed [manifest](#manifest): + * [Content-Type][] must be [rhizome/manifest](#rhizomemanifest) + +* **payload** = the content of the bundle's payload: + * [Content-Type][] is currently ignored, but in future it may be used to + determine the default values of some manifest fields; + * this parameter must come after the *manifest* parameter, otherwise the request fails with status [400 Bad Request][400] and the message ‘Missing "manifest" form part’; * the *payload* parameter must not be supplied if the `filesize` field in - the *manifest* parameter is zero. + the *manifest* parameter is zero; + * if the bundle is encrypted (the manifest `crypt` field is 1) then the + payload must be in encrypted form, not plain text. ----- -**Copyright 2015 Serval Project Inc.** +**Copyright 2015-2017 Serval Project Inc.** ![CC-BY-4.0](./cc-by-4.0.png) Available under the [Creative Commons Attribution 4.0 International licence][CC BY 4.0]. @@ -960,9 +1074,12 @@ Available under the [Creative Commons Attribution 4.0 International licence][CC [Serval Project]: http://www.servalproject.org/ [CC BY 4.0]: ../LICENSE-DOCUMENTATION.md [Rhizome]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:rhizome +[BID]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:bid [Serval Mesh network]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:mesh_network [Serval DNA]: ../README.md [REST-API]: ./REST-API.md +[form part]: ./REST-API.md#multipart-form-data +[POST]: ./REST-API.md#post [JSON result]: ./REST-API.md#json-result [store and forward]: https://en.wikipedia.org/wiki/Store_and_forward [SID]: ./REST-API-Keyring.md#serval-id @@ -970,13 +1087,19 @@ Available under the [Creative Commons Attribution 4.0 International licence][CC [MeshMS]: ./REST-API-MeshMS.md [MeshMS conversations]: ./REST-API-MeshMS.md#conversation [JSON table]: ./REST-API.md#json-table +[Curve25519]: https://en.wikipedia.org/wiki/Curve25519 [Unix time]: https://en.wikipedia.org/wiki/Unix_time [Y2038 problem]: https://en.wikipedia.org/wiki/Year_2038_problem +[query parameters]: https://en.wikipedia.org/wiki/Query_string +[Content-Type]: ./REST-API.md#content-type-header +[multipart/form-data]: ./REST-API.md#multipartform-data +[serval/sid]: ./REST-API.md#servalsid [200]: ./REST-API.md#200-ok [201]: ./REST-API.md#201-created [202]: ./REST-API.md#202-accepted [400]: ./REST-API.md#400-bad-request [404]: ./REST-API.md#404-not-found +[415]: ./REST-API.md#415-unsupported-media-type [419]: ./REST-API.md#419-authentication-timeout [422]: ./REST-API.md#422-unprocessable-entity [423]: ./REST-API.md#423-locked diff --git a/doc/REST-API.md b/doc/REST-API.md index 8821e56a..a5baa213 100644 --- a/doc/REST-API.md +++ b/doc/REST-API.md @@ -1,6 +1,6 @@ REST API ======== -[Serval Project][], February 2016 +[Serval Project][], October 2017 Introduction ------------ @@ -27,14 +27,16 @@ gives applications access to the network through two main classes of [API][]: This document describes the features in common to all the [HTTP REST][] APIs. -### Protocol and port +Protocol and port +----------------- The Serval DNA [HTTP REST][] API is an [HTTP 1.0][] server that only accepts requests on the loopback interface (IPv4 address 127.0.0.1), TCP port 4110. It rejects requests that do not originate on the local host, by replying [403 Forbidden](#403-forbidden). -### Security +Security +-------- The REST API uses plain HTTP *without* encryption. REST requests and responses are not carried over any physical network link, only local (“logical”) links @@ -50,7 +52,8 @@ Serval DNA and its clients, much more effective than intercepting their communications, so encrypting client-server communications would offer no protection whatsoever. -### Authentication +Authentication +-------------- Clients of the HTTP REST API must authenticate themselves using [Basic Authentication][]. This narrows the window for opportunistic attacks on the @@ -77,11 +80,12 @@ PASSWORD is a cleartext secret, so the Serval DNA configuration file must be protected from unauthorised access or modification by other apps. That makes this mechanism unsuitable for general use. -### Request +Request +------- An HTTP REST request is a normal [HTTP 1.0][] [GET](#get) or [POST](#post): -#### GET +### GET A **GET** request consists of an initial "GET" line containing the *path* and *HTTP version*, followed by zero or more header lines, followed by a blank @@ -94,98 +98,153 @@ For example: Accept: */* -GET requests only accept parameters as [query parameters][] in the *path*. +GET requests only accept parameters as [percent encoded][] [query parameters][] +in the *path*. [query parameters]: http://tools.ietf.org/html/rfc3986#section-3.4 -#### POST +### POST -A **POST** request has a similar structure to a GET request except that the first word -of the first line is "POST". +A **POST** request has a similar structure to a GET request except that the +first word of the first line is "POST", and there may be a body of content +following the blank line that ends the header. -POST requests accept parameters as [query parameters][] in the *path* and also -as a request body with a [Content-Type](#request-content-type) of -[multipart/form-data][]. These two kinds of parameters are not exclusive; a -request may contain a mixture of both. +POST requests accept parameters as [percent encoded][] [query parameters][] in +the *path* and also as a request body with a [Content-Type](#content-type-header) +of [multipart/form-data][]. These two kinds of parameters are not exclusive; a +POST request may contain a mixture of both. -A POST request may also include a [Content-Length](#request-content-length) -or [Transfer-Encoding](#request-transfer-encoding) header; -a [Content-Type](#request-content-type); and an [Expect](#request-expect) header as described below. +A POST request may also include the following headers as described below: +* [Content-Length](#content-length-header) +* [Content-Type](#content-type-header) +* [Range](#range-header) +* [Transfer-Encoding](#transfer-encoding-header) +* [Expect](#expect-header) -#### Request Transfer-Encoding +### Content-Length header -In a request, a **Transfer-Encoding** header of "chunked", indicates that the -client can generate and transmit the request body without pre-calculating the -final length. No other transfer encodings are currently supported. +In a [POST](#post) request, the **Content-Length** header gives the exact +number of bytes (octets) in the request's body, which must be correct. Serval +DNA will not process a request until it receives Content-Length bytes, so if +Content-Length is too large, the request will suspend and eventually time out. +Serval DNA will ignore any bytes received after it has read Content-Length +bytes, so if Content-Length is too small, the request body will be malformed. -Each chunk is generated as the size of the chunk in hex, followed -by '\r\n', followed by the request bytes, followed by another '\r\n'. +Serval DNA treats a missing Content-Length header the same as a Content-Length +of zero; it will not attempt to read the request body so any body content will +be ignored if sent. -The end of the input is indicated with a chunk of zero length. +### Content-Type header -#### Request Expect +In a [POST](#post) request, the **Content-Type** header gives the [Internet +Media Type][] of the body. -The presense of an **Expect** header of "100-Continue" indicates that the server -should respond with an intermediate response of "HTTP/1.1 100 Continue" before the -client begins to transmit a request body. - -If for any reason the server determines that the request body is not needed, or -the request is invalid, the server will generate the response immediately without -reading the contents of the request body. - -#### Request Content-Length - -In a request, the **Content-Length** header gives the exact number of bytes -(octets) in the request's body, which must be correct. Serval DNA will not -process a request until it receives Content-Length bytes, so if Content-Length -is too large, the request will suspend and eventually time out. Serval DNA -will ignore any bytes received after it has read Content-Length bytes, so if -Content-Length is too small, the request body will be malformed. - -A missing Content-Length header will be treated the same as a Content-Length of zero. - -#### Request Content-Type - -In a request, the **Content-Type** header gives the [Internet Media Type][] of -the body. Serval DNA currently supports the following media types in requests: - -* **[multipart/form-data][]** is used to send parameters in [POST](#post) - requests. The **boundary** parameter must specify a string that does not - occur anywhere within the content of any form part. - -* **text/plain; charset=utf-8** is used for [MeshMS][] message form parts. - The only supported charset is utf-8; a missing or different charset will - cause a [415 Unsupported Media Type](#415-unsupported-media-type) response. - -* **rhizome/manifest; format=text+binarysig** is used for [Rhizome - manifest][]s in [text+binarysig format][]. - -A missing Content-Type header in a `POST` request will cause a [400 Bad +A missing Content-Type header in a [POST](#post) request will cause a [400 Bad Request](#400-bad-request) response. An unsupported content type will cause a [415 Unsupported Media Type](#415-unsupported-media-type) response. -The following media types are *not supported*: +#### multipart/form-data -* [application/x-www-form-urlencoded][] is commonly used to send parameters in - [POST](#post) requests, and is the predecessor web standard to - [multipart/form-data][]. It has the benefit of being simpler than - [multipart/form-data][] for requests that take short, mainly textual - parameters, but is very inefficient for encoding large binary values and - does not provide any means to associate metadata such as content-type and - encoding with individual parameters. In future, some REST API requests may - support [application/x-www-form-urlencoded][]. +Serval DNA [POST](#post) requests that take their parameters in the request +body accept the **[multipart/form-data][]** content type. -[multipart/form-data]: https://www.ietf.org/rfc/rfc2388.txt -[application/x-www-form-urlencoded]: https://tools.ietf.org/html/rfc1866#section-8.2.1 +The multipart **boundary** must specify a string consisting of at most 70 ASCII +characters that does not occur anywhere within the content of any part, and is +used to delimit the parts of the request body. Typically this string is +generated using a large random number encoded into printable ASCII, eg: -#### Request Range + Content-Type: multipart/form-data;boundary=081d31d4c3d23014 -[HTTP 1.1 Range][] retrieval is partially supported. In a request, the -**Range** header gives the start and end, in byte offsets, of the resource to -be returned. The server may respond with exactly the range requested, in which -case the response status code will be [206 Partial Content](#206-partial-content), -or it may ignore the Range header and respond with the entire requested -resource. +Each form part introduces one named parameter, and consists of its own header +section, followed by a blank line (a CR-LF immediately following the CR-LF that +terminates the last header), followed by a body that contains the parameter's +content, followed by a CR-LF. Every part must have a [Content-Disposition][] +header that gives the parameter's name, and may have an optional +[Content-Type][] header. + +The following example passes two parameters to a request, called `first` and +`second`, the first is a single line of plain text terminated by a LF, the +second is an HTML document with no terminating LF: + + POST /an/example/request HTTP/1.0 + Content-Type: multipart/form-data;boundary=4aceafdc5cc7295d + + --4aceafdc5cc7295d + Content-Disposition: form-data; name=first + Content-Type: text/plain; charset=utf-8 + + Hello, world! + + --4aceafdc5cc7295d + Content-Disposition: form-data; name=second; filename="hello_world.html" + Content-Type: text/html; charset=utf-8 + + + + + + The second parameter + + +

Hello world!

+ + + --4aceafdc5cc7295d-- + +#### application/x-www-form-urlencoded + +Serval DNA [POST](#post) requests *do not support* the +[application/x-www-form-urlencoded][] content type, although some may do so in +future. + +[application/x-www-form-urlencoded][] is the predecessor web standard to +[multipart/form-data](#multipart-form-data). It has the benefit of being +simpler for requests that take short, mainly textual parameters, but is very +inefficient for encoding large binary values and does not provide any means to +associate metadata such as content-type, encoding and file name with individual +parameters. + +#### text/plain + +Serval DNA [POST](#post) requests that take plain text parameters use the +**text/plain** content type in the parameter's [form part](#multipart-form-data). +See the [MeshMS send request][] for an example. + +The only supported charset is **utf-8**; a missing or different charset will +cause a [415 Unsupported Media Type](#415-unsupported-media-type) response. +The correct [Content-Type](#content-type-header) header is: + + Content-Type: text/plain; charset=utf-8 + +#### serval/sid + +Serval DNA [POST](#post) requests that take [SID][] parameters use the +non-standard **serval/sid** content type within the parameter's [form +part](#multipart-form-data). See the [Rhizome insert request][] for an +example. + +At present only the **hex** format is supported, and must be explicitly +specified. A missing or different format will cause a [415 Unsupported Media +Type](#415-unsupported-media-type). The correct [Content-Type](#content-type-header) +header is: + + Content-Type: serval/sid; format=hex + +Hex format parameter values may only contain ASCII characters from the set +`0123456789ABCDEFabcdef`; any other character (such as a trailing newline) will +cause a [400 Bad Request](#400-bad-request) response. + +In future other formats may be supported, such as Base-64, 7-bit binary, or +8-bit binary. + +### Range header + +[HTTP 1.1 Range][] retrieval is partially supported. In a [POST](#post) +request, the **Range** header gives the start and end, in byte offsets, of the +resource to be returned. The server may respond with exactly the range +requested, in which case the response status code will be [206 Partial +Content](#206-partial-content), or it may ignore the Range header and respond +with the entire requested resource. For example, the following header asks that the server omit the first 64 bytes and send only the next 64 bytes (note that ranges are inclusive of their end @@ -200,9 +259,33 @@ response may be the entire content or [501 Not Implemented](#501-not-implemented [HTTP 1.1 Range]: http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35 -### Responses +### Transfer-Encoding header -An HTTP REST response is a normal [HTTP 1.0][] response consisting of a header +In a [POST](#post) request, a **Transfer-Encoding** header of "chunked", +indicates that the client can generate and transmit the request body without +pre-calculating the final length. No other transfer encodings are currently +supported. + +Each chunk is generated as the size of the chunk in hex, followed by CR-LF, +followed by the request bytes, followed by another CR-LF. + +The end of the input is indicated with a chunk of zero length. + +### Expect header + +In a [POST](#post) request, the presense of an **Expect** header of +"100-Continue" indicates that the server should respond with an intermediate +response of "HTTP/1.1 100 Continue" before the client begins to transmit a +request body. + +If for any reason the server determines that the request body is not needed, or +the request is invalid, the server will generate the response immediately without +reading the contents of the request body. + +Responses +--------- + +An HTTP REST *response* is a normal [HTTP 1.0][] response consisting of a header block, a blank line, and an optional body, for example: As usual, all lines are terminated by an ASCII CR-LF sequence. For example: @@ -224,14 +307,15 @@ return to the client; for example, [Rhizome response headers](#rhizome-response- [application/json]: https://tools.ietf.org/html/rfc4627 -### Response status code +Response status code +-------------------- The HTTP REST API response uses the [HTTP status code][] to indicate the outcome of the request as follows: [HTTP status code]: http://www.w3.org/Protocols/HTTP/1.0/spec.html#Status-Codes -#### 200 OK +### 200 OK The operation was successful and no new entity was created. Most requests return this code to indicate success. Requests that create a new entity only @@ -245,13 +329,13 @@ entity twice yields the same state as creating it once. This is an important property for a purely distributed network that has no central arbiter to enforce sequencing of operations.) -#### 201 Created +### 201 Created The operation was successful and the entity was created. This code is only returned by requests that create new entities, in the case that the entity did not exist beforehand and has been created successfully. -#### 202 Accepted +### 202 Accepted The operation was successful but the entity was not created. This code is only returned by requests that create new entities, in the case that the request was @@ -261,23 +345,23 @@ this code when inserting a bundle to a full Rhizome store if the new bundle's rank falls below all other bundles, so the new bundle itself would be evicted to make room. -#### 206 Partial Content +### 206 Partial Content The operation was successful and the response contains part of the requested content. This code is only returned by requests that fetch an entity (the fetched entity forms the body of the response) if the request supplied a -[Range](#request-range) header that specified less than the entire entity. +[Range](#range-header) header that specified less than the entire entity. -#### 400 Bad Request +### 400 Bad Request The HTTP request was malformed, and should not be repeated without modifications. This could be for several reasons: - invalid syntax in the request header block -- a `POST` request MIME part is missing, duplicated or out of order -- a `POST` request was given an unsupported MIME part -- a `POST` request MIME part has missing or malformed content +- a [POST](#post) request parameter is missing, duplicated or out of order +- a [POST](#post) request was given an unsupported parameter +- a [POST](#post) request parameter has missing or malformed content -#### 401 Unauthorized +### 401 Unauthorized The request did not supply an "Authorization" header with a recognised credential. This response contains a "WWW-Authenticate" header that describes @@ -293,12 +377,12 @@ the missing credential: "http_status_message": "Unauthorized" } -#### 403 Forbidden +### 403 Forbidden The request failed because the server does not accept requests from the originating host. -#### 404 Not Found +### 404 Not Found The request failed because the [HTTP request URI][] does not exist. This could be for several reasons: @@ -311,7 +395,7 @@ be for several reasons: [HTTP request URI]: http://www.w3.org/Protocols/HTTP/1.0/spec.html#Request-URI -#### 405 Method Not Allowed +### 405 Method Not Allowed The request failed because the [HTTP request method][] is not supported for the given path. Usually this means that a [GET](#get) request was attempted on a @@ -319,23 +403,23 @@ path that only supports [POST](#post), or vice versa. [HTTP request method]: http://www.w3.org/Protocols/HTTP/1.0/spec.html#Method -#### 411 Length Required +### 411 Length Required -A `POST` request did not supply either a [Content-Length](#request-content-length) -or [Transfer-Encoding](#request-transfer-encoding) header. +A [POST](#post) request did not supply either a [Content-Length](#content-length-header) +or [Transfer-Encoding](#transfer-encoding-header) header. -#### 414 Request-URI Too Long +### 414 Request-URI Too Long The request failed because the [HTTP request URI][] was too long. The server persists the path and a few other pieces of the request in a fixed size request buffer, and this response is triggered if the collective size of these does not leave enough buffer for receiving the remainder of the request. -#### 415 Unsupported Media Type +### 415 Unsupported Media Type -A `POST` request failed because of an unsupported content type, which could be -for several reasons: -- the request's [Content-Type](#request-content-type) header specified an +A [POST](#post) request failed because of an unsupported content type, which +could be for several reasons: +- the request's [Content-Type](#content-type-header) header specified an unsupported media type - a part of a [multipart/form-data][] request body has: - a missing `Content-Disposition` header, or @@ -343,27 +427,27 @@ for several reasons: - a missing or unsupported `Content-Type` header (including a missing or unsupported `charset` parameter) -#### 416 Requested Range Not Satisfiable +### 416 Requested Range Not Satisfiable -The [Range](#request-range) header specified a range whose start position falls +The [Range](#range-header) header specified a range whose start position falls outside the size of the requested entity. -#### 419 Authentication Timeout +### 419 Authentication Timeout The request failed because the server does not possess and cannot derive the necessary cryptographic secret or credential. For example, updating a [Rhizome bundle][] without providing the [bundle secret][]. This code is not part of the HTTP standard. -#### 422 Unprocessable Entity +### 422 Unprocessable Entity -A `POST` request supplied data that was inconsistent or violates semantic +A [POST](#post) request supplied data that was inconsistent or violates semantic constraints, so cannot be processed. For example, the [Rhizome insert](./REST-API-Rhizome.md#post-restfulrhizomeinsert) operation responds with 422 if the manifest *filesize* and *filehash* fields do not match the supplied payload. -#### 423 Locked +### 423 Locked The request cannot be performed because a necessary resource is busy for reasons outside the control of the requester and server. @@ -375,7 +459,7 @@ from directly accessing the Rhizome database. Once these improvements are done, this code should no longer occur except during unusual testing and development situations. -#### 429 Too Many Requests +### 429 Too Many Requests The request cannot be performed because a necessary resource is temporarily unavailable due to a high volume of concurrent requests. @@ -390,7 +474,7 @@ resources. For example, if [Serval DNA][] is ever limited to service only a few HTTP requests at a time, then this code will be returned to new requests that would exceed the limit. -#### 431 Request Header Fields Too Large +### 431 Request Header Fields Too Large The request header block was too long. @@ -399,7 +483,7 @@ buffer memory for each [request](#request), and the HTTP server read each header line entirely into that buffer before parsing it. If a single header exceeded the size of this buffer, then the 431 response was returned. -#### 500 Internal Server Error +### 500 Internal Server Error The request failed because of an internal error in [Serval DNA][], not an error in the request itself. This could be for several reasons: @@ -412,14 +496,15 @@ re-tried, but in general they will persist because the cause is not transient. Temporary failures that can be resolved by re-trying the request are generally indicated by other status codes, such as [423 Locked](#423-locked). -#### 501 Not Implemented +### 501 Not Implemented The requested operation is valid but not yet implemented. This is used for the following cases: -- a request [Range](#request-range) header specifies a multi range +- a [POST](#post) request [Range](#range-header) header specifies a multi range -#### Cross-Origin Resource Sharing (CORS) +Cross-Origin Resource Sharing (CORS) +------------------------------------ To support client-side JavaScript applications, Serval DNA has a limited implementation of [Cross-Origin Resource Sharing][CORS]. If a request contains @@ -445,7 +530,8 @@ Serval DNA will respond: [CORS]: http://www.w3.org/TR/cors/ -#### JSON result +JSON result +----------- All responses that convey no special content return the following *JSON result* object: @@ -468,7 +554,8 @@ found”, “Payload not found”, etc. Some responses augment the *JSON result* object with extra fields; for example, [Rhizome JSON result][] and [Keyring JSON result][]. -### JSON table +JSON table +---------- Many HTTP REST responses that return a list of regular objects (eg, [GET /restful/rhizome/bundlelist.json](./REST-API-Rhizome.md#get-restfulrhizomebundlelistjson)) @@ -517,7 +604,7 @@ expression to perform the transformation: ] ----- -**Copyright 2015 Serval Project Inc.** +**Copyright 2015-2017 Serval Project Inc.** ![CC-BY-4.0](./cc-by-4.0.png) Available under the [Creative Commons Attribution 4.0 International licence][CC BY 4.0]. @@ -546,9 +633,15 @@ Available under the [Creative Commons Attribution 4.0 International licence][CC [Rhizome bundle]: ./REST-API-Rhizome.md#bundle [Rhizome manifest]: ./REST-API-Rhizome.md#manifest [Rhizome JSON result]: ./REST-API-Rhizome.md#rhizome-json-result +[Rhizome insert request]: ./REST-API-Rhizome.md#post-restfulrhizomeinsert +[MeshMS send request]: ./REST-API-MeshMS.md#post-restfulmeshmssendersidrecipientsidsendmessage [Keyring JSON result]: ./REST-API-Keyring.md#keyring-json-result [bundle secret]: ./REST-API-Rhizome.md#bundle-secret -[text+binarysig format]: ./REST-API-Rhizome.md#textbinarysig-manifest-format +[multipart/form-data]: https://www.ietf.org/rfc/rfc7578 +[Content-Disposition]: https://tools.ietf.org/html/rfc7578#section-4.2 +[Content-Type]: https://tools.ietf.org/html/rfc7578#section-4.4 +[application/x-www-form-urlencoded]: https://tools.ietf.org/html/rfc1866#section-8.2.1 +[percent encoded]: https://en.wikipedia.org/wiki/Percent-encoding [JSON]: https://en.wikipedia.org/wiki/JSON [UTF-8]: https://en.wikipedia.org/wiki/UTF-8 [jq(1)]: https://stedolan.github.io/jq/