From 7edf1a912e0c1760a1e8c2bff5afeb52fb844405 Mon Sep 17 00:00:00 2001 From: Andrew Bettison Date: Wed, 22 Nov 2017 13:55:35 +1030 Subject: [PATCH] Improve MDP technical documentation The MDP technical document can now replace the Serval Developer wiki MDP page. --- doc/Mesh-Datagram-Protocol.md | 149 +++++++++++++++++++++++++++------- doc/REST-API-Keyring.md | 44 ++++++++-- 2 files changed, 156 insertions(+), 37 deletions(-) diff --git a/doc/Mesh-Datagram-Protocol.md b/doc/Mesh-Datagram-Protocol.md index 4755fd50..65783642 100644 --- a/doc/Mesh-Datagram-Protocol.md +++ b/doc/Mesh-Datagram-Protocol.md @@ -1,21 +1,36 @@ Mesh Datagram Protocol (MDP) ============================ -[Serval Project], March 2016 +[Serval Project], November 2017 -The [Mesh Datagram Protocol][MDP] is a [layer 3][] network protocol developed -for the [Serval mesh network][], with characteristics that make it particularly -suitable for use in Ad Hoc wireless networks, which can suffer high levels of -packet loss due to weak signal, interference and congestion. +The [Mesh Datagram Protocol][MDP] is a [layer 3][] [datagram][] network +protocol developed for the [Serval mesh network][], with characteristics that +make it particularly suitable for use in Ad Hoc wireless networks, which can +suffer high levels of packet loss due to weak signal, interference and +congestion. -MDP carries [messages](#mdp-message) from [sender](#sender) to [recipient](#recipient) -[node](#node)s, or [broadcasts](#broadcast) to all nodes, guaranteeing only that -message contents will be correct if delivered, like a traditional [datagram][] -service. MDP is similar to [UDP][] in this respect, but it uses per-[link](#link) -retransmission and adaptive link-state routing to boost packet delivery -rates, which largely immunises it from the cumulative packet loss effect -typical of multi-hop wireless networks. This means that, unlike [Internet -protocols][] its end-to-end, multi-hop packet delivery rate remains usefully -high despite adverse and changing network conditions. +MDP carries [messages](#mdp-message) from [sender](#sender) to +[recipient](#recipient) [node](#node)s, or [broadcasts](#broadcast) to all +nodes. MDP guarantees that message contents will be correct if delivered, but +does not guarantee one delivery (messages may be lost or delivered more than +once), arrival time, or message order. + +MDP can be carried over any wireless or wired data link, whether a shared +medium (eg, [CSMA/CA][] used in [Wi-Fi][]) or a dedicated medium (eg, [AX.25 +packet radio][], [serial cable][]). + +MDP is similar to [UDP][], but it uses per-[link](#link) retransmission and +adaptive link-state routing to boost packet delivery rates, which largely +immunises it from the cumulative packet loss effect typical of multi-hop +wireless networks. To carry a packet over **N** hops, where each hop has a +probability **P** of dropping a packet due to interference or collision, the +end-to-end loss is **1 − pow(1−P, N)**. For example, given a per-hop packet loss +of 10%, a five hop route has a net packet loss of 41%, and a ten hop route +has 75% packet loss. + +The MDP retransmission scheme reduces but does not eliminate packet loss, and +sometimes produces duplicate packets. However, it can squeeze useful packet +delivery rates from a high-loss route that would be practically useless with +[Internet protocols][], which rely on end-to-end retransmission. Basic concepts -------------- @@ -48,8 +63,8 @@ the packet does not pass through a Serval [node](#node). ### MDP Address -Every [node](#node) in the [Serval mesh network][] uses a unique [Serval ID][] -(abbreviated to **SID**) as its **MDP address**. +Every [node](#node) in the [Serval mesh network][] uses a unique [Serval +ID][SID] (abbreviated to **SID**) as its **MDP address**. The [Serval DNA][] daemon can have one or many identities in its [keyring][], and each identity has its own unique [SID][]. The daemon chooses the first @@ -67,7 +82,7 @@ Address. At present, Serval routing does not handle this case, so it could cause unwanted effects such as route flapping or dropped messages. In practice, the “duplicate MDP Address” problem is rare for the time being, -because the [Serval Mesh][] app for Android does not provide any way for a +because the [Serval Mesh app for Android][] does not provide any way for a non-expert user to copy the [keyring][] file from one device to another, and other Serval devices such as the [Mesh Extender][] are not operated by end-users. @@ -131,14 +146,14 @@ in the MDP message header are optional, depending on the initial byte of bit flags: | bytes | name | present if | meaning | -|:------:|:-------------- |:----------------------- |:----------------------------------------------------- | +|:------:|:------------------ |:----------------------- |:----------------------------------------------------- | | 1 | FLAGS | | [message flags](#mdp-message-flags) | -| 1-33 | Sender SID | `!SENDER_SAME` | the [sender's](#sender) [address](#mdp-address) | -| 1-33 | Recipient SID | `!BROADCAST` | the [recipient's](#recipient) [address](#mdp-address) | +| 1..33 | sender address | `!SENDER_SAME` | the [sender's](#sender) [address](#mdp-address) | +| 1..33 | recipient address | `!BROADCAST` | the [recipient's](#recipient) [address](#mdp-address) | | 8 | broadcast sequence | `BROADCAST && !ONE_HOP` | [broadcast](#broadcast) message's sequence number | | 1 | TTL and QoS | `!ONE_HOP` | time-to-live counter and service type | | 2 | payload size | | number of bytes in payload | -| 0-max | payload | | | +| 0..max | payload | | | ### MDP message flags @@ -146,7 +161,7 @@ The single FLAGS byte at the start of the [MDP message](#mdp-message) header contains the following bits (bit numbers start with 0 = LSB): | bit | symbol | meaning | -|:-----:|:------------- | --------------------------------------------------------------------------------------------- | +|:-----:|:------------- |:--------------------------------------------------------------------------------------------- | | 0 | `SENDER_SAME` | the [transmitter](#transmitter) is the [sender](#sender) | | 1 | `BROADCAST` | message is [broadcast](#broadcast); has no [recipient](#recipient) | | 2 | `ONE_HOP` | message is on last [link](#link), so the [receiver](#receiver) is the [recipient](#recipient) | @@ -200,9 +215,45 @@ contains the following bits (bit numbers start with 0 = LSB): low-latency, high quality links to maximise throughput by avoiding redundant re-transmissions. +### MDP address fields + +The *sender* and *recipient* address fields in the [MDP message](#mdp-message) +header are encoded as a single qualifier byte **Q** followed by 0 ≤ **N** ≤ 32 +bytes of data: + +| **Q** | symbol | **N** | resolves to... | +|:-----:|:---------- |:-----:|:---------------------------------------------------------------------------------- | +| 0..31 | | **Q** | an [abbreviated address](#abbreviated-address) in binary format | +| 32 | | 32 | a complete [SID][] in binary format | +| 0xFB | `SIGNKEY` | 32 | a complete [Signing ID][] in binary format | +| 0xFC | `P2P_ME` | 0 | the *source* address of a [point-to-point link](#point-to-point-link) | +| 0xFD | `P2P_YOU` | 0 | the *destination* address of a [point-to-point link](#point-to-point-link) | +| 0xFE | `PREVIOUS` | 0 | the previous resolved address | +| 0xFF | `SELF` | 0 | the [transmitter](#transmitter) of the enclosing [overlay packet](#overlay-packet) | + +`SIGNKEY` addresses are only used for [combined IDs][], ie, where the [SID][]s +is cryptographically derived from a [Signing ID][]. Any [node](#node) running +a version of the [Serval DNA][] daemon that pre-dates the combined key upgrade +will not recognise the `SIGNKEY` address type, and will treat it as invalid. + +`P2P_ME` and `P2P_YOU` addresses are only valid in MDP messages that are being +transmitted over a [point-to-point link](#point-to-point-link). If a `P2P_ME` +address is received but the recipient [node](#node) has not yet discovered the +address of the source [node](#node) at the other end of the link, then the +recipient treats the address as invalid and initiates an address discovery +handshake, so that subsequent `P2P_ME` addresses may succeed. + +A `PREVIOUS` address resolves to the previous resolved sender or recipient +address in this MDP message or in the preceding MDP message in the enclosing +[overlay packet](#overlay-packet). This qualifier is generally not useful and +may be deprecated in future. + +The `SELF` qualifier is only valid in MDP messages that are encapsulated within +an [overlay packet](#overlay-packet). + ### Broadcast -TBC +**TODO**: Describe the behaviour of broadcast messages. ### Overlay Packet @@ -210,9 +261,9 @@ MDP transmits a [MDP message](#mdp-message) over a [link](#link) by encapsulating it into an **overlay packet** (also called **MDP packet** or **overlay frame**). The MDP overlay packet is a [layer 2][] concept; it is only concerned with transporting MDP messages across a single link to a -neighbouring *peer* node. Once an overlay packet arrives, the receiver -unpacks all of its MDP messages, consumes those for which it (or one of its -zero-hop identities) is the [recipient](#recipient) and independently +neighbouring *peer* [node](#node). Once an overlay packet arrives, the +receiver unpacks all of its MDP messages, consumes those for which it (or one +of its zero-hop identities) is the [recipient](#recipient) and independently routes each of the remaining messages to its next appropriate peer. Every overlay packet contains the [MDP address](#mdp-address)es of its @@ -220,9 +271,29 @@ Every overlay packet contains the [MDP address](#mdp-address)es of its An overlay packet may contain many MDP messages. The header of each MDP message in an overlay packet is constructed afresh when it is embedded into the -packet, setting its [flag bits](#mdp-message-flags) and using [SID -abbreviation][] to omit or reduce duplicated [SID][]s, in order to conserve -link bandwidth where possible. +packet, setting its [flag bits](#mdp-message-flags) and re-writing the [address +fields](#mdp-address-fields) within the context of the overlay packet, in order +to conserve link bandwidth by avoiding duplication where possible. + +**TODO**: Describe the structure of an overlay packet in detail. + +Abbreviated address +------------------- + +An *abbreviated address* is a truncated [SID][], ie, the initial **N** < 32 +bytes of a whole [SID][]. + +Since SIDs are randomly allocated and only relatively few SIDs are in use +within a local [Serval mesh network][] at a given time, all SIDs in use are +very likely to differ within their first few bytes. Thus, within the context +of the local mesh network, there is no need to use entire SIDs to uniquely +identify [node](#node)s. + +[SID abbreviation][] allows MDP messages to identify their sender and recipient +using far fewer than 32 bytes, typically only 1 or 2 bytes. + +**TODO**: Describe the the abbreviation resolution rules and the *explain* +handshake. MDP Client Interface -------------------- @@ -238,8 +309,17 @@ The *MDP Client API* is a [C language][] [API][] that an application can use to send and receive MDP packets over the [Serval mesh network][] using the [interface](#mdp-interface) provided by the [Serval DNA][] daemon. +History +------- + +MDP was designed and first prototyped in May-June 2012 as part of the [first +New America Foundation contract][naf1] to integrate Serval security into the +OpenBTS base station, and also as part of the development of [release 0.90 +“Shiny”][Batphone 0.90] of the [Serval Mesh app for Android][]. + ----- **Copyright 2014 Serval Project Inc.** +**Copyright 2016-2017 Flinders University** ![CC-BY-4.0](./cc-by-4.0.png) Available under the [Creative Commons Attribution 4.0 International licence][CC BY 4.0]. @@ -248,18 +328,23 @@ Available under the [Creative Commons Attribution 4.0 International licence][CC [CC BY 4.0]: ../LICENSE-DOCUMENTATION.md [Serval mesh network]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:mesh_network [Serval DNA]: ../README.md -[Serval Mesh]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:servalmesh: +[Serval Mesh app for Android]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:servalmesh: [Mesh Extender]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:meshextender: [MDP]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:mdp [datagram]: http://en.wikipedia.org/wiki/Datagram +[CSMA/CA]: http://en.wikipedia.org/wiki/CSMA/CA +[Wi-Fi]: http://en.wikipedia.org/wiki/WiFi +[AX.25 packet radio]: http://en.wikipedia.org/wiki/Packet_radio +[serial cable]: http://en.wikipedia.org/wiki/Serial_cable [Internet protocols]: https://en.wikipedia.org/wiki/Internet_protocol_suite [layer 2]: https://en.wikipedia.org/wiki/Data_link_layer [layer 3]: https://en.wikipedia.org/wiki/Network_layer [layer 4]: https://en.wikipedia.org/wiki/Transport_layer [UDP]: http://en.wikipedia.org/wiki/User_Datagram_Protocol [MTU]: http://en.wikipedia.org/wiki/Maximum_transmission_unit -[Serval ID]: ./REST-API-Keyring.md#serval-id [SID]: ./REST-API-Keyring.md#serval-id +[Signing ID]: ./REST-API-Keyring.md#serval-signing-id +[combined IDs]: ./REST-API-Keyring.md#combined-ids [SID abbreviation]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:sid_abbreviation [configured]: ./Servald-Configuration.md [network interfaces]: ./Servald-Configuration.md#network-interfaces @@ -270,3 +355,5 @@ Available under the [Creative Commons Attribution 4.0 International licence][CC [pipes]: https://en.wikipedia.org/wiki/Pipeline_(Unix) [C language]: http://en.wikipedia.org/wiki/C_(programming_language) [API]:http://en.wikipedia.org/wiki/Application_programming_interface +[naf1]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:activity:naf1 +[Batphone 0.90]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:servalmesh:releases:version_0_90 diff --git a/doc/REST-API-Keyring.md b/doc/REST-API-Keyring.md index f150a4fa..4fa16ee7 100644 --- a/doc/REST-API-Keyring.md +++ b/doc/REST-API-Keyring.md @@ -1,6 +1,6 @@ Keyring REST API ================ -[Serval Project][], November 2016 +[Serval Project][], November 2017 Introduction ------------ @@ -21,24 +21,53 @@ Basic concepts Every identity in the [Serval mesh network][] is represented by its **Serval ID**, (usually abbreviated to [SID][], and formerly known as “Subscriber ID”), which is a unique 256-bit public key in the [Curve25519][] *crypto-box* key -space that is generated from the random *Serval ID secret* when the identity is -created. The SID is used: +space. The SID is used: * as the network address in the [Serval Mesh network][] * to encrypt [MDP][] messages * to identify the senders, recipients and authors of [Rhizome bundles][] * to identify the parties in a [MeshMS conversation][] +The Serval ID public key is derived from the *Serval ID secret key*, which is +generated when the identity is first created, and stored in the keyring. The +original implementation of [Serval DNA][] generated the *Serval ID secret key* +at random, but since the introduction of [combined IDs](#combined-ids), the +secret key is derived from the [Serval Signing ID](#serval-signing-id) secret +key. + ### Serval Signing ID Every identity in the [Serval mesh network][] has a **Serval Signing ID**, which is a unique 256-bit public key in the [Curve25519][] *crypto-sign* key -space that is generated at the same time as the [Serval ID](#serval-id) when -the identity is created. The Signing ID is used: +space that is generated by choosing a *secret signing key* at random when the +the identity is first created. The Signing ID is used: * to prevent forgery of [Serval Mesh network][] routing messages * to authenticate non-encrypted [MDP][] messages +The large size of the key space means that, as long as the secret key choice +is truly random, the [probability of two identities having the same +ID][birthday paradox] is negligible, even if billions of identities are +generated. + +### Combined IDs + +Since July 2016, [Serval DNA][] generates [Serval ID](#serval-id)s by deriving +them from the [Serval Signing ID](#serval-signing-id), so when creating a new +identity, only a single secret key is generated at random instead of two. This +change was made possible by replacing the [NaCl][] cryptographic library with +[libsodium][], which provides a primitive to map keys from the *crypto-sign* +key space to the *crypto-box* key space (but not vice versa, which is +mathematically impossible). + +These related keys are called *combined IDs*. + +Eventually, the [Serval ID](#serval-id) will be replaced by the [Serval Signing +ID](#serval-signing-id) throughout the Serval software and protocols, but will +still be called “Serval ID”. The encryption key (in the *crypto-box* key +space) will be derived from the signing key on the fly whenever needed, so there +will only be one kind of Serval ID, which will be a great simplification. + ### DID The **DID** ([Dialled Identity][]) is a telephone number, represented as a @@ -175,7 +204,7 @@ result](#keyring-json-result) describes the identity that was locked. ----- **Copyright 2015 Serval Project Inc.** -**Copyright 2016 Flinders University** +**Copyright 2016-2017 Flinders University** ![CC-BY-4.0](./cc-by-4.0.png) Available under the [Creative Commons Attribution 4.0 International licence][CC BY 4.0]. @@ -188,7 +217,10 @@ Available under the [Creative Commons Attribution 4.0 International licence][CC [Keyring]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:keyring [cryptographic identities]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:security_framework [Curve25519]: https://en.wikipedia.org/wiki/Curve25519 +[libsodium]: https://libsodium.org/ +[NaCl]: https://nacl.cr.yp.to/ [SID]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:sid +[birthday paradox]: http://en.wikipedia.org/wiki/Birthday_problem [Dialled Identity]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:did [DNA]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:dna [MDP]: ./Mesh-Datagram-Protocol.md