Improve MDP technical documentation

The MDP technical document can now replace the Serval Developer wiki MDP
page.

doc/Mesh-Datagram-Protocol.md

@@ -1,21 +1,36 @@
 Mesh Datagram Protocol (MDP)
 ============================
-[Serval Project], March 2016
+[Serval Project], November 2017
 
-The [Mesh Datagram Protocol][MDP] is a [layer 3][] network protocol developed
+The [Mesh Datagram Protocol][MDP] is a [layer 3][] [datagram][] network
-for the [Serval mesh network][], with characteristics that make it particularly
+protocol developed for the [Serval mesh network][], with characteristics that
-suitable for use in Ad Hoc wireless networks, which can suffer high levels of
+make it particularly suitable for use in Ad Hoc wireless networks, which can
-packet loss due to weak signal, interference and congestion.
+suffer high levels of packet loss due to weak signal, interference and
+congestion.
 
-MDP carries [messages](#mdp-message) from [sender](#sender) to [recipient](#recipient)
+MDP carries [messages](#mdp-message) from [sender](#sender) to
-[node](#node)s, or [broadcasts](#broadcast) to all nodes, guaranteeing only that
+[recipient](#recipient) [node](#node)s, or [broadcasts](#broadcast) to all
-message contents will be correct if delivered, like a traditional [datagram][]
+nodes.  MDP guarantees that message contents will be correct if delivered, but
-service.  MDP is similar to [UDP][] in this respect, but it uses per-[link](#link)
+does not guarantee one delivery (messages may be lost or delivered more than
-retransmission and adaptive link-state routing to boost packet delivery
+once), arrival time, or message order.
-rates, which largely immunises it from the cumulative packet loss effect
+
-typical of multi-hop wireless networks.  This means that, unlike [Internet
+MDP can be carried over any wireless or wired data link, whether a shared
-protocols][] its end-to-end, multi-hop packet delivery rate remains usefully
+medium (eg, [CSMA/CA][] used in [Wi-Fi][]) or a dedicated medium (eg, [AX.25
-high despite adverse and changing network conditions.
+packet radio][], [serial cable][]).
+
+MDP is similar to [UDP][], but it uses per-[link](#link) retransmission and
+adaptive link-state routing to boost packet delivery rates, which largely
+immunises it from the cumulative packet loss effect typical of multi-hop
+wireless networks.  To carry a packet over **N** hops, where each hop has a
+probability **P** of dropping a packet due to interference or collision, the
+end-to-end loss is **1 − pow(1−P, N)**. For example, given a per-hop packet loss
+of 10%, a five hop route has a net packet loss of 41%, and a ten hop route
+has 75% packet loss.
+
+The MDP retransmission scheme reduces but does not eliminate packet loss, and
+sometimes produces duplicate packets.  However, it can squeeze useful packet
+delivery rates from a high-loss route that would be practically useless with
+[Internet protocols][], which rely on end-to-end retransmission.
 
 Basic concepts
 --------------
@@ -48,8 +63,8 @@ the packet does not pass through a Serval [node](#node).
 
 ### MDP Address
 
-Every [node](#node) in the [Serval mesh network][] uses a unique [Serval ID][]
+Every [node](#node) in the [Serval mesh network][] uses a unique [Serval
-(abbreviated to **SID**) as its **MDP address**.
+ID][SID] (abbreviated to **SID**) as its **MDP address**.
 
 The [Serval DNA][] daemon can have one or many identities in its [keyring][],
 and each identity has its own unique [SID][].  The daemon chooses the first
@@ -67,7 +82,7 @@ Address.  At present, Serval routing does not handle this case, so it could
 cause unwanted effects such as route flapping or dropped messages.
 
 In practice, the “duplicate MDP Address” problem is rare for the time being,
-because the [Serval Mesh][] app for Android does not provide any way for a
+because the [Serval Mesh app for Android][] does not provide any way for a
 non-expert user to copy the [keyring][] file from one device to another, and
 other Serval devices such as the [Mesh Extender][] are not operated by
 end-users.
@@ -131,14 +146,14 @@ in the MDP message header are optional, depending on the initial byte of bit
 flags:
 
 | bytes  | name               | present if              | meaning                                               |
-|:------:|:--------------     |:----------------------- |:----------------------------------------------------- |
+|:------:|:------------------ |:----------------------- |:----------------------------------------------------- |
 | 1      | FLAGS              |                         | [message flags](#mdp-message-flags)                   |
-| 1-33   | Sender SID         | `!SENDER_SAME`          | the [sender's](#sender) [address](#mdp-address)       |
+| 1..33  | sender address     | `!SENDER_SAME`          | the [sender's](#sender) [address](#mdp-address)       |
-| 1-33   | Recipient SID      | `!BROADCAST`            | the [recipient's](#recipient) [address](#mdp-address) |
+| 1..33  | recipient address  | `!BROADCAST`            | the [recipient's](#recipient) [address](#mdp-address) |
 | 8      | broadcast sequence | `BROADCAST && !ONE_HOP` | [broadcast](#broadcast) message's sequence number     |
 | 1      | TTL and QoS        | `!ONE_HOP`              | time-to-live counter and service type                 |
 | 2      | payload size       |                         | number of bytes in payload                            |
-| 0-max  | payload            |                         |                                                       |
+| 0..max | payload            |                         |                                                       |
 
 ### MDP message flags
 
@@ -146,7 +161,7 @@ The single FLAGS byte at the start of the [MDP message](#mdp-message) header
 contains the following bits (bit numbers start with 0 = LSB):
 
 |  bit  | symbol        | meaning                                                                                       |
-|:-----:|:------------- | --------------------------------------------------------------------------------------------- |
+|:-----:|:------------- |:--------------------------------------------------------------------------------------------- |
 |   0   | `SENDER_SAME` | the [transmitter](#transmitter) is the [sender](#sender)                                      |
 |   1   | `BROADCAST`   | message is [broadcast](#broadcast); has no [recipient](#recipient)                            |
 |   2   | `ONE_HOP`     | message is on last [link](#link), so the [receiver](#receiver) is the [recipient](#recipient) |
@@ -200,9 +215,45 @@ contains the following bits (bit numbers start with 0 = LSB):
   low-latency, high quality links to maximise throughput by avoiding redundant
   re-transmissions.
 
+### MDP address fields
+
+The *sender* and *recipient* address fields in the [MDP message](#mdp-message)
+header are encoded as a single qualifier byte **Q** followed by 0 ≤ **N** ≤ 32
+bytes of data:
+
+| **Q** | symbol     | **N** | resolves to...                                                                     |
+|:-----:|:---------- |:-----:|:---------------------------------------------------------------------------------- |
+| 0..31 |            | **Q** | an [abbreviated address](#abbreviated-address) in binary format                    |
+|  32   |            |   32  | a complete [SID][] in binary format                                                |
+| 0xFB  | `SIGNKEY`  |   32  | a complete [Signing ID][] in binary format                                         |
+| 0xFC  | `P2P_ME`   |    0  | the *source* address of a [point-to-point link](#point-to-point-link)              |
+| 0xFD  | `P2P_YOU`  |    0  | the *destination* address of a [point-to-point link](#point-to-point-link)         |
+| 0xFE  | `PREVIOUS` |    0  | the previous resolved address                                                      |
+| 0xFF  | `SELF`     |    0  | the [transmitter](#transmitter) of the enclosing [overlay packet](#overlay-packet) |
+
+`SIGNKEY` addresses are only used for [combined IDs][], ie, where the [SID][]s
+is cryptographically derived from a [Signing ID][].  Any [node](#node) running
+a version of the [Serval DNA][] daemon that pre-dates the combined key upgrade
+will not recognise the `SIGNKEY` address type, and will treat it as invalid.
+
+`P2P_ME` and `P2P_YOU` addresses are only valid in MDP messages that are being
+transmitted over a [point-to-point link](#point-to-point-link).  If a `P2P_ME`
+address is received but the recipient [node](#node) has not yet discovered the
+address of the source [node](#node) at the other end of the link, then the
+recipient treats the address as invalid and initiates an address discovery
+handshake, so that subsequent `P2P_ME` addresses may succeed.
+
+A `PREVIOUS` address resolves to the previous resolved sender or recipient
+address in this MDP message or in the preceding MDP message in the enclosing
+[overlay packet](#overlay-packet).  This qualifier is generally not useful and
+may be deprecated in future.
+
+The `SELF` qualifier is only valid in MDP messages that are encapsulated within
+an [overlay packet](#overlay-packet).
+
 ### Broadcast
 
-TBC
+**TODO**: Describe the behaviour of broadcast messages.
 
 ### Overlay Packet
 
@@ -210,9 +261,9 @@ MDP transmits a [MDP message](#mdp-message) over a [link](#link) by
 encapsulating it into an **overlay packet** (also called **MDP packet** or
 **overlay frame**).  The MDP overlay packet is a [layer 2][] concept; it is
 only concerned with transporting MDP messages across a single link to a
-neighbouring *peer* node.  Once an overlay packet arrives, the receiver
+neighbouring *peer* [node](#node).  Once an overlay packet arrives, the
-unpacks all of its MDP messages, consumes those for which it (or one of its
+receiver unpacks all of its MDP messages, consumes those for which it (or one
-zero-hop identities) is the [recipient](#recipient) and independently
+of its zero-hop identities) is the [recipient](#recipient) and independently
 routes each of the remaining messages to its next appropriate peer.
 
 Every overlay packet contains the [MDP address](#mdp-address)es of its
@@ -220,9 +271,29 @@ Every overlay packet contains the [MDP address](#mdp-address)es of its
 
 An overlay packet may contain many MDP messages.  The header of each MDP
 message in an overlay packet is constructed afresh when it is embedded into the
-packet, setting its [flag bits](#mdp-message-flags) and using [SID
+packet, setting its [flag bits](#mdp-message-flags) and re-writing the [address
-abbreviation][] to omit or reduce duplicated [SID][]s, in order to conserve
+fields](#mdp-address-fields) within the context of the overlay packet, in order
-link bandwidth where possible.
+to conserve link bandwidth by avoiding duplication where possible.
+
+**TODO**: Describe the structure of an overlay packet in detail.
+
+Abbreviated address
+-------------------
+
+An *abbreviated address* is a truncated [SID][], ie, the initial **N** < 32
+bytes of a whole [SID][].
+
+Since SIDs are randomly allocated and only relatively few SIDs are in use
+within a local [Serval mesh network][] at a given time, all SIDs in use are
+very likely to differ within their first few bytes.  Thus, within the context
+of the local mesh network, there is no need to use entire SIDs to uniquely
+identify [node](#node)s.
+
+[SID abbreviation][] allows MDP messages to identify their sender and recipient
+using far fewer than 32 bytes, typically only 1 or 2 bytes.
+
+**TODO**: Describe the the abbreviation resolution rules and the *explain*
+handshake.
 
 MDP Client Interface
 --------------------
@@ -238,8 +309,17 @@ The *MDP Client API* is a [C language][] [API][] that an application can use to
 send and receive MDP packets over the [Serval mesh network][] using the
 [interface](#mdp-interface) provided by the [Serval DNA][] daemon.
 
+History
+-------
+
+MDP was designed and first prototyped in May-June 2012 as part of the [first
+New America Foundation contract][naf1] to integrate Serval security into the
+OpenBTS base station, and also as part of the development of [release 0.90
+“Shiny”][Batphone 0.90] of the [Serval Mesh app for Android][].
+
 -----
 **Copyright 2014 Serval Project Inc.**  
+**Copyright 2016-2017 Flinders University**  
 ![CC-BY-4.0](./cc-by-4.0.png)
 Available under the [Creative Commons Attribution 4.0 International licence][CC BY 4.0].
 
@@ -248,18 +328,23 @@ Available under the [Creative Commons Attribution 4.0 International licence][CC
 [CC BY 4.0]: ../LICENSE-DOCUMENTATION.md
 [Serval mesh network]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:mesh_network
 [Serval DNA]: ../README.md
-[Serval Mesh]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:servalmesh:
+[Serval Mesh app for Android]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:servalmesh:
 [Mesh Extender]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:meshextender:
 [MDP]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:mdp
 [datagram]: http://en.wikipedia.org/wiki/Datagram
+[CSMA/CA]: http://en.wikipedia.org/wiki/CSMA/CA
+[Wi-Fi]: http://en.wikipedia.org/wiki/WiFi
+[AX.25 packet radio]: http://en.wikipedia.org/wiki/Packet_radio
+[serial cable]: http://en.wikipedia.org/wiki/Serial_cable
 [Internet protocols]: https://en.wikipedia.org/wiki/Internet_protocol_suite
 [layer 2]: https://en.wikipedia.org/wiki/Data_link_layer
 [layer 3]: https://en.wikipedia.org/wiki/Network_layer
 [layer 4]: https://en.wikipedia.org/wiki/Transport_layer
 [UDP]: http://en.wikipedia.org/wiki/User_Datagram_Protocol
 [MTU]: http://en.wikipedia.org/wiki/Maximum_transmission_unit
-[Serval ID]: ./REST-API-Keyring.md#serval-id
 [SID]: ./REST-API-Keyring.md#serval-id
+[Signing ID]: ./REST-API-Keyring.md#serval-signing-id
+[combined IDs]: ./REST-API-Keyring.md#combined-ids
 [SID abbreviation]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:sid_abbreviation
 [configured]: ./Servald-Configuration.md
 [network interfaces]: ./Servald-Configuration.md#network-interfaces
@@ -270,3 +355,5 @@ Available under the [Creative Commons Attribution 4.0 International licence][CC
 [pipes]: https://en.wikipedia.org/wiki/Pipeline_(Unix)
 [C language]: http://en.wikipedia.org/wiki/C_(programming_language)
 [API]:http://en.wikipedia.org/wiki/Application_programming_interface
+[naf1]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:activity:naf1
+[Batphone 0.90]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:servalmesh:releases:version_0_90

doc/REST-API-Keyring.md

@@ -1,6 +1,6 @@
 Keyring REST API
 ================
-[Serval Project][], November 2016
+[Serval Project][], November 2017
 
 Introduction
 ------------
@@ -21,24 +21,53 @@ Basic concepts
 Every identity in the [Serval mesh network][] is represented by its **Serval
 ID**, (usually abbreviated to [SID][], and formerly known as “Subscriber ID”),
 which is a unique 256-bit public key in the [Curve25519][] *crypto-box* key
-space that is generated from the random *Serval ID secret* when the identity is
+space.  The SID is used:
-created.  The SID is used:
 
 *  as the network address in the [Serval Mesh network][]
 *  to encrypt [MDP][] messages
 *  to identify the senders, recipients and authors of [Rhizome bundles][]
 *  to identify the parties in a [MeshMS conversation][]
 
+The Serval ID public key is derived from the *Serval ID secret key*, which is
+generated when the identity is first created, and stored in the keyring.  The
+original implementation of [Serval DNA][] generated the *Serval ID secret key*
+at random, but since the introduction of [combined IDs](#combined-ids), the
+secret key is derived from the [Serval Signing ID](#serval-signing-id) secret
+key.
+
 ### Serval Signing ID
 
 Every identity in the [Serval mesh network][] has a **Serval Signing ID**,
 which is a unique 256-bit public key in the [Curve25519][] *crypto-sign* key
-space that is generated at the same time as the [Serval ID](#serval-id) when
+space that is generated by choosing a *secret signing key* at random when the
-the identity is created.  The Signing ID is used:
+the identity is first created.  The Signing ID is used:
 
 *  to prevent forgery of [Serval Mesh network][] routing messages
 *  to authenticate non-encrypted [MDP][] messages
 
+The large size of the key space means that, as long as the secret key choice
+is truly random, the [probability of two identities having the same
+ID][birthday paradox] is negligible, even if billions of identities are
+generated.
+
+### Combined IDs
+
+Since July 2016, [Serval DNA][] generates [Serval ID](#serval-id)s by deriving
+them from the [Serval Signing ID](#serval-signing-id), so when creating a new
+identity, only a single secret key is generated at random instead of two.  This
+change was made possible by replacing the [NaCl][] cryptographic library with
+[libsodium][], which provides a primitive to map keys from the *crypto-sign*
+key space to the *crypto-box* key space (but not vice versa, which is
+mathematically impossible).
+
+These related keys are called *combined IDs*.
+
+Eventually, the [Serval ID](#serval-id) will be replaced by the [Serval Signing
+ID](#serval-signing-id) throughout the Serval software and protocols, but will
+still be called “Serval ID”.  The encryption key (in the *crypto-box* key
+space) will be derived from the signing key on the fly whenever needed, so there
+will only be one kind of Serval ID, which will be a great simplification.
+
 ### DID
 
 The **DID** ([Dialled Identity][]) is a telephone number, represented as a
@@ -175,7 +204,7 @@ result](#keyring-json-result) describes the identity that was locked.
 
 -----
 **Copyright 2015 Serval Project Inc.**  
-**Copyright 2016 Flinders University**  
+**Copyright 2016-2017 Flinders University**  
 ![CC-BY-4.0](./cc-by-4.0.png)
 Available under the [Creative Commons Attribution 4.0 International licence][CC BY 4.0].
 
@@ -188,7 +217,10 @@ Available under the [Creative Commons Attribution 4.0 International licence][CC
 [Keyring]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:keyring
 [cryptographic identities]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:security_framework
 [Curve25519]: https://en.wikipedia.org/wiki/Curve25519
+[libsodium]: https://libsodium.org/
+[NaCl]: https://nacl.cr.yp.to/
 [SID]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:sid
+[birthday paradox]: http://en.wikipedia.org/wiki/Birthday_problem
 [Dialled Identity]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:did
 [DNA]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:dna
 [MDP]: ./Mesh-Datagram-Protocol.md