serval-dna/doc/Cooee.md

Cooee Service Discovery Protocol
================================
[Serval Project], May 2014

[Cooee][] is a protocol for discovering services offered by reachable nodes in
a [Serval mesh network][].  It is named after the [Australian bush shout][].

Cooee was funded by a [grant][] from the [New America Foundation][NAF]'s [Open
Technology Institute][OTI].

What is a Service?
------------------

A *service* is a program (called a *server*) running on a single network node,
that accepts spontaneous requests from other programs (called *clients*) via
the network, performs a useful function, and replies to the client via the
network.

A server may decline to respond to some clients based on their [SID][]
(typically a blacklist or whitelist) or some other criterion such as load or
some aspect of the request.  In general, however, a service is not selective
about which clients it will serve, and does not have prior knowledge of who
will contact it, or when.  In order to avoid service delays or absences, a
server must be able to handle many requests concurrently.

Service protocols and state
---------------------------

Each service has its own particular protocol, and clients wishing to use the
service must use that protocol.

A *one-shot* service protocol contains the entire request in a single [MDP][]
packet, and typically the response is a single packet sent back to the client's
originating [MDP port][] number.  A one-shot service with very rapid reply
generation may not need to be concurrent; it need only queue all incoming
requests and deal with them in turn.  In a one-shot service, it is possible for
the server to be fully *stateless*, because it need not remember any prior
requests or responses.  Service congestion can be dealt with by simply dropping
new requests when the queue is full.  The absence of a response causes the
client to re-send the request after some time-out.

More elaborate service protocols involve establishing a *session* between the
client and server, which is discarded once the service has been performed.  The
[Mesh Stream Protocol][MSP] is commonly used for this kind of service, to
provide lossless transmission of messages between client and server.
Session-based servers must retain some state per session.

Server MDP port number
----------------------

A server operates by listening for [MDP][] packets sent to a single [MDP
port][] on its node.  The port number may be reserved in advance for that
service, which means that the service can have at most one server running at a
time per node.  Alternatively, the port number may be allocated whenever the
server starts, so is not known in advance, but each node may support many
servers offering the same service.

In general, clients that wish to use a service do not know the [MDP port][]
number or [SID][] of the node that hosts the server.  They must use [Cooee][]
to discover the SID and port number.

What is Cooee?
--------------

[Cooee][] is a *one-shot* [MDP][] service available on [MDP port][] **11** of
every node, that resolves service names to port numbers and other qualifying
information.

In Cooee, every service is described by a *stanza* of `name=value` pairs, using
the same [UTF-8][] text format as the [Serval DNA configuration file][].

A Cooee request packet contains one or more *patterns* that are each matched
against the names in each service's stanza.  The request packet is typically
broadcast to all nodes in the reachable (local) Mesh network, and every service
with a name that matches any pattern sends a reply packet containing the lines
from its stanza that match.

How does a client use Cooee?
----------------------------

Client programs use Cooee by sending a broadcast packet to port 11 and
collecting all the reply packets that it provokes.  Any packet whose content
does not strictly conform to the stanza grammar should be ignored.

For example, a client searching for a [SOCKS5][]-over-[MSP][] [Internet][]
access point may send a broadcast packet containing the following pattern:

    socks5.msp.*

It may receive the following response from one node:

    socks5.msp.port=34
    socks5.msp.name.en=Filtered Internet service
    socks5.msp.rx_bps=174000
    socks5.msp.tx_bps=36000

... and the following response from another node:

    socks5.msp.port=116
    socks5.msp.name.en=Telco mobile data plan
    socks5.msp.name.es=Móvil internet de Telco

The exact meanings of the lines in the replies is defined below.

Clients may simply ignore any lines that they do not recognize, or may present
them to the human user to assist in choosing between services.  For example,
even though the `.rx_bps` and `.tx_bps` lines shown above may not be recognised
by all clients, their presence or absence will not cause any client to
disregard the service itself.

The remote (sending) [SID][] of the Cooee reply packet gives the node that
hosts the service, so a node cannot reply for services available on other
nodes.

How does a server advertise itself using Cooee?
-----------------------------------------------

Every server that wishes to make itself discoverable via Cooee must know its
own stanza, and must respond to Cooee requests on port 11 (as well as
responding to the port on which it provides its own service).  The [MDP port
sharing][] feature of [Serval DNA][] allows many servers to listen on port 11
at the same time, and all of them will receive a copy of every packet received
on port 11.

Whenever a server receives a Cooee request packet, it must match the requested
pattern against its own stanza.  If any lines match, it must construct and send
a reply packet containing the matching lines.  If a server provides many
services by listening on many ports, then it will know one stanza per service,
and if more than one of those stanzas match a Cooee request, then it must send
one reply packet per matching stanza -- it cannot combine all the matches into
a single packet.

This means that a single Cooee request may provoke more than one response
packet from one node, if more than one server running on that node provides the
service being sought.

Service description stanzas
---------------------------

A Cooee service description stanza is a block of [UTF-8][] text which conforms
to the following grammar:

    STANZA := LINE {1..}

    LINE := NAME "=" VALUE "\n"

    NAME := WORD ( "." WORD ){0..}
    WORD := WCHAR {1..}
    WCHAR := DIGIT | UPPERCASE | LOWERCASE | "_"
    DIGIT := "0".."9"      (ASCII digits)
    UPPERCASE := "A".."Z"  (ASCII uppercase)
    LOWERCASE := "a".."z"  (ASCII lowercase)

    VALUE := VCHAR {0..}
    VCHAR := any UTF-8 character except "\n" and NUL

In other words, **NAME** is a sequence of one or more alphanumeric words
separated by period characters, and **VALUE** is any sequence of characters not
containing NUL (zero) or newline `"\n"`.

**Note** that the grammar does not permit blank lines or white space before, in
or after a NAME.

### Standard service names

In general, every service description NAME must have the form:

    servicename.protocol.attribute

#### Standard `servicename` values:

 * **`socks5`** -- a SOCKS5 forward proxy that provides access to the public
   [Internet][]

 * **`http_proxy`** -- a Web forward proxy that provides access to the public
   [World Wide Web][]

#### Standard `protocol` values:

 * **`mdp`** -- [Mesh Datagram Protocol][MSP], generally only of use for
   one-shot services

 * **`msp`** -- [Mesh Stream Protocol][MSP], generally used by session-based
   services

#### Standard `attribute` values:

 * **`port`** -- the [MDP port][] number on which the service may be found, the
   value is either decimal (eg, `31`) or hexadecimal with a `0x` prefix (eg,
   `0x1f`)

 * **`name.en`** -- a textual description of the service that can be presented
   to an English-speaking human user (this is just a case of the `name.XX`
   attribute below)

 * **`name.XX`** -- a textual description of the service in a human language
   identified by the `XX` two-letter [ISO 639-1][] code, eg, `name.es` for
   Spanish, `name.ja` for Japanese, etc.

### Non-standard service names

Services may put any lines they wish into their Cooee stanza.  However, names
not defined in this standard run the risk of being incompatible with a future
expansion of the standard.

If a service wishes to publish *extra-standard* information about itself, it
must use NAME components that start with the underscore `_` character.  The
Cooee standard will never use these names, so the only risk is collision with
other services that have used the same name independently.

Service name patterns
---------------------

A Cooee request *pattern* uses a [glob][]-like syntax.  For example:

    [A-Z_]*.(ms|tc)p.(port|name.es)

will match all service names that start with an uppercase letter or underscore,
and whose protocol is either `msp` or `tcp`, and only matches the `port` and
`name.es` attributes.

 * __`text`__ -- matches exactly `text`, which may contain periods

 * __`*`__ -- matches zero or more of any character excluding period

 * __`**`__ -- matches zero or more of any character including periods

 * __`[SET]`__ -- matches any single character in SET, where SET is a
   concatenation of:
   * __`C`__ -- a single character `C`
   * __`A-B`__ -- a range of characters in [ASCII][] code order between `A` and
     `B` inclusive

 * __`[!SET]`__ -- matches any single character not in SET

 * __`(one|two|...)`__ matches either exactly `one` or `two` or any other
   alternatives separated by bars `|` and enclosed in parentheses

Some more examples:

 * Find all [MSP][] services: `*.msp.**`

 * Find only the port numbers of all [MDP][] services: `*.mdp.port`

 * Find all lines of all stanzas of all services: `**`

 * Find all non-standard lines of all stanzas of all services: `(_**|**._**)`

-----
**Copyright 2014 Serval Project Inc.**  
![CC-BY-4.0](./cc-by-4.0.png)
Available under the [Creative Commons Attribution 4.0 International licence][CC BY 4.0].


[Serval Project]: http://www.servalproject.org/
[CC BY 4.0]: ../LICENSE-DOCUMENTATION.md
[Australian bush shout]: http://en.wikipedia.org/wiki/Cooee
[Cooee]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:cooee
[grant]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:activity:naf6
[NAF]: http://www.newamerica.net/
[OTI]: http://oti.newamerica.net/
[Serval mesh network]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:mesh_network
[Serval DNA]: ../README.md
[Serval DNA configuration file]: ./Servald-Configuration.md
[SID]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:sid
[MDP]: ./Mesh-Datagram-Protocol.md
[MSP]: ./Mesh-Stream-Protocol.md
[MDP port]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:mdp_port_number
[MDP port sharing]: http://developer.servalproject.org/dokuwiki/doku.php?id=content:tech:mdp_port_sharing
[SOCKS5]: http://en.wikipedia.org/wiki/SOCKS
[Internet]: http://en.wikipedia.org/wiki/Internet
[World Wide Web]: http://en.wikipedia.org/wiki/World_Wide_Web
[UTF-8]: http://en.wikipedia.org/wiki/UTF-8
[ASCII]: http://en.wikipedia.org/wiki/ASCII
[ISO 639-1]: http://en.wikipedia.org/wiki/ISO_639-1
[glob]: http://en.wikipedia.org/wiki/Glob_(programming)