Docs: remove roadmap, add a new doc page on oracles.

docs/source/index.rst

@@ -38,12 +38,12 @@ Read on to learn:
 
    tutorial
    protocol-state-machines
+   oracles
 
 .. toctree::
    :maxdepth: 2
    :caption: Appendix
 
    visualiser
-   roadmap
    codestyle
 

docs/source/inthebox.rst

@@ -3,15 +3,16 @@ What's included?
 
 The current prototype consists of a small amount of code that defines:
 
-* Key data structures
+* Key data structures.
 * Algorithms that work with them, such as serialising, hashing, signing, and verification of the signatures.
-* Three smart contracts that implement a notion of a cash claim, a basic commercial paper and a crowdfunding contract.
-  These are simplified versions of the real things.
+* Two smart contracts that implement a notion of a cash claim and basic commercial paper (implemented twice, in two
+  different programming languages). These are simplified versions of the real things.
 * Unit tests that check the algorithms do what is expected, and which verify the behaviour of the smart contracts.
 * API documentation and tutorials (what you're reading)
-* A simple standalone node that uses an embedded message queue broker as its P2P messaging layer
+* A simple standalone node that uses an embedded message queue broker as its P2P messaging layer.
 * A trading demo that runs the node in either a listening/buying mode, or a connecting/selling mode, and swaps some
-  fake commercial paper assets for some self-issued IOU cash.
+  fake commercial paper assets for some self-issued IOU cash, using a generic *protocol framework*.
+* It also includes two oracles: one for precise timestamping and another for interest rate swaps.
 
 Some things it does not currently include but should gain later are:
 
@@ -27,8 +28,9 @@ You can browse `the JIRA bug tracker <https://r3-cev.atlassian.net/>`_.
 The prototype's goal is rapid exploration of ideas. Therefore in places it takes shortcuts that a production system
 would not in order to boost productivity:
 
-* It uses a serialization framework instead of a well specified, vendor neutral protocol.
+* It uses an object graph serialization framework instead of a well specified, vendor neutral protocol.
 * It uses secp256r1, an obsolete elliptic curve.
+* It uses the default, out of the box Apache Artemis MQ protocol instead of AMQP/1.0 (although switching should be easy)
 
 Contracts
 ---------

docs/source/oracles.rst

@@ -0,0 +1,186 @@
+.. highlight:: kotlin
+.. raw:: html
+
+   <script type="text/javascript" src="_static/jquery.js"></script>
+   <script type="text/javascript" src="_static/codesets.js"></script>
+
+Writing oracle services
+=======================
+
+This article covers *oracles*: network services that link the ledger to the outside world by providing facts that
+affect the validity of transactions.
+
+The current prototype includes two oracles:
+
+1. A timestamping service
+2. An interest rate fixing service
+
+We will examine the similarities and differences in their design, whilst covering how the oracle concept works.
+
+Introduction
+------------
+
+Oracles are a key concept in the block chain/decentralised ledger space. They can be essential for many kinds of
+application, because we often wish to condition a transaction on some fact being true or false, but the ledger itself
+has a design that is essentially functional: all transactions are *pure* and *immutable*. Phrased another way, a
+smart contract cannot perform any input/output or depend on any state outside of the transaction itself. There is no
+way to download a web page or interact with the user, in a smart contract. It must be this way because everyone must
+be able to independently check a transaction and arrive at an identical conclusion for the ledger to maintan its
+integrity: if a transaction could evaluate to "valid" on one computer and then "invalid" a few minutes later on a
+different computer, the entire shared ledger concept wouldn't work.
+
+But it is often essential that transactions do depend on data from the outside world, for example, verifying that an
+interest rate swap is paying out correctly may require data on interest rates, verifying that a loan has reached
+maturity requires knowledge about the current time, knowing which side of a bet receives the payment may require
+arbitrary facts about the real world (e.g. the bankruptcy or solvency of a company or country) ... and so on.
+
+We can solve this problem by introducing services that create digitally signed data structures which assert facts.
+These structures can then be used as an input to a transaction and distributed with the transaction data itself. Because
+the statements are themselves immutable and signed, it is impossible for an oracle to change its mind later and
+invalidate transactions that were previously found to be valid. In contrast, consider what would happen if a contract
+could do an HTTP request: it's possible that an answer would change after being downloaded, resulting in loss of
+consensus (breaks).
+
+The two basic approaches
+------------------------
+
+The architecture provides two ways of implementing oracles with different tradeoffs:
+
+1. Using commands
+2. Using attachments
+
+When a fact is encoded in a command, it is embedded in the transaction itself. The oracle then acts as a co-signer to
+the entire transaction. The oracle's signature is valid only for that transaction, and thus even if a fact (like a
+stock price) does not change, every transaction that incorporates that fact must go back to the oracle for signing.
+
+When a fact is encoded as an attachment, it is a separate object to the transaction which is referred to by hash.
+Nodes download attachments from peers at the same time as they download transactions, unless of course the node has
+already seen that attachment, in which case it won't fetch it again. Contracts have access to the contents of
+attachments and attachments can be digitally signed (in future).
+
+As you can see, both approaches share a few things: they both allow arbitrary binary data to be provided to transactions
+(and thus contracts). The primary difference is whether the data is a freely reusable, standalone object or whether it's
+integrated with a transaction.
+
+Here's a quick way to decide which approach makes more sense for your data source:
+
+* Is your data *continuously changing*, like a stock price, the current time, etc? If yes, use a command.
+* Is your data *commercially valuable*, like a feed which you are not allowed to resell unless it's incorporated into
+  a business deal? If yes, use a command, so you can charge money for signing the same fact in each unique business
+  context.
+* Is your data *very small*, like a single number? If yes, use a command.
+* Is your data *large*, *static* and *commercially worthless*, for instance, a holiday calendar? If yes, use an
+  attachment.
+* Is your data *intended for human consumption*, like a PDF of legal prose, or an Excel spreadsheet? If yes, use an
+  attachment.
+
+Asserting continuously varying data that is publicly known
+----------------------------------------------------------
+
+Let's look at the timestamping oracle that can be found in the ``TimestamperService`` class. This is an example of
+an oracle that uses a command because the current time is a constantly changing fact that everybody knows.
+
+The most obvious way to implement such a service would be:
+
+1. The creator of the transaction that depends on the time reads their local clock
+2. They insert a command with that time into the transaction
+3. They then send it to the oracle for signing.
+
+But this approach has a problem. There will never be exact clock synchronisation between the party creating the
+transaction and the oracle. This is not only due to physics, network latencies etc but because between inserting the
+command and getting the oracle to sign there may be many other steps, like sending the transaction to other parties
+involved in the trade as well, or even requesting human signoff. Thus the time observed by the oracle may be quite
+different to the time observed in step 1. This problem can occur any time an oracle attests to a constantly changing
+value.
+
+.. note:: It is assumed that "true time" for a timestamping oracle means GPS/NaviStar time as defined by the atomic
+   clocks at the US Naval Observatory. This time feed is extremely accurate and available globally for free.
+
+We fix it by including explicit tolerances in the command, which is defined like this:
+
+.. sourcecode:: kotlin
+
+   data class TimestampCommand(val after: Instant?, val before: Instant?) : CommandData
+       init {
+           if (after == null && before == null)
+               throw IllegalArgumentException("At least one of before/after must be specified")
+           if (after != null && before != null)
+               check(after <= before)
+       }
+   }
+
+This defines a class that has two optional fields: before and after, along with a constructor that imposes a couple
+more constraints that cannot be expressed in the type system, namely, that "after" actually is temporally after
+"before", and that at least one bound must be present. A timestamp command that doesn't contain anything is illegal.
+
+Thus we express that the *true value* of the fact "the current time" is actually unknowable. Even when both before and
+after times are included, the transaction could have occurred at any point between those two timestamps. In this case
+"occurrence" could mean the execution date, the value date, the trade date etc ... the oracle doesn't care what precise
+meaning the timestamp has to the contract.
+
+By creating a range that can be either closed or open at one end, we allow all of the following facts to be modelled:
+
+* This transaction occurred at some point after the given time (e.g. after a maturity event)
+* This transaction occurred at any time before the given time (e.g. before a bankruptcy event)
+* This transaction occurred at some point roughly around the given time (e.g. on a specific day)
+
+This same technique can be adapted to other types of oracle.
+
+Asserting occasionally varying data that is not publicly known
+--------------------------------------------------------------
+
+Sometimes you may want a fact that changes, but is not entirely continuous. Additionally the exact value may not be
+public, or may only be semi-public (e.g. easily available to some entities on the network but not all). An example of
+this would be a LIBOR interest rate fix.
+
+In this case, the following design can be used. The oracle service provides a query API which returns the current value,
+and a signing service that signs a transaction if the data in the command matches the answer being returned by the
+query API. Probably the query response contains some sort of timestamp as well, so the service can recognise values
+that were true in the past but no longer are (this is arguably a part of the fact being asserted).
+
+Because the signature covers the transaction, and transactions may end up being forwarded anywhere, the fact itself
+is independently checkable. However, this approach can be useful when the data itself costs money, because the act
+of issuing the signature in the first place can be charged for (e.g. by requiring the submission of a fresh
+``Cash.State`` that has been re-assigned to a key owned by the oracle service). Because the signature covers the
+*transaction* and not only the *fact*, this allows for a kind of weak pseudo-DRM over data feeds. Whilst a smart
+contract could in theory include a transaction parsing and signature checking library, writing a contract in this way
+would be conclusive evidence of intent to disobey the rules of the service (*res ipsa loquitur*). In an environment
+where parties are legally identifiable, usage of such a contract would by itself be sufficient to trigger some sort of
+punishment.
+
+Here is an extract from the ``NodeService.Oracle`` class and supporting types:
+
+.. sourcecode:: kotlin
+
+   /** A [FixOf] identifies the question side of a fix: what day, tenor and type of fix ("LIBOR", "EURIBOR" etc) */
+   data class FixOf(val name: String, val forDay: LocalDate, val ofTenor: Duration)
+
+   /** A [Fix] represents a named interest rate, on a given day, for a given duration. It can be embedded in a tx. */
+   data class Fix(val of: FixOf, val value: BigDecimal) : CommandData
+
+   class Oracle {
+       fun query(queries: List<FixOf>): List<Fix>
+
+       fun sign(wtx: WireTransaction): DigitalSignature.LegallyIdentifiable
+   }
+
+Because the fix contains a timestamp (the ``forDay`` field), there can be an arbitrary delay between a fix being
+requested via ``query`` and the signature being requested via ``sign``.
+
+Implementing oracles in the framework
+-------------------------------------
+
+Implementation involves the following steps:
+
+1. Defining a high level oracle class, that exposes the basic API operations.
+2. Defining a lower level service class, that binds network messages to the API.
+3. Defining a protocol using the :doc:`protocol-state-machines` framework to make it easy for a client to interact
+   with the oracle.
+
+An example of how to do this can be found in the ``NodeInterestRates.Oracle``, ``NodeInterestRates.Service`` and
+``RateFixProtocol`` classes. The exact details of how this code works will change in future, so for now consulting
+the protocols tutorial and the code for the server-side oracles implementation will have to suffice. There will be more
+detail added once the platform APIs have settled down.
+
+Currently, there's no network map service, so the location and identity keys of an oracle must be distributed out of
+band.

docs/source/roadmap.rst

@@ -1,37 +0,0 @@
-Roadmap
-=======
-
-The canonical place to learn about pending tasks is the `R3 JIRA <https://r3-cev.atlassian.net/>`_ site. This
-page gives some examples of tasks that we wish to explore in future milestones as part of proving (or disproving)
-our core thesis
-
-Data distribution and management:
-
-* Introduce a pluggable network messaging backend with a mock implementation for testing, and an Apache Kafka based
-  implementation for bringing up first networking capability. Using Kafka as a message routing/storage layer is not
-  necessarily the final approach or suitable for P2P WAN messaging, but it should be a good next step for prototyping
-  and may even be a useful for internal deployments.
-* Flesh out the core code enough to have a server that downloads and verifies transactions as they are uploaded to the
-  cluster. At this stage all transactions are assumed to be public to the network (this will change later). Some basic
-  logging/JMX/monitoring dashboard should be present to see what the node is doing.
-* Experimentation with block-free conflict/double spend resolution using a voting pool of *observers* with lazy consensus.
-  Logic for rolling back losing transaction subgraphs when a conflict is resolved, reporting these events to observer
-  APIs and so on.
-* Support a pluggable storage layer for recording seen transactions and their validity states.
-
-Contracts API:
-
-* Upgrades to the composability of contracts: demonstrate how states can require the presence of other states as a way
-  to mix in things like multi-signature requirements.
-* Demonstrate how states can be separated into two parts, the minimum necessary for conflict resolution (e.g. owner keys)
-  and a separated part that contains data useful for auditing and building confidence in the validity of a transaction
-  (e.g. amounts).
-* Explorations of improved time handling, and how best to express temporal logic in the contract API/DSL.
-
-JVM adaptations:
-
-* Implement a sandbox and packaging system for contract logic. Contracts should be distributable through the network
-  layer.
-* Experiment with modifications to HotSpot to allow for safely killing threads (i.e. fixing the issues that make
-  Thread.stop() unsafe to use), and to measure and enforce runtime limits to handle runaway code.
-