Added more docs for reference states to key concepts and api docs. (#4013)

* Added more docs for reference states to key concepts and api docs.

* Updated with requested changes

docs/source/api-states.rst

@@ -151,4 +151,46 @@ Where:
 * ``notary`` is the notary service for this state
 * ``encumbrance`` points to another state that must also appear as an input to any transaction consuming this
   state
-* ``constraint`` is a constraint on which contract-code attachments can be used with this state
+* ``constraint`` is a constraint on which contract-code attachments can be used with this state
+
+Reference States
+----------------
+
+A reference input state is a ``ContractState`` which can be referred to in a transaction by the contracts of input and
+output states but whose contract is not executed as part of the transaction verification process. Furthermore,
+reference states are not consumed when the transaction is committed to the ledger but they are checked for
+"current-ness". In other words, the contract logic isn't run for the referencing transaction only. It's still a normal
+state when it occurs in an input or output position.
+
+Reference data states enable many parties to reuse the same state in their transactions as reference data whilst
+still allowing the reference data state owner the capability to update the state. A standard example would be the
+creation of financial instrument reference data and the use of such reference data by parties holding the related
+financial instruments.
+
+Just like regular input states, the chain of provenance for reference states is resolved and all dependency transactions
+verified. This is because users of reference data must be satisfied that the data they are referring to is valid as per
+the rules of the contract which governs it and that all previous participants of teh state assented to updates of it.
+
+**Known limitations:**
+
+*Notary change:* It is likely the case that users of reference states do not have permission to change the notary
+assigned to a reference state. Even if users *did* have this permission the result would likely be a bunch of
+notary change races. As such, if a reference state is added to a transaction which is assigned to a
+different notary to the input and output states then all those inputs and outputs must be moved to the
+notary which the reference state uses.
+
+If two or more reference states assigned to different notaries are added to a transaction then it follows that this
+transaction cannot be committed to the ledger. This would also be the case for transactions not containing reference
+states. There is an additional complication for transaction including reference states, however. It is unlikely that the
+party using the reference states has the authority to change the notary for the state (in other words, the party using the
+reference state would not be listed as a participant on it). Therefore, it is likely that a transaction containing
+reference states with two different notaries cannot be committed to the ledger.
+
+As such, if reference states assigned to multiple different notaries are added to a transaction builder
+then the check below will fail.
+
+        .. warning:: Currently, encumbrances should not be used with reference states. In the case where a state is
+                     encumbered by an encumbrance state, the encumbrance state should also be referenced in the same
+                     transaction that references the encumbered state. This is because the data contained within the
+                     encumbered state may take on a different meaning, and likely would do, once the encumbrance state
+                     is taken into account.

docs/source/api-transactions.rst

@@ -94,15 +94,6 @@ to "walk the chain" and verify that each input was generated through a valid seq
 Reference input states
 ~~~~~~~~~~~~~~~~~~~~~~
 
-A reference input state is a ``ContractState`` which can be referred to in a transaction by the contracts of input and
-output states but whose contract is not executed as part of the transaction verification process. Furthermore,
-reference states are not consumed when the transaction is committed to the ledger but they are checked for
-"current-ness". In other words, the contract logic isn't run for the referencing transaction only. It's still a normal
-state when it occurs in an input or output position.
-
-Reference data states enable many parties to "reuse" the same state in their transactions as reference data whilst
-still allowing the reference data state owner the capability to update the state.
-
 A reference input state is added to a transaction as a ``ReferencedStateAndRef``. A ``ReferencedStateAndRef`` can be
 obtained from a ``StateAndRef`` by calling the ``StateAndRef.referenced()`` method which returns a
 ``ReferencedStateAndRef``.
@@ -123,20 +114,32 @@ obtained from a ``StateAndRef`` by calling the ``StateAndRef.referenced()`` meth
             :end-before: DOCEND 55
             :dedent: 12
 
-**Known limitations:**
+**Handling of update races:**
 
-*Notary change:* It is likely the case that users of reference states do not have permission to change the notary assigned
-to a reference state. Even if users *did* have this permission the result would likely be a bunch of
-notary change races. As such, if a reference state is added to a transaction which is assigned to a
-different notary to the input and output states then all those inputs and outputs must be moved to the
-notary which the reference state uses.
+When using reference states in a transaction, it may be the case that a notarisation failure occurs. This is most likely
+because the creator of the state (being used as a reference state in your transaction), has just updated it.
 
-If two or more reference states assigned to different notaries are added to a transaction then it follows
-that this transaction likely *cannot* be committed to the ledger as it unlikely that the party using the
-reference state can change the assigned notary for one of the reference states.
+Typically, the creator of such reference data will have implemented flows for syndicating the updates out to users.
+However it is inevitable that there will be a delay between the state being used as a reference being consumed, and the
+nodes using it receiving the update.
 
-As such, if reference states assigned to multiple different notaries are added to a transaction builder
-then the check below will fail.
+This is where the ``WithReferencedStatesFlow`` comes in. Given a flow which uses reference states, the
+``WithReferencedStatesFlow`` will execute the the flow as a subFlow. If the flow fails due to a ``NotaryError.Conflict``
+for a reference state, then it will be suspended until the state refs for the reference states are consumed. In this
+case, a consumption means that:
+
+1. the owner of the reference state has updated the state with a valid, notarised transaction
+2. the owner of the reference state has shared the update with the node attempting to run the flow which uses the
+   reference state
+3. The node has successfully committed the transaction updating the reference state (and all the dependencies), and
+   added the updated reference state to the vault.
+
+At the point where the transaction updating the state being used as a reference is committed to storage and the vault
+update occurs, then the ``WithReferencedStatesFlow`` will wake up and re-execute the provided flow.
+
+.. warning:: Caution should be taken when using this flow as it facilitates automated re-running of flows which use
+             reference states. The flow using reference states should include checks to ensure that the reference data is
+             reasonable, especially if the economics of the transaction depends upon the data contained within a reference state.
 
 Output states
 ^^^^^^^^^^^^^

docs/source/index.rst

@@ -79,6 +79,7 @@ We look forward to seeing what you can do with Corda!
    design/kafka-notary/design.md
    design/monitoring-management/design.md
    design/sgx-integration/design.md
+   design/reference-states/design.md
    design/sgx-infrastructure/design.md
    design/threat-model/corda-threat-model.md
    design/data-model-upgrades/signature-constraints.md

docs/source/key-concepts-states.rst

@@ -59,4 +59,13 @@ is aware of, and which it considers to be relevant to itself:
    :align: center
 
 We can think of the ledger from each node's point of view as the set of all the current (i.e. non-historic) states that
-it is aware of.
+it is aware of.
+
+Reference states
+----------------
+
+Not all states need to be updated by the parties which use them. In the case of reference data, there is a common pattern
+where one party creates reference data, which is then used (but not updated) by other parties. For this use-case, the
+states containing reference data are referred to as "reference states". Syntactically, reference states are no different
+to regular states. However, they are treated different by Corda transactions. See :doc:`key-concepts-transactions` for
+more details.

docs/source/key-concepts-transactions.rst

@@ -33,7 +33,7 @@ Here is an example of an update transaction, with two inputs and two outputs:
    :scale: 25%
    :align: center
 
-A transaction can contain any number of inputs and outputs of any type:
+A transaction can contain any number of inputs, outputs and references of any type:
 
 * They can include many different state types (e.g. both cash and bonds)
 * They can be issuances (have zero inputs) or exits (have zero outputs)
@@ -108,6 +108,18 @@ Each required signers should only sign the transaction if the following two cond
 If the transaction gathers all the required signatures but these conditions do not hold, the transaction's outputs
 will not be valid, and will not be accepted as inputs to subsequent transactions.
 
+Reference states
+----------------
+
+As mentioned in :doc:`key-concepts-states`, some states need to be referred to by the contracts of other input or output
+states but not updated/consumed. This is where reference states come in. When a state is added to the references list of
+a transaction, instead of the inputs or outputs list, then it is treated as a *reference state*. There are two important
+differences between regular states and reference states:
+
+* The specified notary for the transaction **does** check whether the reference states are current. However, reference
+  states are not consumed when the transaction containing them is committed to the ledger.
+* The contracts for reference states are not executed for the transaction containing them.
+
 Other transaction components
 ----------------------------
 As well as input states and output states, transactions contain: