ENT-3161 - Align docs with ENT (#5113)

docs/source/corda-nodes-index.rst

@@ -13,6 +13,7 @@ Nodes
    node-database
    node-database-access-h2
    node-database-tables
+   node-operations-upgrade-cordapps
    shell
    clientrpc
    generating-a-node

docs/source/node-operations-upgrade-cordapps.rst

@@ -0,0 +1,74 @@
+Upgrading CorDapps on a node
+============================
+
+In order to upgrade a CorDapp on a node to a new version, it needs to be determined whether any backwards compatible
+changes have been made. These could range from database changes, to changes in the protocol.
+
+For developer information on upgrading CorDapps, see :doc:`upgrading-cordapps`.
+
+CorDapps must ship with database migration scripts or clear documentation about how to update the database to be compatible with the new version.
+
+
+Flow upgrades
+~~~~~~~~~~~~~
+
+If any backwards-incompatible changes have been made (see :ref:`upgrading-cordapps-backwards-incompatible-flow-changes`
+for more information), the upgrade method detailed below will need to be followed. Otherwise the CorDapp JAR can just
+be replaced with the new version.
+
+Contract and State upgrades
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are two types of contract/state upgrade:
+
+1. *Implicit:* By allowing multiple implementations of the contract ahead of time, using constraints. See
+   :doc:`api-contract-constraints` to learn more.
+2. *Explicit:* By creating a special *contract upgrade transaction* and getting all participants of a state to sign it using the
+   contract upgrade flows.
+
+This documentation only considers the *explicit* type of upgrade, as implicit contract upgrades are handled by the application.
+
+In an explicit upgrade contracts and states can be changed in arbitrary ways, if and only if all of the state's participants
+agree to the proposed upgrade. The following combinations of upgrades are possible:
+
+* A contract is upgraded while the state definition remains the same.
+* A state is upgraded while the contract stays the same.
+* The state and the contract are updated simultaneously.
+
+Running the upgrade
+~~~~~~~~~~~~~~~~~~~
+
+If a contract or state requires an explicit upgrade then all states will need updating to the new contract at a time agreed
+by all participants. The updated CorDapp JAR needs to be distributed to all relevant parties in advance of the changeover
+time.
+
+In order to perform the upgrade, follow the following steps:
+
+* If required, do a flow drain to avoid the definition of states or contracts changing whilst a flow is in progress (see :ref:`upgrading-cordapps-flow-drains` for more information)
+
+  * By RPC using the ``setFlowsDrainingModeEnabled`` method with the parameter ``true``
+  * Via the shell by issuing the following command ``run setFlowsDrainingModeEnabled enabled: true``
+
+* Check that all the flows have completed
+
+  * By RPC using the ``stateMachinesSnapshot`` method and checking that there are no results
+  * Via the shell by issuing the following command ``run stateMachinesSnapshot``
+
+* Once all flows have completed, stop the node
+* Replace the existing JAR with the new one
+* Make any database changes required to any custom vault tables for the upgraded CorDapp,
+  following the database upgrade steps in :doc:`node-operations-cordapp-deployment`.
+  The database update for a CorDapp upgrade follows the same steps as database setup for a new CorDapp.
+
+* Restart the node
+* If you drained the node prior to upgrading, switch off flow draining mode to allow the node to continue to receive requests
+
+  * By RPC using the ``setFlowsDrainingModeEnabled`` method with the parameter ``false``
+  * Via the shell by issuing the following command ``run setFlowsDrainingModeEnabled enabled: false``
+
+* Run the contract upgrade authorisation flow (``ContractUpgradeFlow$Initiate``) for each state that requires updating on every node.
+
+  * You can do this manually via RPC but for anything more than a couple of states it is assumed that a script will be
+    provided by the CorDapp developer to query the vault and run this for all states
+  * The contract upgrade initiate flow only needs to be run on one of the participants for each state. The flow will
+    automatically upgrade the state on all participants.

docs/source/upgrading-cordapps.rst

@@ -17,8 +17,26 @@ CorDapp versioning
 .. UPDATE - This is no longer accurate! Needs to talk about the different types of artifacts ( kernel, workflows) each versioned independently
 
 The Corda platform does not mandate a version number on a per-CorDapp basis. Different elements of a CorDapp are
-allowed to evolve separately. Sometimes, however, a change to one element will require changes to other elements. For
+allowed to evolve separately:
-example, changing a shared data structure may require flow changes that are not backwards-compatible.
+
+* States
+* Contracts
+* Services
+* Flows
+* Utilities and library functions
+* All, or a subset, of the above
+
+Sometimes, however, a change to one element will require changes to other elements. For example, changing a shared data
+structure may require flow changes that are not backwards-compatible.
+
+Areas of consideration
+----------------------
+This document will consider the following types of versioning:
+
+* Flow versioning
+* State and contract versioning
+* State and state schema versioning
+* Serialisation of custom types
 
 Flow versioning
 ---------------
@@ -32,8 +50,8 @@ The ``version`` property, which defaults to 1, specifies the flow's version. Thi
 whenever there is a release of a flow which has changes that are not backwards-compatible. A non-backwards compatible
 change is one that changes the interface of the flow.
 
-Defining a flow's interface
+What defines the interface of a flow?
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 The flow interface is defined by the sequence of ``send`` and ``receive`` calls between an ``InitiatingFlow`` and an
 ``InitiatedBy`` flow, including the types of the data sent and received. We can picture a flow's interface as follows:
 
@@ -59,8 +77,10 @@ As long as both the ``InitiatingFlow`` and the ``InitiatedBy`` flows conform to
 be implemented in any way you see fit (including adding proprietary business logic that is not shared with other
 parties).
 
-Non-backwards compatible flow changes
+.. _upgrading-cordapps-backwards-incompatible-flow-changes:
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+What constitutes a non-backwards compatible flow change?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 A flow can become backwards-incompatible in two main ways:
 
 * The sequence of ``send`` and ``receive`` calls changes:
@@ -70,8 +90,8 @@ A flow can become backwards-incompatible in two main ways:
 
 * The types of the ``send`` and ``receive`` calls changes
 
-Consequences of running flows with incompatible versions
+What happens when running flows with incompatible versions?
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Pairs of ``InitiatingFlow`` flows and ``InitiatedBy`` flows that have incompatible interfaces are likely to exhibit the
 following behaviour:
 
@@ -82,8 +102,24 @@ following behaviour:
 * One of the flows ends with an exception: "Counterparty flow terminated early on the other side", because one flow
   sends some data to another flow, but the latter flow has already ended
 
-Ensuring flow backwards-compatibility
+How do I upgrade my flows?
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+1. Update the flow and test the changes. Increment the flow version number in the ``InitiatingFlow`` annotation.
+2. Ensure that all versions of the existing flow have finished running and there are no pending ``SchedulableFlows`` on
+   any of the nodes on the business network. This can be done by *draining the node* (see below).
+3. Shut down the node.
+4. Replace the existing CorDapp JAR with the CorDapp JAR containing the new flow.
+5. Start the node.
+
+If you shut down all nodes and upgrade them all at the same time, any incompatible change can be made.
+
+In situations where some nodes may still be using previous versions of a flow and thus new versions of your flow may
+talk to old versions, the updated flows need to be backwards-compatible. This will be the case for almost any real
+deployment in which you cannot easily coordinate the roll-out of new code across the network.
+
+How do I ensure flow backwards-compatibility?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 The ``InitiatingFlow`` version number is included in the flow session handshake and exposed to both parties via the
 ``FlowLogic.getFlowContext`` method. This method takes a ``Party`` and returns a ``FlowContext`` object which describes
 the flow running on the other side. In particular, it has a ``flowVersion`` property which can be used to
@@ -127,8 +163,8 @@ This code shows a flow that in its first version expected to receive an Int, but
 expect a String. This flow is still able to communicate with parties that are running the older CorDapp containing
 the older flow.
 
-Handling interface changes to inlined subflows
+How do I deal with interface changes to inlined subflows?
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Here is an example of an in-lined subflow:
 
 .. container:: codeset
@@ -222,26 +258,10 @@ Flows which are not an ``InitiatingFlow`` or ``InitiatedBy`` flow, or inlined su
 ``InitiatingFlow`` or ``InitiatedBy`` flow, can be updated without consideration of backwards-compatibility. Flows of
 this type include utility flows for querying the vault and flows for reaching out to external systems.
 
-Performing flow upgrades
+.. _upgrading-cordapps-flow-drains:
-~~~~~~~~~~~~~~~~~~~~~~~~
 
-1. Update the flow and test the changes. Increment the flow version number in the ``InitiatingFlow`` annotation
+Flow drains
-2. Ensure that all versions of the existing flow have finished running and there are no pending ``SchedulableFlows`` on
+~~~~~~~~~~~
-   any of the nodes on the business network. This can be done by :ref:`draining_the_node`
-3. Shut down the node
-4. Replace the existing CorDapp JAR with the CorDapp JAR containing the new flow
-5. Start the node
-
-If you shut down all nodes and upgrade them all at the same time, any incompatible change can be made.
-
-In situations where some nodes may still be using previous versions of a flow and thus new versions of your flow may
-talk to old versions, the updated flows need to be backwards-compatible. This will be the case for almost any real
-deployment in which you cannot easily coordinate the roll-out of new code across the network.
-
-.. _draining_the_node:
-
-Draining the node
-~~~~~~~~~~~~~~~~~
 
 A flow *checkpoint* is a serialised snapshot of the flow's stack frames and any objects reachable from the stack.
 Checkpoints are saved to the database automatically when a flow suspends or resumes, which typically happens when
@@ -268,10 +288,9 @@ Contract and state versioning
 
 There are two types of contract/state upgrade:
 
-1. *Implicit:* By allowing multiple implementations of the contract ahead of time, using constraints. See
+1. *Implicit:* By allowing multiple implementations of the contract ahead of time, using constraints. See :doc:`api-contract-constraints` to learn more.
-   :doc:`api-contract-constraints` to learn more
+2. *Explicit:* By creating a special *contract upgrade transaction* and getting all participants of a state to sign it using the
-2. *Explicit:* By creating a special *contract upgrade transaction* and getting all participants of a state to sign it
+   contract upgrade flows.
-   using the contract upgrade flows
 
 The general recommendation for Corda 4 is to use **implicit** upgrades for the reasons described :ref:`here <implicit_vs_explicit_upgrades>`.
 
@@ -364,7 +383,7 @@ Have each node operator restart their node.
 
 8. Authorise the upgrade
 ^^^^^^^^^^^^^^^^^^^^^^^^
-Now that new states and contracts are on the classpath for all the relevant nodes, the nodes must all run the
+Now that new states and contracts are on the classpath for all the relevant nodes, the next step is for all node to run the
 ``ContractUpgradeFlow.Authorise`` flow. This flow takes a ``StateAndRef`` of the state to update as well as a reference
 to the new contract, which must implement the ``UpgradedContract`` interface.
 
@@ -373,7 +392,7 @@ At any point, a node administrator may de-authorise a contract upgrade by runnin
 
 9. Perform the upgrade
 ^^^^^^^^^^^^^^^^^^^^^^
-Once all nodes have performed the authorisation process, a **single** node must initiate the upgrade via the
+Once all nodes have performed the authorisation process, a participant must be chosen to initiate the upgrade via the
 ``ContractUpgradeFlow.Initiate`` flow for each state object. This flow has the following signature:
 
 .. sourcecode:: kotlin
@@ -403,21 +422,71 @@ Capabilities of the contract upgrade flows
   a ``Dog`` state, provided that all participants in the ``Cat`` state agree to the change
 * If a node has not yet run the contract upgrade authorisation flow, they will not be able to upgrade the contract
   and/or state objects
+* Upgrade authorisations can subsequently be deauthorised
+* Upgrades do not have to happen immediately. For a period, the two parties can use the old states and contracts
+  side-by-side
 * State schema changes are handled separately
 
+Writing new states and contracts
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+* If a property is removed from a state, any references to it must be removed from the contract code. Otherwise, you
+  will not be able to compile your contract code. It is generally not advisable to remove properties from states. Mark
+  them as deprecated instead
+* When adding properties to a state, consider how the new properties will affect transaction validation involving this
+  state. If the contract is not updated to add constraints over the new properties, they will be able to take on any
+  value
+* Updated state objects can use the old contract code as long as there is no requirement to update it
+
+Permissioning
+^^^^^^^^^^^^^
+* Only node administrators are able to run the contract upgrade authorisation and deauthorisation flows
+
 Logistics
 ^^^^^^^^^
-* All nodes need to run the contract upgrade authorisation flow to upgrade the contract and/or state objects
+* All nodes need to run the contract upgrade authorisation flow
-* Only node administrators are able to run the contract upgrade authorisation and deauthorisation flows
-* Upgrade authorisations can subsequently be deauthorised
 * Only one node should run the contract upgrade initiation flow. If multiple nodes run it for the same ``StateRef``, a
   double-spend will occur for all but the first completed upgrade
-* Upgrades do not have to happen immediately. For a period, the two parties can use the old states and contracts
-  side-by-side
 * The supplied upgrade flows upgrade one state object at a time
 
-State schema versioning
+Serialisation
------------------------
+-------------
+
+Currently, the serialisation format for everything except flow checkpoints (which uses a Kryo-based format) is based
+upon AMQP 1.0, a self-describing and controllable serialisation format. AMQP is desirable because it allows us to have
+a schema describing what has been serialized alongside the data itself. This assists with versioning and deserialising
+long-ago archived data, among other things.
+
+Writing classes
+~~~~~~~~~~~~~~~
+Although not strictly related to versioning, AMQP serialisation dictates that we must write our classes in a particular way:
+
+* Your class must have a constructor that takes all the properties that you wish to record in the serialized form. This
+  is required in order for the serialization framework to reconstruct an instance of your class
+* If more than one constructor is provided, the serialization framework needs to know which one to use. The
+  ``@ConstructorForDeserialization`` annotation can be used to indicate the chosen constructor. For a Kotlin class
+  without the ``@ConstructorForDeserialization`` annotation, the primary constructor is selected
+* The class must be compiled with parameter names in the .class file. This is the default in Kotlin but must be turned
+  on in Java (using the ``-parameters`` command line option to ``javac``)
+* Your class must provide a Java Bean getter for each of the properties in the constructor, with a matching name. For
+  example, if a class has the constructor parameter ``foo``, there must be a getter called ``getFoo()``. If ``foo`` is
+  a boolean, the getter may optionally be called ``isFoo()``. This is why the class must be compiled with parameter
+  names turned on
+* The class must be annotated with ``@CordaSerializable``
+* The declared types of constructor arguments/getters must be supported, and where generics are used the generic
+  parameter must be a supported type, an open wildcard (*), or a bounded wildcard which is currently widened to an open
+  wildcard
+* Any superclass must adhere to the same rules, but can be abstract
+* Object graph cycles are not supported, so an object cannot refer to itself, directly or indirectly
+
+Writing enums
+~~~~~~~~~~~~~
+Elements cannot be added to enums in a new version of the code. Hence, enums are only a good fit for genuinely static
+data that will never change (e.g. days of the week). A ``Buy`` or ``Sell`` flag is another. However, something like
+``Trade Type`` or ``Currency Code`` will likely change. For those, it is preferable to choose another representation,
+such as a string.
+
+State schemas
+-------------
 By default, all state objects are serialised to the database as a string of bytes and referenced by their ``StateRef``.
 However, it is also possible to define custom schemas for serialising particular properties or combinations of
 properties, so that they can be queried from a source other than the Corda Vault. This is done by implementing the