mirror of
https://github.com/corda/corda.git
synced 2024-12-20 05:28:21 +00:00
b0903efa50
* Introducing a new type of exception and a new hospital staff member to pause flows by immediately hospitalising them. * Renaming exception to "HospitalizeFlowException". * Making HospitalizeFlowException an open class. * Overloading constructors of HospitalizeFlowException to be available in Java. * Using Throwable#mentionsThrowable. * Moving HospitalizeFlowException in its own file. * Update kdocs for HospitalizeFlowException and StaffedFlowHospital#SedationNurse. * Added tests, testing various HospitalizeFlowException types thrown. * Fix Detekt issues. * Imports optimizing. * Add safe casting. * Update api-flows and node-flow-hospital docs. * Minor code comment change. * Add DOCSTART-DOCEND signs in HospitalizeFlowException for makeDocs. It is referenced by api-flows.rst. * Minor change in note. * Code formatting. * Remove comment. * Remove if statement that makes example worse. * Remove redundant comment. * Moving 'Internal Corda errors' at the bottom. * Changing node-flow-hospital.rst as per review. * Change HospitalizeFlowException description as per review. * Adding an example for FlowException. * Minor indentation fix. * Update FlowException example label as per review. * Correcting handling of custom exception.
82 lines
5.1 KiB
ReStructuredText
82 lines
5.1 KiB
ReStructuredText
Flow Hospital
|
||
=============
|
||
|
||
Overview
|
||
--------
|
||
|
||
The **flow hospital** refers to a built-in node service that manages flows that have encountered an error.
|
||
|
||
This service is responsible for recording, tracking, diagnosing, recovering and retrying flows. It determines whether errored flows should be retried
|
||
from their previous checkpoints or have their errors propagate. Flows may be recoverable under certain scenarios (eg. manual intervention
|
||
may be required to install a missing contract JAR version). For a given errored flow, the flow hospital service determines the next course of
|
||
action towards recovery and retry.
|
||
|
||
.. note:: If the raised exception cannot be handled from the hospital, it will be propagated to the application code.
|
||
If the exception is not handled by the application code either, then the flow will terminate and any records of it will be removed from the hospital.
|
||
Below, you can find a list of the errors that are handled by the hospital.
|
||
|
||
This concept is analogous to *exception management handling* associated with enterprise workflow software, or
|
||
*retry queues/stores* in enterprise messaging middleware for recovering from failure to deliver a message.
|
||
|
||
Functionality
|
||
-------------
|
||
|
||
Flow hospital functionality is enabled by default in |release|. No explicit configuration settings are required.
|
||
|
||
There are two aspects to the flow hospital:
|
||
|
||
- run-time behaviour in the node upon failure, including retry and recovery transitions and policies.
|
||
- visualisation of failed flows in the Explorer UI.
|
||
|
||
.. _flow-hospital-runtime:
|
||
|
||
Run-time behaviour
|
||
~~~~~~~~~~~~~~~~~~
|
||
|
||
Specifically, there are two main ways a flow is hospitalized:
|
||
|
||
1. A counterparty invokes a flow on your node that isn’t installed (i.e. missing CorDapp):
|
||
this will cause the flow session initialisation mechanism to trigger a ``ClassNotFoundException``.
|
||
If this happens, the session initiation attempt is kept in the hospital for observation and will retry if you restart the node.
|
||
Corrective action requires installing the correct CorDapp in the node's "cordapps" directory.
|
||
|
||
.. warning:: There is currently no retry API. If you don’t want to install the cordapp, you should be able to call `killFlow` with the UUID
|
||
associated with the failing flow in the node's log messages.
|
||
|
||
2. Once started, if a flow experiences an error, the following failure scenarios are handled:
|
||
|
||
* ``SQLException`` mentioning a **deadlock**:
|
||
If this happens, the flow will retry. If it retries more than once, a back off delay is applied to try and reduce contention.
|
||
Current policy means these types of failed flows will retry forever (unless explicitly killed). No intervention required.
|
||
|
||
* **Database constraint violation** (``ConstraintViolationException``):
|
||
This scenario may occur due to natural contention between racing flows as Corda delegates handling using the database's optimistic concurrency control.
|
||
If this exception occurs, the flow will retry. After retrying a number of times, the errored flow is kept in for observation.
|
||
|
||
* ``SQLTransientConnectionException``:
|
||
Database connection pooling errors are dealt with. If this exception occurs, the flow will retry. After retrying a number of times, the errored flow is kept in for observation.
|
||
|
||
* All other instances of ``SQLException``:
|
||
Any ``SQLException`` that is thrown and not handled by any of the scenarios detailed above, will be kept in for observation after their first failure.
|
||
|
||
* **Finality Flow handling** - Corda 3.x (old style) ``FinalityFlow`` and Corda 4.x ``ReceiveFinalityFlow`` handling:
|
||
If on the receive side of the finality flow, any error will result in the flow being kept in for observation to allow the cause of the
|
||
error to be rectified (so that the transaction isn’t lost if, for example, associated contract JARs are missing).
|
||
Intervention is expected to be “rectify error, perhaps uploading attachment, and restart node” (or alternatively reject and call `killFlow`).
|
||
|
||
* ``FlowTimeoutException``:
|
||
This is used internally by the notary client flow when talking to an HA notary. It’s used to cause the client to try and talk to a different
|
||
member of the notary cluster if it doesn't hear back from the original member it sent the request to within a “reasonable” time.
|
||
The time is hard to document as the notary members, if actually alive, will inform the requester of the ETA of a response.
|
||
This can occur an infinite number of times. i.e. we never give up notarising. No intervention required.
|
||
|
||
* ``HospitalizeFlowException``:
|
||
The aim of this exception is to provide user code a way to retry a flow from its last checkpoint if a known intermittent failure occurred.
|
||
Any ``HospitalizeFlowException`` that is thrown and not handled by any of the scenarios detailed above, will be kept in for observation.
|
||
|
||
* **Internal Corda errors**:
|
||
Flows that experience errors from inside the Corda statemachine, that are not handled by any of the scenarios details above, will be retried a number of times
|
||
and then kept in for observation if the error continues.
|
||
|
||
.. note:: Flows that are kept in for observation are retried upon node restart.
|