## Summary
This change deals with multiple issues:
* Errors that occur during flow initialisation.
* Errors that occur when handling the outcome of an existing flow error.
* Failures to rollback and close a database transaction when an error
occurs in `TransitionExecutorImpl`.
* Removal of create and commit transaction actions around retrying a flow.
## Errors that occur during flow initialisation
Flow initialisation has been moved into the try/catch that exists inside
`FlowStateMachineImpl.run`. This means if an error is thrown all the way
out of `initialiseFlow` (which should rarely happen) it will be caught
and move into a flow's standard error handling path. The flow should
then properly terminate.
`Event.Error` was changed to make the choice to rollback be optional.
Errors during flow initialisation cause the flow to not have a open
database transaction. Therefore there is no need to rollback.
## Errors that occur when handling the outcome of an existing flow error
When an error occurs a flow goes to the flow hospital and is given an
outcome event to address the original error. If the transition that was
processing the error outcome event (`StartErrorPropagation` and
`RetryFlowFromSafePoint`) has an error then the flow aborts and
nothing happens. This means that the flow is left in a runnable state.
To resolve this, we now retry the original error outcome event whenever
another error occurs doing so.
This is done by adding a new staff member that looks for
`ErrorStateTransitionException` thrown in the error code path of
`TransitionExecutorImpl`. It then takes the last outcome for that flow
and schedules it to run again. This scheduling runs with a backoff.
This means that a flow will continually retry the original error outcome
event until it completes it successfully.
## Failures to rollback and close a database transaction when an error occurs in `TransitionExecutorImpl`
Rolling back and closing the database transaction inside of
`TransitionExecutorImpl` is now done inside individual try/catch blocks
as this should not prevent the flow from continuing.
## Removal of create and commit transaction actions around retrying a flow
The database commit that occurs after retrying a flow can fail which
required some custom code just for that event to prevent inconsistent
behaviour. The transaction was only needed for reading checkpoints from
the database, therefore the transaction was moved into
`retryFlowFromSafePoint` instead and the commit removed.
If we need to commit data inside of `retryFlowFromSafePoint` in the
future, a commit should be added directly to `retryFlowFromSafePoint`.
The commit should occur before the flow is started on a new fiber.
The state machines state is held within `InnerState` which lived inside
the SMM. `InnerState` has been extracted out of the SMM to allow the SMM
to be refactored in the future. Smaller classes can now be made that
focus on a single goal as the locking of the state can be accessed from
external classes. To achieve this, pass the `InnerState` into the class
and request a lock if needed.
The locking of `InnerState` has been made a property of the `InnerState`
itself. It has a `lock` field that allows locks to be taken out when
needed.
An inline `withLock` function has been added to tidy up the code and not
harm performance.
Some classes have been made internal to prevent invalid usage of purely
node internal classes.
As part of this change, flow timeouts have been extracted out into
`FlowTimeoutScheduler`.
Only hit the database if `StateMachineState.isAnyCheckpointPersisted`
returns true. Otherwise, there will be no checkpoint to retrieve from the
database anyway. This can prevent errors due to a transient loss of
connection to the database.
Update tests after merging to 4.6
* Decouple DatabaseConfig and CordaPersistence etc.
* Add schema sync to schema migration + test
* Add command line parameters for synchronising schema