* CORDA-3995 Redeliver external events in number of suspends differs
When retrying a flow, only redeliver external events held in a flow's
pending deduplication handlers if there is a difference in the
`numberOfSuspends` on the `currentState`'s checkpoint or the checkpoint
in the database.
If the checkpoint committed, but the flow retried, then the external
events would have been persisted to the database as part of the same
transaction. Therefore there is no need to replay them, as they have
already been processed as saved as part of the checkpoint.
This change is only relevant when the checkpoint persists, but the flow
still needs to retry after this occurs (within the same
transition/event).
* CORDA-3995 Redeliver external events in number of commits differs
When retrying a flow, only redeliver external events held in a flow's
pending deduplication handlers if there is a difference in the
`numberOfCommits` on the `currentState`'s checkpoint or the checkpoint
in the database.
If the checkpoint committed, but the flow retried, then the external
events would have been persisted to the database as part of the same
transaction. Therefore there is no need to replay them, as they have
already been processed as saved as part of the checkpoint.
This change is only relevant when the checkpoint persists, but the flow
still needs to retry after this occurs (within the same
transition/event).
* CORDA-3995 Redeliver external events if number of commits differs
When retrying a flow, only redeliver external events held in a flow's
pending deduplication handlers if there is a difference in the
`currentState`'s `numberOfCommits` or the `numberOfCommits`
the checkpoint has recorded in the database.
If the checkpoint committed, but the flow retried, then the external
events would have been persisted to the database as part of the same
transaction. Therefore there is no need to replay them, as they have
already been processed as saved as part of the checkpoint.
This change is only relevant when the checkpoint persists, but the flow
still needs to retry after this occurs (within the same
transition/event).
* Add @Suspendable to a test flow.
I am surprised this worked at all.
* Fix a few minor things based on review.
Co-authored-by: Will Vigor <william.vigor@r3.com>
We should check that PAUSED Checkpoints can be deserialised on node
startup as we do for RUNNABLE checkpoints. Otherwise a user might
get into trouble if they update the CorDapp.
We should not overwrite the stack trace of local errors thrown by
`FlowContinuation.Throw` as it hides the real cause of the error.
Exceptions received from peer nodes are still overwritten.
Flows that were started with a client id would hang because it would
retrieve the existing flow's future and wait for it to finish. But,
because the flow has failed its flow init and not saved its initial
checkpoint, it is relying on `startFlow` to start the flow again (by
redelivering the start flow external event).
`FlowWithClientIdStatus` now holds the flow id that it is related to.
This is then checked in `startFlow`. If a matching client id is found
for a flow start, it then checks the flow id as well. If the flow id
matches, then it lets the `startFlow` call continue, allowing it to
actually start the flow again (how a flow without a client id would
retry in this situation).
Previously we were just throwing this away when pausing, meaning
updates would not be passed back to the user.
The progress tracker is now maintained in the `NonResidentFlow`
allowing it to be reused in the flow when it is retried.
* ENT-5672 Update database query to get paused flows which have previously been hospitalised
* NOTICK Remove unneeded check if a database exception was removed when switching a flow to RUNNABLE since we were to remove it anyway
Always attempt to load a checkpoint from the database when a flow
retries.
This is to prevent transient errors where the checkpoint is committed to
the database but throws an error back to the node. When the node tries
to retry in this scenario, `isAnyCheckpointPersisted` is false, meaning
that it will try to insert when it tries to save its initial checkpoint
again.
By loading from the existing checkpoint, even though it doesn't
really use it because it is `Unstarted`, the flag gets put into the
right state and will update rather than insert later on.
The issue with the test was that the environment variable are kept as a static member so it passed if it was the first one to run, but failed if another test runs the config beforehand.
Terminate sessions that need to be removed instantly in whatever transition is currently executing, rather than scheduling another event and doing so at a later time.
To do this, update the transition being created in `TopLevelTransition` to remove the sessions and append the `RemoveSessionBindings` action to it.
This achieves the same outcome as the original code but does so with 1 less transition. Doing this also removes the race condition that can occur where another external event is added to the flow's event queue before the terminate event could be added.
* CORDA-3986 Increase sleep in `FlowSessionCloseTest`
A sleep duration needed to be increased to ensure that an end session
message has time to be processed by the other node.
Locks do not fully fix this because some internal processing needs to be
completed that can't be waited for using a lock. Therefore the sleep
time was increased generously.
* removed dependency from tools.jar
I removed the log line in /node/src/main/kotlin/net/corda/node/internal/NodeStartup.kt because I felt it was not so important
and I modified the checkpoint agent detection simply using a static field (I tested both with and without the checkpoint agent running and detection works correctly)
* move method to node-api to address review comments
Co-authored-by: Walter Oggioni <walter.oggioni@r3.com>
* CORDA-3954: Added step to run database migration scripts during initial node registration with -s / --skip-schema-creation option (default to false) to prevent migration.
* CORDA-3954: Applied code convention to if statement
* CORDA-3954: Marked NodeCmdLineOptions' -s/--skip-schema-creation as deprecated and hidden in line with --initial-registration
Disable `SignatureConstraintMigrationFromHashConstraintsTests.HashConstraint cannot be migrated to SignatureConstraint if a HashConstraint is specified for one state and another uses an AutomaticPlaceholderConstraint()` as it frequently appears in reports about Gradle process failures, to try isolating the actual cause.
* fix suggestion and tests
* detekt suppress
* making sure the forced join works with IndirectStatePersistable and removing unnecessary joinPredicates from parse with sorting
* remove joinPredicates and add tests
* rename sorting
* revert deleting joinPredicates and modify the force join to use `OR` instead of `AND`
* add system property switch
Fix memory leak due to DB not shutting down in FlowFrameworkPersistenceTests.flow restarted just after receiving payload.
Also reduces number of class-wide variables to reduce scope for references being accidentally held between runs.
`KillFlowTest` is failing quite often. This is probably due to issues in ordering when taking and releasing locks. By using `CountDownLatch` in places instead of `Semaphore`s should reduce the likelihood of tests failing.
If an error occurs when creating a transition (a.k.a anything inside of
`TopLevelTransition`) then resume the flow with the error that occurred.
This is needed, because the current code is swallowing all errors thrown
at this point and causing the flow to hang.
This change will allow better debugging of errors since the real error
will be thrown back to the flow and will get handled and logged by the
normal error code path.
Extra logging has been added to `processEventsUntilFlowIsResumed`, just
in case an exception gets thrown out of the normal code path. We do not
want this exception to be swallowed as it can make it impossible to
debug the original error.
Save the exception for flows that fail during session init when they are
kept for observation.
Change the exception tidy up logic to only update the flow's status if
the exception was removed.
Add `CordaRPCOps.reattachFlowWithClientId` to allow clients to reattach
to an existing flow by only providing a client id. This behaviour is the
same as calling `startFlowDynamicWithClientId` for an existing
`clientId`. Where it differs is `reattachFlowWithClientId` will return
`null` if there is no flow running or finished on the node with the same
client id.
Return `null` if record deleted from race-condition
Update the compatible flag in the DB if the flowstate cannot be deserialised.
The most common cause of this problem is if a CorDapp has been upgraded
without draining flows from the node.
`RUNNABLE` and `HOSPITALISED` flows are restored on node startup so
the flag is set for these then. The flag can also be set when a flow
retries for some reason (see retryFlowFromSafePoint) in this case the
problem has been caused by another reason.
Added a newpause event to the statemachine which returns an Abort
continuation and causes the flow to be moved into the Paused flow Map.
Flows can receive session messages whilst paused.
Add a lock to `StateMachineState`, allowing every flow to lock
themselves when performing a transition or when an external thread (such
as `killFlow`) tries to interact with a flow from occurring at the same
time.
Doing this prevents race-conditions where the external threads mutate
the database or the flow's state causing an in-flight transition to
fail.
A `Semaphore` is used to acquire and release the lock. A `ReentrantLock`
is not used as it is possible for a flow to suspend while locked, and
resume on a different thread. This causes a `ReentrantLock` to fail when
releasing the lock because the thread doing so is not the thread holding
the lock. `Semaphore`s can be used across threads, therefore bypassing
this issue.
The lock is copied across when a flow is retried. This is to prevent
another thread from interacting with a flow just after it has been
retried. Without copying the lock, the external thread would acquire the
old lock and execute, while the fiber thread acquires the new lock and
also executes.
