* CORDA-3601 Record a flow's finish time
Record a flow's finish time by updating its metadata record. It is set
in `updateCheckpoint` by checking the status of the checkpoint. If it is
`COMPLETED` it will set the `finishInstant` on the metadata object and
update it.
* CORDA-3601 Record flow finish time for all finished statuses
Update the flow finish time for the following statuses:
- COMPLETED
- KILLED
- FAILED
* CORDA-3601 Use platform clock in `DBCheckpointStorage`
* CORDA-3669 Do not execute `ExecuteAsyncOperation` multiple times
When a `FlowExternalOperation` or `FlowExternalAsyncOperation` executes
and completes a flag (`isFlowResumed`) is switched to true.
This flag was used inside of `DoRemainingWorkTransition` to decide
whether to skip over the execution of an event.
Since this flag was being switched to true when the external operation's
future completed, it was possible for _unexpected_ events to be placed
in the fiber's queue that would retrigger the
`FlowIORequest.ExecuteAsyncOperation`, that is held as the checkpoint's
next `FlowIORequest`to process.
By using the existing `StateMachineState.isTransactionTracked` (and
renaming it to `isWaitingForFuture`) we can decide to not process the
`FlowIORequest.ExecuteAsyncOperation` if it has already been called
before. This moves this code path in line with
`FlowIORequest.WaitForLedgerCommit`.
Random `DoRemainingWork` events can now be pushed to the fiber's queue
without causing the `FlowIORequest.ExecuteAsyncOperation` to execute
again.
* CORDA-3596 Record flow metadata
Record flow metadata during the zero'th checkpoint that occurs before
calling the flow's `call` function.
This required adding an RPC call's arguments to the `InvocationContext`
that gets created. These arguments are then accessible within the
statemachine and from the `Checkpoint` class. The arguments are then
extracted when recording a flow's metadata inside of
`DBCheckpointStorage`.
Updated the size of the started by column to 128 since it was not long
enough to hold the fully qualified class of a service that started a
flow.
* CORDA-3596 Remove arguments from in-memory checkpoint
When executing a flows first real suspend (from flow code) the arguments
contained in the `InvocationContext` are removed. This saves holding
these arguments for the whole lifecyle of a flow.
* CORDA-3596 Increase `cordapp_name` column to 128
* CORDA-3596 Join metadata by `flow_id`
Due to changes in where metadata is recorded, there is no need for
having `invocation_id` as the metadata table's primary key. The
`flow_id` is now the primary key of the table and is used to join to the
main checkpoints table.
The `invocation_id` has been removed from the checkpoints table since it
is not needed for the join anymore.
* CORDA-3596 Remove `received_time` from metadata table
* CORDA-3596 Remove unused `StartReason` enum
* CORDA-3596 Simple `DBCheckpointStorageTests` for metadata
* CORDA-3596 Truncate really long flow names
* CheckpointStorage.getAllCheckpoints will not fetch COMPLETED, FAILED and KILLED flows by default
* Rename getAllCheckpoints to getAllRunnableCheckpoints for clarity
* Fix Detekt issue
* Rename getAllRunnableCheckpoints to getRunnableCheckpoints
* Minor kdoc update
* Bring back in CheckpointStorage.getAllCheckpoints to co-exist with getRunnableCheckpoints
* Add progress tracker information to checkpoint
The checkpoint Datebase is updated when the statemachine suspends
with the progress trackers current step name. This is truncated if
it is longer than the Database column.
* Minor rename in statemachine for clarity
* Set/ Reset Checkpoint.status to RUNNABLE after when suspending
* Removing/ Moving comment as it makes no longer sense to be there since, we now always create a new Checkpoint object in SingleThreadedStateMachineManager.createFlowFromCheckpoint through tryDeserializeCheckpoint
* Set -in memory- Checkpoint.status to RUNNABLE when a flow is retrying from Checkpoint
Due to a change in how messaging works, `ActionExecutorImpl
.executeSendInitial` was no longer being called. Changing the byteman
script to throw exception on hits to `ActionExecutorImpl
.executeSendMultiple` allowed the tests to pass.
When a flow is finished do not delete the checkpoint from the DB.
Instead, the FlowStatus is marked as Completed in the DB.
Updated numerous tests which relied on the flow being removed
when finished.
* Update Checkpoint DB to update flow io request
* Modify flow monitor to update Checkpoint DB with waiting flows
This happens periodically.
* Refactored code to avoid looping twice and updated tests
* Fix tests after rebasing
* Fix MR comments (non-functional refactor of tests + FlowMonitor).
* Made visible for testing method private in DBCheckpointStorage
This is not needed anymore.
* Explicity check if ioRequestType has changed in update method
* Fix shadowing warning
* Import non deprecated Assert into test
* Use AssertEquals not assert in test
* Address more comments (minor refactor) of DBCheckpointStorage
* Minor fix use it instead of referencing object explicitly
* Add null check to DBCheckpointStorage
* Revert changes to Flow Monitor.
We will instead store the information in the main thread of the
state machine.
* Remove now uneeded API and make statemachine update ioRequest
* Add Integration Test to check statemachine updates DB on Recieve
* Use simpleName in checkpoint storage instead of class.
Hibernate was previously resetting the class field this is now
set to null (when getting checkpoint form DB) and a new method
for getting back the simple name as a string.
* Update StateMachineState to store simple name.
* Fix after rebase broke stuff + renamed test
* Fix Detekt issue
* Remove uneeded null assertion
* [CORDA-3628] - Implement sendAll API
* detekt
* Some minor refactorings and docs
* Eliminate warnings
* Address Rick's comments
* Switch sendAll to use a set
Do not cascade updates to checkpoint error and result tables to hopefully
improve database performance moving forward. Because the joined tables
are no longer being updated by updating the main `DBFlowCheckpoint` entity,
they must be created/updated/deleted manually.
The checkpoint blobs still cascade as they pretty much always evolve in
tandem with the main checkpoint table.
* CORDA-3651: addManifest now uses separate files for reading and writing.
* CORDA-3651: The jar scanning loader now closes itsself.
Co-authored-by: Adel El-Beik <adelel-beik@19LDN-MAC108.local>
* Split out node-api tests that require test-utils/node-driver
* Add node-api test artefacts to publication list.
* Make test-common a transient dependency - downstream tests assume that it's available.
* Switch dependencies to java-library
* Fix magic package name for cordapp scanning in test
* CORDA-3644: Scan the CorDapp classloader directly for SerializationWhitelist.
* CORDA-3644: Filter CorDapps from out-of-process node classpaths by their manifest attributes. Also exclude directories and blatant test artifacts.
* Fix IRS Demo - its "tests" artifact had a non-standard classifier of "test".
* Remove unused dependencies from test-common
* Explicit imports and formatting
* Add core-test-utils project
* Add dependency
* Move Kryo serialization context to node-api (not serialization as we do not want to pull kryo into the serialization lib)
* Move AMQP server serialization scheme to node api
* Move serialization tests to node-api
* Move internal test helpers without further dependencies.
* Move out some types from RPCClientProxyHandler to node-api in preparation for moving the AMQP scheme
* Move client AMQP context to node-api so we can move the test serialization rule out.
* Move InternalSerializationTestHelpers to core-test-utils
* Moved testing.core to core-test-utils
* Make detekt happy
* Add api-scanner to core-test-utils
* Remove inlined package names introduced by IntelliJ refactoring
* Update api-current.txt to account for reordering.
* Add core-test-utils to list of published artifacts.
* Add missing import
* Location of things in api text has moved again (publish name of artefact?)
* Revert all additions to the API, leaving just the reordering
* Code review: fix up core-test-utils build.gradle and introduce kryo version constant.
* Remove OpenSsl flag from ssl config stub (can't be used from node-api)
* Suppress detekt warning
* Move core test util tests to the right module
* Expose kotlin test as a transient dependency - projects have come to rely on that.
* Fix typo in package name
* Replace old Checkpoint table with new one.
Adds some of the new fields into the table where needed (I have
guessed this stuff but we can update it as we go along).
* Fix database constraints + name table correctly opps.
* Fixed typos in Liquidbase script
Also corrected constraints and added missed fields in hibernate
checkpoint class and liquibase scripts.
* Update CheckpointStorage to pass in serialization context.
This is cleaner than passing both the checkpoint and the
serialized checkpoint into the methods. Also fixed
CordaPersistanceServiceTests which I accidentally broke.
* Fix detekt problem
* Revert "Update CheckpointStorage to pass in serialization context."
This reverts commit b71e78f202.
* Fix test broken by reverting commit
* CORDA-3597 Update metadata join, timestamp columns and serialization
- Change the metadata join to the checkpoints table to use
`invocation_id` instead of `flow_id`. There were issues joining
between the tables because `flow_id` was not the primary key of the
metadata table. Switching over to `invocation_id` has at least allowed
us to bypass this issue. The information about the `invocation_id` is
stored in the `Checkpoint` class which makes it simple to save at
runtime.
- Some of timestamp columns were nullable when they should always be
populated, the nullable flags have now been removed.
- Previously the whole checkpoint was being serialized and stored into
the `checkpoints_blob.checkpoint` column. This meant duplicated saving
as the `flow_state` was contained in this object. Only the
`CheckpointState` property of `Checkpoint` is now being serialized and
saved to this field. Furthermore, it now uses the default
`STORAGE_CONTEXT` serialization (AMQP) instead of Kryo (which is only
used for serializing the `flow_state` / flow stack).
- The checkpoint database performance metrics recording has been
abstracted to its own class.
* CORDA-3597 Make metadata join non optional
Remove the nullable declaration on the metadata field of
`DBFlowCheckpoint`
* CORDA-3597 Rename `node_checkpoints_blobs` to `node_checkpoint_blobs`
* CORDA-3597 Update some kdocs
Co-authored-by: Dan Newton <danknewton@hotmail.com>
* CORDA-3484: Now cope with 2 contract jars with same hash but different name, we just select one and use that.
* ENT-3584: Contract jars are now generated on the fly.
* CORDA-3584: Reverted changes to CordappProviderImpl. Exception is raised if node started with multiple jars with same hash.
* ENT-3584: Fixing test failure.
* CORDA-3584: Switch to test extension method instead of reflection to access internal member.
* ENT-3584: Address review comment. Dont fully qualify exception.
* CORDA-3584: Address review comment and converted lazy to a resettable one.
* CORDA-3584: Removed unused logger.
* CORDA-3584: Fixed visibility.
* CORDA-3584: Removed synchronized
* CORDA-3584: Removed CordappResolver
* CORDA-3584: Reverted change in gradle file and fixed test.
* CORDA-3584: Removed V3 from test description as it wasn't actually V3 specific.
* CORDA-3584: Address review comment. Let classes be garbage collected.
* Throw SQLException or PersistenceException plain, that may come out of an unsafe subscriber
* Add explanatory comment about why we changed Observer.tee to use unsafe subscribe
* Introducing not unsubscribing version of Rx.Subscriber
* Wrap PublishSubjects with FlowSafeSubjects in all tests that test Observer.tee
* Minor code formatting
* Make rawUpdates Rx.Observers not unsubscribe when accessed from CordaServices - Do not allow rawUpdates subscribing from flows
* Warning fix: Add else block to when statement
* Revert "Wrap PublishSubjects with FlowSafeSubjects in all tests that test Observer.tee"
This reverts commit e419af86
* Correcting log message
* Improve log message
* Add fiber's id to log message and exception message
* Added test, asserting FlowSafeSubscriber is alive and re-accessed upon flow retry
* Logging flow name instead of flow id at VaultService.rawUpdates subscribing error
* Add kdoc to OnNextFailedException
* Minor text correction
* Update kdocs of FlowSafeSubject/ PreventSubscriptionsSubject
* Moved FlowSafeSubject under package node.internal as it is only used by NodeVaultService
* Add comment and update kdoc explaining how to subscribe with SafeSubscriber to FlowSafeSubject
* Change PreventSubscriptionsSubject#errorAction to be more specific; to return an Exception
* Minor text update
* Update messy comment
* Replace assertThat with assertEquals
* Splitting heartBeat to heartBeat1 and hearBeat2 for more clear asserting
* Correcting comment
* Update messy comment
* Splitting heartBeat into heartBeatOnNext and heartBeatOnError
* Update test name
* Add explanatory comment to test
* Update test name
* Update test and add test comment
* Moving NotarisedTxs from SendStateFlow to VaultObserverExceptionTest inside NodeHandle.getNotarisedTransactionIds
* Moving SubscribingRawUpdatesFlow from ErrorHandling to VaultObserverExceptionTest
* Update kdoc of FlowSafeSubscriber and FlowSafeSubscriber.onNext
* Make kdoc more clear
* Throw exception upon accessing VaultService.rawUpdates from within a flow
* Changing exception thrown when accessing VaultService.rawUpdates from within a flow to a CordaRuntimeException
* Minor kdoc update
* Update test comment
* Update kdoc of FlowSafeSubscriber
* Introducing Observable.flowSafeSubscribe public API method to subscribe with -non unsubscribing- Rx.Subscribers to Observables. It also replaced FlowSafeSubject
* Move CustomSafeSubscriber outside test methods
* Minor text update
* Add timeout to tests
* Update kdoc of flowSafeSubscribe
* Update kdoc of flowSafeSubscribe
* Update kdoc of flowSafeSubscribe
* Move FlowSafeSubscriber and flowSafeSubscribe under their own package
* Fix detekt issue
* Update Detekt baseline
* Revert "Update Detekt baseline"
This reverts commit 793a8ed9
* Fix Detekt issue
* Moved strictMode flag from flowSafeSubscribe to OnFlowSafeSubscribe
Moved OnFlowSafeSubscribe into internal package
Integration tested flowSafeLooseSubscribe
* Suppress Rx Deprecation
* Rename flowSafeSubscribe to flowSafeObservable
* Renaming flowSafeObservable to continueOnError and FlowSafeSubscriber to ResilientSubscriber
* Split StateMachine State into 2 classes
The idea is this better reflects the database structure. Added a
few helper methods to copy and update state.
* Doc + Improve Checkpoint API
* Rename methods to be more clear
