Commit Graph

3112 Commits

Author SHA1 Message Date
Dan Newton
ac4907a429
CORDA-3721 Finishing + retrying a flow cancels its future (#6454)
Cancel the future being run by a flow when finishing or retrying it. The
cancellation of the future no longer cares about what type of future it
is.

`StateMachineState` has the `future` field, which holds the 3
(currently) possible types of futures:
- sleep
- wait for ledger commit
- async operation / external operation

Move the starting of all futures triggered by actions into
`ActionFutureExecutor`.
2020-07-14 08:04:52 +01:00
Chris Rankin
e684cfd1e6
NOTICK: Avoid configuring more tasks in the deterministic modules. (#6452) 2020-07-13 10:33:23 +01:00
Christian Sailer
057a8d8ae9
NOTICK fix smoke tests and slow integration tests (#6422)
* Add schema migration to smoke tests
* Fix driver to work correctly for out-of-proc node with persistent database.
Co-authored-by: Ross Nicoll <ross.nicoll@r3.com>
2020-07-09 15:13:20 +01:00
Dan Newton
52240dc3d1
Merge pull request #6451 from corda/dan/os-4.5-to-4.6-merge-2020-07-08
NOTICK OS 4.5 to 4.6 merge 2020-07-08
2020-07-09 14:02:22 +01:00
LankyDan
8b231dc936 NOTICK Suppress detekt warning on StaffedFlowHospital 2020-07-08 16:18:16 +01:00
LankyDan
b6d649634f ENT-5196 handle errors during flow initialisation (#6378)
Update flow error handling tests after merging from earlier releases
2020-07-08 16:02:17 +01:00
LankyDan
b05c0f0cc1 ENT-5196 handle errors during flow initialisation (#6378)
Changes to `StaffedFlowHospital` after merging from earlier releases.
2020-07-08 14:43:24 +01:00
LankyDan
2204f44332 ENT-5196 handle errors during flow initialisation (#6378)
Changes to `TopLevelTransition` after merging from earlier releases.

When a flow is kept for observation and its checkpoint is saved as
HOSPITALIZED in the database, we must acknowledge the session init and
flow start events so that they are not replayed on node startup.

Otherwise the same flow will be ran twice when the node is restarted,
one from the checkpoint and one from artemis.
2020-07-08 14:41:29 +01:00
Chris Rankin
72938fdf38
CORDA-3892: Upgrade to Gradle 5.6.4 again, with build fixes. (#6448) 2020-07-08 11:34:25 +01:00
LankyDan
fdae04fc28 Merge branch 'release/os/4.5' into dan/os-4.5-to-4.6-merge-2020-07-08 2020-07-08 10:44:47 +01:00
Chris Rankin
16eb9dfc08
NOTICK: Fix some Gradle technical debt. (#6430) 2020-07-07 09:08:58 +01:00
Kyriakos Tharrouniatis
b619356bff
Reduce exception_message and stack_trace lengths in table node_flow_exceptions (#6432)
Reduce exception_message and stack_trace lengths in table node_flow_exceptions from 4000 to 2000 to fix Oracle failing with: 'ORA-00910: specified length too long for its datatype'
2020-07-07 09:07:01 +01:00
Dan Newton
9a8ae0fd32
CORDA-3841 Update session init flow error handling tests (#6431)
These tests were removed after doing a merge from 4.4. They needed
updating after the changes from 4.4 anyway. These have been included in
this change.

Also fix kill flow tests and send initial tests.
2020-07-07 08:47:09 +01:00
Dan Newton
a1e1bf4e6d
CORDA-3848 Uncaught exception hospitalises flow (#6377)
When an uncaught exception propagates all the way to the flow exception
handler, the flow will be forced into observation/hospitalised.

The updating of the checkpoints status is done on a separate thread as
the fiber cannot be relied on anymore. The new thread is needed to allow
database transaction to be created and committed. Failures to the status
update will be rescheduled to ensure that this information is eventually
reflected in the database.
2020-07-06 22:43:48 +01:00
Ryan Fowler
0d5bed5243
ENT-5131: Avoid NPE by throwing a catchable exception when openAttachment fails. (#6408) 2020-07-06 11:42:36 +01:00
Ross Nicoll
6aa19723e6
INFRA-417 Improve driver DSL test stability (#6415)
* Move log messages that are not useful in typical usage from info to debug level to reduce log spam.
* Add node startup check before attempting to connect.
2020-07-03 20:42:29 +01:00
Dan Newton
6bc2c79e23
NOTICK NodeBasedTest take in cordapps (#6424)
In enterprise, `AuthDBTests` picked up a schema from a unit test and
included it in the cordapp it builds. This schema does not have a
migration and therefore fails the integration tests.

`NodeBasedTest` now lets cordapps to be defined and passed in to avoid
this issue. It defaults to making a cordapp from the tests base
directory if none are provided.
2020-07-02 16:14:51 +01:00
Denis Rekalov
adc0879e8e
CORDA-3867: Add tests for AMQ_VALIDATED_USER (#6418)
* CORDA-3867: Add tests for AMQ_VALIDATED_USER

* CORDA-3867: detekt
2020-07-02 09:20:23 +01:00
Ross Nicoll
9f12e6bbc5
INFRA-433 Rebuild node config parsing tests (#6423)
Replace node configuration parsing tests with lighter weight equivalents which just parse the configuration rather than starting a full node.
2020-07-01 22:47:09 +01:00
Dan Newton
5f4401d16a NOTICK Non-database error handling in withEntityManager (#6239)
When a non-database exception is thrown out of a `withEntityManager`
block, always check if the session needs to be rolled back.

This means if a database error is caught and a new non-database error is
thrown out of the `withEntityManager` block, the transaction is still
rolled back. The flow can then continue progressing as normal.
2020-07-01 18:18:21 +01:00
LankyDan
c06830d851 Merge branch 'release/os/4.4' into dan/os-4.4-to-4.5-merge-2020-07-01
# Conflicts:
#	core/src/main/kotlin/net/corda/core/node/ServiceHub.kt
#	node/src/integration-test-slow/kotlin/net/corda/node/services/statemachine/StatemachineGeneralErrorHandlingTest.kt
#	node/src/integration-test-slow/kotlin/net/corda/node/services/statemachine/StatemachineKillFlowErrorHandlingTest.kt
#	node/src/integration-test/kotlin/net/corda/node/flows/FlowEntityManagerTest.kt
#	node/src/main/kotlin/net/corda/node/internal/AbstractNode.kt
#	node/src/main/kotlin/net/corda/node/services/statemachine/TransitionExecutorImpl.kt
#	node/src/main/kotlin/net/corda/node/services/statemachine/interceptors/HospitalisingInterceptor.kt
2020-07-01 17:32:36 +01:00
Kyriakos Tharrouniatis
5499e2c050
ENT-5384 Rename MAX_SQL_IN_CLAUSE_SET (#6414)
Rename constant to `DEFAULT_SOFT_LOCKING_SQL_IN_CLAUSE_SIZE`
2020-07-01 09:31:12 +01:00
Dan Newton
efd633c7b9 CORDA-3722 withEntityManager can rollback its session (#6187)
* CORDA-3722 withEntityManager can rollback its session

Improve the handling of database transactions when using
`withEntityManager` inside a flow.

Extra changes have been included to improve the safety and
correctness of Corda around handling database transactions.

This focuses on allowing flows to catch errors that occur inside an
entity manager and handle them accordingly.

Errors can be caught in two places:

- Inside `withEntityManager`
- Outside `withEntityManager`

Further changes have been included to ensure that transactions are
rolled back correctly.

Errors caught inside `withEntityManager` require the flow to manually
`flush` the current session (the entity manager's individual session).
By manually flushing the session, a `try-catch` block can be placed
around the `flush` call, allowing possible exceptions to be caught.

Once an error is thrown from a call to `flush`, it is no longer possible
to use the same entity manager to trigger any database operations. The
only possible option is to rollback the changes from that session.
The flow can continue executing updates within the same session but they
will never be committed. What happens in this situation should be handled
by the flow. Explicitly restricting the scenario requires a lot of effort
and code. Instead, we should rely on the developer to control complex
workflows.

To continue updating the database after an error like this occurs, a new
`withEntityManager` block should be used (after catching the previous
error).

Exceptions can be caught around `withEntityManager` blocks. This allows
errors to be handled in the same way as stated above, except the need to
manually `flush` the session is removed. `withEntityManager` will
automatically `flush` a session if it has not been marked for rollback
due to an earlier error.

A `try-catch` can then be placed around the whole of the
`withEntityManager` block, allowing the error to be caught while not
committing any changes to the underlying database transaction.

To make `withEntityManager` blocks work like mini database transactions,
save points have been utilised. A new savepoint is created when opening
a `withEntityManager` block (along with a new session). It is then used
as a reference point to rollback to if the session errors and needs to
roll back. The savepoint is then released (independently from
completing successfully or failing).

Using save points means, that either all the statements inside the
entity manager are executed, or none of them are.

- A new session is created every time an entity manager is requested,
but this does not replace the flow's main underlying database session.
- `CordaPersistence.transaction` can now determine whether it needs
to execute its extra error handling code. This is needed to allow errors
escape `withEntityManager` blocks while allowing some of our exception
handling around subscribers (in `NodeVaultService`) to continue to work.
2020-06-30 11:54:16 +01:00
Dan Newton
f6b5737277
ENT-5196 handle errors during flow initialisation (#6378)
## Summary

This change deals with multiple issues:

* Errors that occur during flow initialisation.

* Errors that occur when handling the outcome of an existing flow error.

* Failures to rollback and close a database transaction when an error
occurs in `TransitionExecutorImpl`.

* Removal of create and commit transaction actions around retrying a flow.

## Errors that occur during flow initialisation

Flow initialisation has been moved into the try/catch that exists inside
`FlowStateMachineImpl.run`. This means if an error is thrown all the way
out of `initialiseFlow` (which should rarely happen) it will be caught 
and move into a flow's standard error handling path. The flow should 
then properly terminate.

`Event.Error` was changed to make the choice to rollback be optional. 
Errors during flow initialisation cause the flow to not have a open 
database transaction. Therefore there is no need to rollback.

## Errors that occur when handling the outcome of an existing flow error

When an error occurs a flow goes to the flow hospital and is given an 
outcome event to address the original error. If the transition that was 
processing the error outcome event (`StartErrorPropagation` and
`RetryFlowFromSafePoint`) has an error then the flow aborts and
nothing happens. This means that the flow is left in a runnable state.

To resolve this, we now retry the original error outcome event whenever 
another error occurs doing so.

This is done by adding a new staff member that looks for 
`ErrorStateTransitionException` thrown in the error code path of 
`TransitionExecutorImpl`. It then takes the last outcome for that flow 
and schedules it to run again. This scheduling runs with a backoff.

This means that a flow will continually retry the original error outcome
event until it completes it successfully.

## Failures to rollback and close a database transaction when an error occurs in `TransitionExecutorImpl`
   
Rolling back and closing the database transaction inside of 
`TransitionExecutorImpl` is now done inside individual try/catch blocks
as this should not prevent the flow from continuing.

## Removal of create and commit transaction actions around retrying a flow

The database commit that occurs after retrying a flow can fail which 
required some custom code just for that event to prevent inconsistent 
behaviour. The transaction was only needed for reading checkpoints from 
the database, therefore the transaction was moved into 
`retryFlowFromSafePoint` instead and the commit removed.

If we need to commit data inside of `retryFlowFromSafePoint` in the 
future, a commit should be added directly to `retryFlowFromSafePoint`. 
The commit should occur before the flow is started on a new fiber.
2020-06-30 11:50:42 +01:00
Tamas Veingartner
18cd24c2b4
NOTICK Ignore pagination large volume test as its completion is environment dependent and might cause issues in slower environments (#6420) 2020-06-30 10:12:31 +01:00
Denis Rekalov
e75dc9e415 Merge branch 'release/os/4.5' into denis/CORDA-3856-amqp-header-os-4.6 2020-06-29 13:13:48 +01:00
Denis Rekalov
7686ca2944 Merge branch 'release/os/4.4' into denis/CORDA-3856-amqp-header-os-4.5 2020-06-29 12:35:05 +01:00
pnemeth
6fc9f2eacf
EG-2854 Failing Test: net.corda.node.internal.NodeStartupCliTest.--nodeconf using absolute path will not be changed (#6393)
* EG-2854

Failing Test: net.corda.node.internal.NodeStartupCliTest.--nodeconf using absolute path will not be changed

* fix bug EG-2854

* PR comment
2020-06-29 09:40:12 +01:00
Denis Rekalov
3f03de6fbd
CORDA-3856: Add Artemis plugin for validating AMQP message header and type (#6407) 2020-06-29 09:23:29 +01:00
Dan Newton
516a4bf3a1
Merge pull request #6403 from corda/NOTICK-rfowler-merge-OS4.5-OS4.6-20200626
NOTICK rfowler merge os4.5 os4.6 20200626
2020-06-29 08:40:48 +01:00
Dan Newton
796e92b512
CORDA-3720 Extract locking of InnerState out of SMM (#6289)
The state machines state is held within `InnerState` which lived inside
the SMM. `InnerState` has been extracted out of the SMM to allow the SMM
to be refactored in the future. Smaller classes can now be made that
focus on a single goal as the locking of the state can be accessed from
external classes. To achieve this, pass the `InnerState` into the class
and request a lock if needed.

The locking of `InnerState` has been made a property of the `InnerState`
itself. It has a `lock` field that allows locks to be taken out when
needed.

An inline `withLock` function has been added to tidy up the code and not
harm performance.

Some classes have been made internal to prevent invalid usage of purely
node internal classes.

As part of this change, flow timeouts have been extracted out into
`FlowTimeoutScheduler`.
2020-06-26 12:48:15 +01:00
Ryan Fowler
31a1ee8803 Merge branch 'release/os/4.5' into NOTICK-rfowler-merge-OS4.5-OS4.6-20200626 2020-06-26 12:14:56 +01:00
Ryan Fowler
881d8d687c Merge branch 'release/os/4.4' into NOTICK-rfowler-merge-OS4.4-OS4.5-20200626 2020-06-26 09:52:16 +01:00
Ryan Fowler
ef582900cf
NOTICK Expand the regex to match what we already do in ENT (#6400) 2020-06-25 17:26:55 +01:00
Kyriakos Tharrouniatis
6ec2910f15
NOTICK Revert node_flow_exceptions type length back to 256 (#6395) 2020-06-25 09:40:58 +01:00
Chris Rankin
6485a025c7
ENT-5430: Increase test coverage when serializing Optional fields. (#6387) 2020-06-22 16:51:40 +01:00
Waldemar Zurowski
1a4efbac7f Merge branch 'release/os/4.5' into merge-45-to-46 2020-06-19 19:48:08 +01:00
pnemeth
d6cab0e131
EG-1557 - Configuration data from "include" section ignored while command line contains the path to config file without leading ./ (#6354)
Configuration data from "include" section ignored while command line contains the path to config file without leading ./
2020-06-19 10:32:55 +01:00
alicer3
e77f7a7546
center console message for registration (#6191) 2020-06-19 09:49:07 +01:00
LankyDan
56d0bbc036 CORDA-3841 Check isAnyCheckpointPersisted in startFlowInternal (#6351)
Only hit the database if `StateMachineState.isAnyCheckpointPersisted`
returns true. Otherwise, there will be no checkpoint to retrieve from the
database anyway. This can prevent errors due to a transient loss of
connection to the database.

Update tests after merging to 4.6
2020-06-18 16:15:15 +01:00
LankyDan
e8b17ff7b9 Merge branch 'release/os/4.5' into dan/os-4.5-to-4.6-merge-2020-06-18
# Conflicts:
#	node/src/main/kotlin/net/corda/node/services/persistence/DBCheckpointStorage.kt
#	node/src/main/kotlin/net/corda/node/services/statemachine/SingleThreadedStateMachineManager.kt
2020-06-18 15:50:46 +01:00
Christian Sailer
4091fdc8b1
ENT-5264 Synchronise schema on the command line (#6353)
* Decouple DatabaseConfig and CordaPersistence etc.
* Add schema sync to schema migration + test
* Add command line parameters for synchronising schema
2020-06-18 11:38:46 +01:00
Chris Rankin
1ef62870bb Merge commit 'fe617818895edab334d80c5e8de2b38f39e67af6' into chrisr3-os44-merge 2020-06-17 18:54:54 +01:00
Chris Rankin
d0c0a1d9ba
ENT-5430: Fix deserialisation of commands containing generic types. (#6359) 2020-06-17 17:28:26 +01:00
James Higgs
24b0240d82
EG-2654 - Ensure stack traces are printed to the logs in error reporting (#6345)
* EG-2654 Ensure stack trace is printed to the logs in error reporting

* EG-2654 - Add a test case for exception logging
2020-06-17 14:32:12 +01:00
Dan Newton
7ab6a8f600
CORDA-3841 Check isAnyCheckpointPersisted in startFlowInternal (#6351)
Only hit the database if `StateMachineState.isAnyCheckpointPersisted`
returns true. Otherwise, there will be no checkpoint to retrieve from the
database anyway. This can prevent errors due to a transient loss of
connection to the database.
2020-06-16 09:22:26 +01:00
Tamas Veingartner
26d4bfb89f
CORDA-3578 add corda prefix to conf file names as original issue was … (#6322)
* CORDA-3578 add corda prefix to conf file names as original issue was having non-corda reference.conf files on classpath causes DriverDSLImp failure

it is better to have this naming convention and avoid further conflicts of conf files.

* fixed weird duplicates

* revert renaming changes for web-reference.conf and loadtest-reference.conf
2020-06-16 09:15:51 +01:00
Christian Sailer
836dd559e8
ENT-5316 split schema migration
* ENT-5273 Split schema migration into separate core and app schema migration, with separate command line flags
2020-06-15 15:52:31 +01:00
Christian Sailer
2c26f4db5d Merge remote-tracking branch 'origin/release/os/4.6' into christians/update-fb-2020-06-12 2020-06-15 09:02:55 +01:00
Christian Sailer
f1126226a8 Fix config tests (remove tx isolation level from config files) 2020-06-12 20:54:36 +01:00
Christian Sailer
35c661b9f6
Merge pull request #6341 from corda/chrisr3-45-merge
NOTICK: Merge from OS 4.5 up to ef00fa1.
2020-06-12 16:42:51 +01:00
Stefano Franz
64f0011a62
Make Checkpoint classes data classes (#6342)
* Make Checkpoint classes data classes

* tidy up null-checks for array equality
2020-06-12 16:35:32 +01:00
Christian Sailer
d00dc42b18 Merge remote-tracking branch 'origin/release/os/4.6' into christians/update-fb-2020-06-12 2020-06-12 14:51:43 +01:00
Chris Rankin
3f67e314c0 Merge commit 'ef00fa1388db37e155ab8cfed3763c14801f8aa9' into chrisr3-45-merge 2020-06-12 13:14:44 +01:00
James Higgs
6e349f298e
NOTICK - Ignore a potentially dodgy test (#6336) 2020-06-11 16:47:48 +01:00
James Higgs
ab023d0b07 Merge branch 'release/os/4.5' into jamesh/os-4.5-4.6-merge-11062020 2020-06-11 09:40:39 +01:00
James Higgs
58af87c988
EG-2225 - Create log directory in correct place with verbose flag set (#6321)
* Ensure logs directory is written to correct location
* Remove a superfluous set of log path property
* Add a unit test to catch bad log paths
* Address detekt issues
2020-06-10 10:46:57 +01:00
James Higgs
8b7275eb97
EG-2564 - Move printed error to logger (#6323) 2020-06-10 10:45:50 +01:00
Schife
fb184839f4 Merge branch 'release/os/4.5' of https://github.com/corda/corda into razvan-os-4.5-to-4.6-merge 2020-06-05 07:55:48 +01:00
nikinagy
caf5482244
ENT-4064, ENT-4608 - checking for unsigned cordapps in prod mode and checking the PV (#6291)
* checking for unsigned cordapps in prod mode and shutting down if it the contracts are not signed

* inheriting from CordaRuntimeException instead of Exception

* had the same tests twice, removed the shutdown for minimumplatformversion, modified some of the tests

* shut down when we get invalid platformversion and modified the test according to this

* making the message more accurate
2020-06-04 16:24:49 +01:00
Denis Rekalov
45614cf29e
Merge pull request #6266 from corda/denis/CORDA-3805-custom-migration-scripts
CORDA-3805: cut dependency from PersistentIdentityService for custom migration scripts
2020-06-04 14:20:26 +01:00
Dan Newton
f0d2c9fe71
NOTICK Correct StatemachineGeneralErrorHandlingtest (#6306) 2020-06-04 08:04:41 +01:00
Christian Sailer
98f62f60f1 Merge branch 'release/os/4.6' into christians/update-feat-20200502 2020-06-02 18:15:32 +01:00
Christian Sailer
2cb89897b4 Merge branch 'release/os/4.6' into christians/update-feat-20200502 2020-06-02 17:55:14 +01:00
Kyriakos Tharrouniatis
b8b462f68e
NOTICK Fix schema validation error for flow parameters (#6304)
Adding 'corda-blob' type, fixed 'Schema-validation: wrong column type'
2020-06-02 17:52:50 +01:00
williamvigorr3
0554c98d18
NOTICK Restrict to shell commands (#6303)
Remove shell command for pausing flows
2020-06-02 16:04:10 +01:00
James Higgs
04ddb267fd
[EG-2225] Prevent extra directories being created when relative base directories are specified (#6282)
Don't create an extra directory if a relative base path specified
2020-06-02 15:26:40 +01:00
Tamas Veingartner
9c4a76d367
CORDA-3176 Inefficient query generated on vault queries with custom p… (#6241)
* CORDA-3176 Inefficient query generated on vault queries with custom paging

a check added to avoid self joins
local postgres db tests were executed on large volume test data (50k states) to ensure that the DB optimizes the suspicious query and so it does not cuase performance issues.
A test added to check that a pagination query on large volume of sorted data is completed in a reasonable time

* foreach detekt fix
2020-06-02 10:33:59 +01:00
Christian Sailer
0ed6307577 Merge remote-tracking branch 'origin/release/os/4.6' into christians/update-feat-20200502 2020-06-02 09:03:11 +01:00
Denis Rekalov
98af7f10f9 CORDA-3805: Make custom migration scripts independent on the latest PersistentIdentityService schema 2020-06-01 16:22:21 +01:00
Rick Parker
9f2bd1dcae
Merge pull request #6295 from corda/feature/checkpoint_table_improvements
CORDA-3432 Feature/checkpoint table improvements
2020-06-01 11:31:13 +01:00
Kyriakos Tharrouniatis
4507b55857
CORDA-3725 Fix SQL deadlocks coming from soft locking states (#6287)
Adding to the query -explicitly- the flow's locked states resolved the SQL Deadlocks in SQL server. The SQL Deadlocks would come up  from `softLockRelease` when only the `lockId` was passed in as argument. In that case the query optimizer would use `lock_id_idx(lock_id, state_status)` to search and update entries in `VAULT_STATES` table. 

However, all rest of the queries would follow the opposite direction meaning they would use PK's `index(output_index, transaction_id)` but they would also update the `lock_id` column and therefore the `lock_id_idx` as well, because `lock_id` is a part of it. That was causing a circular locking among the different transactions (SQL processes) within the database.

To resolve this, whenever a flow attempts to reserve soft locks using their flow id (remember the flow id is always the flow id for the very first flow in the flow stack), we then save these states to the fiber. Then, upon releasing soft locks the fiber passes that set to the release soft locks query. That way the database query optimizer will use the primary key index of VAULT_STATES table, instead of lock_id_idx in order to search rows to update. That way the query will be aligned with the rest of the queries that are following that route as well (i.e. making use of the primary key), and therefore its locking order of resources within the database will be aligned with the rest queries' locking orders (solving SQL deadlocks).

* Fixed SQL deadlocks caused from softLockRelease, by saving locked states per fiber; NodeVaultService.softLockRelease query then uses VAULT_STATES PK index instead of lock_id_idx

Speed up SQL server by breaking down queries with > 16 elements in their IN clause, into sub queries with 16 elements max in their IN clauses

* Allow softLockedStates to remove states

* Add to softLockedStates only states soft locked under our flow id

* Fix softLockRelease not to take into account flowStateMachineImpl.softLockedStates when using lockId != ourFlowId

* Moved CriteriaBuilder.executeUpdate at the bottom of the file
2020-05-29 12:35:05 +01:00
Denis Rekalov
499c09e77b
Merge pull request #6285 from corda/denis/CORDA-3818-sync-identity-service
CORDA-3818: Synchronize OS implementation of PublicKeyToOwningIdentityCache with CE
2020-05-28 14:21:04 +01:00
Kyriakos Tharrouniatis
598228634f
CORDA-3608 Performance tune the new checkpoint schema (#6277)
Performance tuning of the new checkpoint schema;

- Checkpoint tables are now using `flowId` as join keys. 
- Indexes consist of a PK's index on `node_checkpoints(flow_id)` and then unique indexes on `node_checkpoint_blobs(flow_id)` and `node_flow_metadata(flow_id)`.
- Serialization of `checkpointState` is being done with `CHECKPOINT_CONTEXT` so that we can have compression. This is needed when messages get passed into `checkpointState.sessions` therefore `checkpointState` grows in size upon serialized and 
saved into the database.

* Deserialize checkpointState with CHECKPOINT_CONTEXT

* Align tests with schema update; We cannot add and update a checkpoint in the same session now, ends up with hibernate complaining: two different objects with same identifier

* Fix indentation and format

* Ignore tests that assert DBFlowResult or DBFlowException

* Set DBFlowCheckpoint.blob to null whenever the flow errors or hospitalizes; this way we save an extra SELECT in such cases;

* Fix test; cleared Hibernate session, it would fail at checkpoint with 'org.hibernate.NonUniqueObjectException'

* Changing VARCHAR to NVARCHAR

* Rename v17 liquibase scripts to v19 to resolve collision with ENT v17 scripts
2020-05-27 16:46:56 +01:00
Chris Rankin
6e156bc0ea Merge commit '6ebc6e9b16575a7afbb781b0d93a8f04b3affba5' into chrisr3-45-merge 2020-05-27 10:40:36 +01:00
Denis Rekalov
5afdb63c94 CORDA-3818: Synchronize OS implementation of PublicKeyToOwningIdentityCache with CE 2020-05-26 16:10:14 +01:00
Chris Rankin
6ebc6e9b16
CORDA-3750: Reimplement Corda's Crypto object for use inside the sandbox. (#6193)
* CORDA-3750: Use hand-written sandbox Crypto object that delegates to the node.

* CORDA-3750: Add integration test for deterministic CashIssueAndPayment flow.

* Tidy up generics for Array instances.

* Upgrade to DJVM 1.1-RC04.
2020-05-26 15:46:29 +01:00
Dan Newton
330a95cb68
ENT-4627 Log transaction context missing exception (#6278)
When a flow is missing its database transaction the _real_ stacktrace
of the exception is missing in the logs. This is due to the normal error
handling path way does not work due the transaction being missing. When
it goes to process the next error event (due to the original transaction
context missing error) it will fail for the same error as it needs the
context to be able to process the error event.

The extra logging should help diagnose future errors.
2020-05-26 11:32:42 +01:00
Christian Sailer
70f1ea0a9d
ENT-5258 db schema set-up only via command line flag (#6280)
Removing the ability to initialise schema from the node config, and add a new sub-command to initialise the schema (that does not do anything else and exits afterwards).
Also adding a command line flag that allow app schema to be maintained by hibernate for legacy cordapps, tests or rapid development.
Patching up mock net and driver test frameworks so they create the required schemas for tests to work, defaulting schema migration and hibernate schema management to true to match pre-existing behaviour.
Modified network bootstrapper to run an initial schema set-up so it can register nodes.
2020-05-22 16:27:10 +01:00
Kyriakos Tharrouniatis
57b1a5e0fc
ENT-5339 Failing tests against Oracle in VaultObserverExceptionTest (#6275)
* Fix erroneous sql statement for oracle; It was failing tests with 'ORA-00933: SQL command not properly ended'

* Fixed flaky test; it didn't wait for counter party flow to get hospitalized as the test implied
2020-05-22 10:15:51 +01:00
Rick Parker
67298eb780
Merge pull request #6270 from corda/nnagy-os-4.5-os-4.6-20200520
Nnagy os 4.5 os 4.6 20200520
2020-05-21 13:19:58 +01:00
Viktor Kolomeyko
31612901cf
CORDA-3800: Flaky ProtonWrapperTests test investigation (#6258) 2020-05-21 12:33:56 +01:00
nikinagy
301fc9f928 Merge branch 'release/os/4.5' into nnagy-os-4.5-os-4.6-20200520 2020-05-20 10:38:45 +01:00
nikinagy
938d038c97
ENT-5278 - add missed checks for empty lists (#6261)
* add missed checks for empty lists

* adding missing brackets

* adding missing timeout
2020-05-19 16:55:09 +01:00
Dan Newton
36a11e868a
NOTICK Fix timing issue in FlowIsKilledTest (#6264)
Also remove `Configurator` that seems to fail on some agents.
2020-05-19 16:54:34 +01:00
williamvigorr3
eb52de1b40
CORDA-3490 Add option to start node without starting checkpointed flows (#6136)
Added command-line option: `--pause-all-flows` to the Node to control this.
This mode causes all checkpoints to be set to status PAUSED when the
state machine starts up (in StartMode.Safe mode). 

Changed the state machine so that PAUSED checkpoints are loaded into
memory (the checkpoint is deserialised but the flow state is left serialised)
but not started.

Messages from peers are queued whilst the flow is paused and processed
once the flow is resumed.
2020-05-19 16:27:41 +01:00
nikinagy
cc8ce3ca99
empty list checks (#6262) 2020-05-19 14:04:19 +01:00
Adel El-Beik
b9b0ca58e3 Merge remote-tracking branch 'origin/release/os/4.5' into adel/merge-from-4.5 2020-05-15 10:23:17 +01:00
Ryan Fowler
6dd72ac079
ENT-5275: Replace hardcoded platform versions with descriptive values (#6236)
* ENT-5275: Replace hardcoded platform versions with descriptive values

* PR updates

* missed versions

* missed versions

* fix

* add FIRST_VERSION
2020-05-14 14:43:52 +01:00
James Higgs
06467d04b1
EG-1980 - Remove stale docs links from OS (#6251)
Remove hardcoded docs references, push docs link to gradle
2020-05-14 12:57:59 +01:00
Denis Rekalov
8387e99535
[NOTICK]: Do not use Security.addProvider(BouncyCastleProvider()) in tests to avoid disruptions of other tests (#6250) 2020-05-14 10:53:54 +01:00
Ivan Schasny
bd657fad65
Merge pull request #6245 from corda/EG-1991-ConfigHelper-update
EG-1991 Ignore warning for CORDA environment variables in all caps
2020-05-13 15:37:42 +01:00
Dan Newton
c2ac22a41f
NOTICK Non-database error handling in withEntityManager (#6239)
When a non-database exception is thrown out of a `withEntityManager`
block, always check if the session needs to be rolled back.

This means if a database error is caught and a new non-database error is
thrown out of the `withEntityManager` block, the transaction is still
rolled back. The flow can then continue progressing as normal.
2020-05-13 15:17:51 +01:00
john.buckle@r3.com
febb5a44b7 Added comment 2020-05-13 21:55:04 +08:00
Ryan Fowler
9caf6538ce
ENT-5298: Increase timeout by two minutes because the test occasionally takes longer than five minutes (#6246) 2020-05-13 13:15:05 +01:00
john.buckle@r3.com
c3ea2c1470 Ignore warning for CORDA environment variables in all caps 2020-05-13 09:51:35 +08:00
nikinagy
e0b5319515 Merge branch 'release/os/4.4' into nnagy-os-4.4-os-4.5-20200511 2020-05-11 16:32:53 +01:00
nikinagy
350066d386
fix for handling empty lists in vault query (#6231) 2020-05-11 15:41:50 +01:00
Ryan Fowler
5f8cceae45 Merge branch 'release/os/4.4' into rfowler-os-4.4-os-4.5-20200511 2020-05-11 11:40:09 +01:00
LankyDan
8125b9bdb6 NOTICK Change platform version assert in FlowMetadataRecordingTest 2020-05-06 10:03:05 +01:00
LankyDan
565afc5fdb NOTICK Fix kill flow tests due to storing failed flows
Failed flows are stored after the checkpoint table rework. This meant
that some of the asserts in `FlowIsKilledTest` and `KillFlowTest` were
wrong.
2020-05-06 10:02:40 +01:00
Chris Rankin
9a5be9e47d
CORDA-3738: Upgrade to DJVM 1.1-RC03. (#6219) 2020-05-05 17:32:39 +01:00
LankyDan
217fd906b3 NOTICK Remove unused imports 2020-05-05 17:22:42 +01:00
LankyDan
7fe2845272 NOTICK Fix KilledFlowTransition after merge 2020-05-05 17:15:54 +01:00
LankyDan
4ccd0fd3df Merge branch 'release/os/4.6' into dan/4.6-into-checkpoint-feature-branch-2020-05-05
# Conflicts:
#	node/src/main/kotlin/net/corda/node/services/statemachine/Event.kt
#	node/src/main/kotlin/net/corda/node/services/statemachine/FlowStateMachineImpl.kt
#	node/src/main/kotlin/net/corda/node/services/statemachine/SingleThreadedStateMachineManager.kt
#	node/src/main/kotlin/net/corda/node/services/statemachine/transitions/TopLevelTransition.kt
2020-05-05 17:05:23 +01:00
Ryan Fowler
b6dc1d8c4a
NOTICK - Reduce flow count for test to improve run time (#6213) 2020-05-05 09:38:55 +01:00
nargas-ritu
54e4503866
Merge pull request #6212 from corda/dan/update-byteman-dependency
NOTICK Update byteman dependencies
2020-05-04 16:29:29 +01:00
Ryan Fowler
55797612b2 ENT-5237: Remove DISTINCT (backport) 2020-05-04 15:02:33 +01:00
LankyDan
96c1120194 NOTICK Update byteman dependencies
Tests that were using byteman when running on Java 11 were failing.
Updating the byteman version fixed the issue while still working on Java
8.
2020-05-04 14:08:38 +01:00
Denis Rekalov
32407b2c46
CORDA-3753: Increase Artemis security-invalidation-interval to avoid frequent CRL checks (#6207) 2020-05-04 13:00:22 +01:00
Ryan Fowler
ebdd40049c CORDA-3662: Use an INNER JOIN for network map cache queries, (#6062)
- rename add or update function for clarity
- put removal of old nodes after retrieval of new ones to avoid gaps in the map
- plus add a test
2020-05-04 11:10:21 +01:00
Chris Rankin
107819f5b5
CORDA-3745: Modify DJVM serializers to support Enum Evolution. (#6189) 2020-04-30 14:59:10 +01:00
Joseph Zuniga-Daly
2ce76e407d
ENT-5237: Remove DISTINCT because Oracle cannot apply DISTINCT to BLOB fields (#6203) 2020-04-30 14:27:45 +01:00
Katelyn Baker
99fd08909d
EG-433 - Error Reporting Framework (Merge)
Error Reporting Framework - Phase 1
2020-04-30 13:12:54 +01:00
Chris Rankin
83dd9a96da
CORDA-3738: Upgrade to DJVM 1.1-RC02. (#6195)
* CORDA-3738: Upgrade to DJVM 1.1-RC02.

* Update comment for DJVM 1.1's new requirements.
2020-04-30 11:58:03 +01:00
Adel El-Beik
9bcb9b2f54
CORDA-3191: When querying for attachment ids now only retrieve the attachment id columm not whole object. (#6199) 2020-04-30 11:52:47 +01:00
Dan Newton
cb84fd86ee
CORDA-3291 Fix broken tests (#6197) 2020-04-30 10:27:36 +01:00
Adel El-Beik
3259b595d7
CORDA-3715: Check contract classes hav… (#6155)
* CORDA-3715: When loading cordapps now check that contract classes have class version between 49 and 52

* CORDA-3715: Now check class version when contract verification takes place.

* CORDA-3715: Making detekt happy with number of levels in func

* CORDA-3715: Make use of new ClassGraph release which provides class file major version number.

* CORDA-3715: Changed package name in test jars

* CORDA-3715: Use ClassGraph when loading attachments.

* CORDA-3715: Reverted file to 4.5 version

* CORDA-3715: Updating method to match non deterministic version.

* CORDA-3715: Added in default param.

* CORDA-3715: Adjusted min JDK version to 1.1

* CORDA-3715: Switching check to JDK 1.2

* CORDA-3715: Now version check SerializationWhitelist classes.

* CORDA-3715: Switched default to null for range.
2020-04-30 08:57:37 +01:00
James Higgs
1956bdb5d4 [MERGE] Fix detekt issues 2020-04-29 11:57:37 +01:00
James Higgs
adbe030a2c Merge branch 'release/os/4.5' into jamesh/error-reporting-sync-29-04-20
# Conflicts:
#	node/src/main/kotlin/net/corda/node/internal/AbstractNode.kt
2020-04-29 11:47:14 +01:00
James Higgs
ab95aa57a2
[EG-440] Add some error codes and the error resource generation tool (#6192)
* [EG-438] First commit of error code interface

* [EG-438] Implement error reporter and a few error codes

* [EG-438] Add unit tests and default properties files

* [EG-438] Add the error table builder

* [EG-438] Update initial properties files

* [EG-438] Add some Irish tests and the build.gradle

* [EG-438] Fall back for aliases and use different resource strategy

* [EG-438] Define the URL using a project-specific context

* [EG-438] Tidy up initialization code

* [EG-438] Add testing to generator and tidy up

* [EG-438] Remove direct dependency on core and add own logging config

* [EG-438] Fix compiler warnings and tidy up logging

* [EG-438] Fix detekt warnings

* [EG-438] Improve error messages

* [EG-438] Address first set of review comments

* [EG-438] Use enums and a builder for the reporter

* [EG-438] Address first set of review comments

* [EG-438] Use enums and a builder for the reporter

* [EG-438] Add kdocs for error resource static methods

* [EG-440] Add error code for duplicate CorDapp loading

* [EG-438] Handle enums defined with underscores

* [EG-440] Add errors for some CorDapp loading scenarios

* [EG-440] Finish adding errors for CorDapp loading

* [EG-440] Fix up errors in properties files

* [EG-440] Start change to error code definition

* [EG-440] Update error code definition and add resource generation tool

* [EG-440] Tidy up error resource generation tool frontend

* [EG-440] Small refactorings and add kdocs

* [EG-440] Generate all missing resources

* [EG-440] Some refactoring and start writing a test

* [EG-440] Update unit test for resource generator

* [EG-440] Renaming of various parts of the error tool

* [EG-440] Add testing for errors and fix an issue in resource generation

* [EG-440] Add a kdoc for context provider API

* [EG-440] Remove old code from repository

* [EG-440] Address some review comments
2020-04-29 11:21:50 +01:00
Dan Newton
297e504740
CORDA-3291 isKilled flag and session errors for killed flows (#6170)
* CORDA-3291 `isKilled` flag and session errors for killed flows

## Summary

Two major improvements have been worked on:

- A new flag named `isKilled` has been added to `FlowLogic` to allow
developers to break out of loops without suspension points.
- Killed flows now send session errors to their counter parties allowing
their flows to also terminate without further coordination.

Achieving these changes required a __fundamental__ change to how flows are
killed as well as how they sleep.

## `isKilled` flag

The addition of `FlowLogic.isKilled` allows flows to check if the
current flow has been killed. They can then throw an exception to lead
to the flow's termination (following the standard error pathway). They
can also perform some extra logic or not throw an exception if they
really wanted to.

No matter what, once the flag is set, the flow will terminate. Due to
timing, a killed flow might successfully process its next suspension
event, but it will then process a killed transition and terminate.

## Send session errors when killing a flow

A flow will now send session errors to all of its counter parties. They
are transferred as `UnexpectedFlowEndException`s. This allows initiated
flows to handle these errors as they see fit, although they should
probably just terminate.

## How flows are killed

### Before

Originally we were relying on Quasar to interrupt a flow's fiber, we
could then handle the resulting `InterruptedException`. The problem with
this solution is that it only worked when a flow was already suspended
or when a flow moved into suspension. Flows stuck in loops did not work.

### After

We now *do not* use Quasar to interrupt a flow's fiber. Instead, we
switch `FlowStateMachine.isKilled` to true and schedule a new event.
Any event that is processed after switching this flag will now cause a
`KilledFlowTransition`. This transition follows similar logic to how
error propagation works. Note, the extra event allows a suspended flow
to be killed without waiting for the event that it was _really_ waiting
for.

This allows a lot of the tidy up code in `StateMachineManager.killFlow`
to be removed as tidy up is executed as part of removing a flow.
Deleting a flow's checkpoint and releasing related soft locks is still
handled manually in case of infinite loops but also triggered as part
of the actions executed in a transition.

This required flow sleeping to be changed as we no longer rely on
quasar.

## How flows now sleep

The reliance on Quasar to make a flow sleep has been removed.

Instead, when a flow sleeps we create a `ScheduledFuture` that is
delayed for the requested sleep duration. When the future executes it
schedules a `WakeUpFromSleep` event that wakes up the flow... Duh.

`FlowSleepScheduler` handles the future logic. It also uses the same
scheduled thread pool that timed flows uses.

A future field was added to `StateMachineState`. This removes the 
need for concurrency control around flow sleeps as the code path does
not need to touch any concurrent data structures.

To achieve this:

- `StateMachineState.future` added as a `var`
- When the `ScheduledFuture` is created to wake up the flow the passed
in `StateMachineState` has its `future` value changed
- When resumed `future` and `isWaitingForFuture` are set to `null` and
`false` respectively
- When cancelling a sleeping flow, the `future` is cancelled and nulled
out. `isWaitingForFuture` is not changed since the flow is ending anyway
so really the value of the field is not important.
2020-04-28 15:53:44 +01:00
James Higgs
ab43238420
[EG-438] Error Reporting Framework (#6125)
* [EG-438] First commit of error code interface

* [EG-438] Implement error reporter and a few error codes

* [EG-438] Add unit tests and default properties files

* [EG-438] Add the error table builder

* [EG-438] Update initial properties files

* [EG-438] Add some Irish tests and the build.gradle

* [EG-438] Fall back for aliases and use different resource strategy

* [EG-438] Define the URL using a project-specific context

* [EG-438] Tidy up initialization code

* [EG-438] Add testing to generator and tidy up

* [EG-438] Remove direct dependency on core and add own logging config

* [EG-438] Fix compiler warnings and tidy up logging

* [EG-438] Fix detekt warnings

* [EG-438] Improve error messages

* [EG-438] Address first set of review comments

* [EG-438] Use enums and a builder for the reporter

* [EG-438] Add kdocs for error resource static methods

* [EG-438] Handle enums defined with underscores

* [EG-438] Slight refactoring of startup code

* [EG-438] Port changes to error reporting code from future branch

* [EG-438] Also port test changes

* [EG-438] Suppress a deliberately unused parameter
2020-04-28 14:07:50 +01:00
Dan Newton
9a2ae8ae19
CORDA-3722 withEntityManager can rollback its session (#6187)
* CORDA-3722 withEntityManager can rollback its session

## Summary

Improve the handling of database transactions when using
`withEntityManager` inside a flow.

Extra changes have been included to improve the safety and
correctness of Corda around handling database transactions.

This focuses on allowing flows to catch errors that occur inside an
entity manager and handle them accordingly.

Errors can be caught in two places:

- Inside `withEntityManager`
- Outside `withEntityManager`

Further changes have been included to ensure that transactions are
rolled back correctly.

## Catching errors inside `withEntityManager`

Errors caught inside `withEntityManager` require the flow to manually
`flush` the current session (the entity manager's individual session).
By manually flushing the session, a `try-catch` block can be placed
around the `flush` call, allowing possible exceptions to be caught.

Once an error is thrown from a call to `flush`, it is no longer possible
to use the same entity manager to trigger any database operations. The
only possible option is to rollback the changes from that session.
The flow can continue executing updates within the same session but they
will never be committed. What happens in this situation should be handled
by the flow. Explicitly restricting the scenario requires a lot of effort
and code. Instead, we should rely on the developer to control complex
workflows.

To continue updating the database after an error like this occurs, a new
`withEntityManager` block should be used (after catching the previous
error).

## Catching errors outside `withEntityManager`

Exceptions can be caught around `withEntityManager` blocks. This allows
errors to be handled in the same way as stated above, except the need to
manually `flush` the session is removed. `withEntityManager` will
automatically `flush` a session if it has not been marked for rollback
due to an earlier error.

A `try-catch` can then be placed around the whole of the
`withEntityManager` block, allowing the error to be caught while not
committing any changes to the underlying database transaction.

## Savepoints / Transactionality

To make `withEntityManager` blocks work like mini database transactions,
save points have been utilised. A new savepoint is created when opening
a `withEntityManager` block (along with a new session). It is then used
as a reference point to rollback to if the session errors and needs to
roll back. The savepoint is then released (independently from
completing successfully or failing).

Using save points means, that either all the statements inside the
entity manager are executed, or none of them are.

## Some implementation details

- A new session is created every time an entity manager is requested,
but this does not replace the flow's main underlying database session.
- `CordaPersistence.transaction` can now determine whether it needs
to execute its extra error handling code. This is needed to allow errors
escape `withEntityManager` blocks while allowing some of our exception
handling around subscribers (in `NodeVaultService`) to continue to work.
2020-04-28 11:20:00 +01:00
nikinagy
ec96a844bd
CORDA-3659 - adding kdoc for RestrictedEntityManager and RestrictedConnection (#6179)
* adding kdoc for RestrictedEntityManager and RestrictedConnection

* adding kdoc for RestrictedEntityManager and RestrictedConnection
2020-04-28 09:35:47 +01:00
Matthijs van den Bos
a298a94960
Load CordaServices before NotaryService (#6173)
On node start, load CordaServices before starting the NotaryService,
so that the NotaryService can check that the services it requires are
available when starting.

Resolves #6172.
2020-04-24 09:15:38 +01:00
nikinagy
335372cfc1 Merge branch 'release/os/4.3' into nnagy-os-4.3-os-4.4-20200423 2020-04-23 16:03:17 +01:00
Joseph Zuniga-Daly
69a4f80cd2
ENT-5141: Fix ConcurrentModificationException in FetchDataFlow (#6176)
* ENT-5141: Fix ConcurrentModificationException in FetchDataFlow

* Make detekt happy

* Make CheckpointSerializationEnvironmentRule inheritable
2020-04-23 13:11:23 +01:00
Viktor Kolomeyko
257026f606
CORDA-3726: Remove memory leaks when many InProcess nodes started (#6169)
* CORDA-3762: Integration test exposing the problem reported

* CORDA-3726: Additional logging

* CORDA-3726: Prevent thread leaks

* CORDA-3726: New `journalBufferTimeout` parameter

* CORDA-3726: Override `journalBufferTimeout` parameter

* CORDA-3726: Making Detekt happier

* CORDA-3276: Account for extra thread user in MockNetwork

For real node this does not matter as `shutdown` can safely be called multiple times, which is not true for server thread provided by MockNetwork

* CORDA-3276: Do not make SMM shutdown "executor" as it belongs to AbstractNode

* CORDA-3276: Address input from @rick-r3

* CORDA-3276: Fix test after rebase
2020-04-23 08:53:51 +01:00
Denis Rekalov
0d441c3760
ENT-4912: Enable CRL checking with embedded Artemis (#6154) 2020-04-22 10:35:58 +01:00
nikinagy
2bcaa2ac80
CORDA-3569 - Add RestrictedConnection and more blocked methods to RestrictedEntityManager (#6129)
* adding blocked functions ro RestrictedEntityManager and creating RestrictedConnection class

* adding flow tests and fixing issues regarding the review

* adding quasar util to gradle

* updating flow tests

* adding space before } at .isThrownBy()

* adding spaces
2020-04-21 14:39:41 +01:00
Dimos Raptis
8faf72f7b5
[ENT-5210] - Whitelist SNAPPY encoding (#6163)
* [ENT-5210] - Whitelist SNAPPY encoding

* Remove unused imports
2020-04-20 08:09:38 +01:00
Ramzi El-Yafi
45b43f116d
[EG-503] Test cases for exposed notaryService (#6161) 2020-04-17 16:51:27 +01:00
Ramzi El-Yafi
6a07284324
[EG-503] Spent state audit tool (#6107)
* [EG-503] Spent state audit tool

Fixes

* Refinements to notary query interfaces. Feature complete.

* EG-503: Introduce optional `notaryService` in `ServiceHubCoreInternal`

* Remove redundant logic following change to use extensions API

Co-authored-by: Viktor Kolomeyko <viktor.kolomeyko@r3.com>
2020-04-16 16:05:21 +01:00
Adel El-Beik
27ea570fbb
CORDA-3696: JDK 11 Testing branch (#6131)
* CORDA-3696: Temporary update to enable JDK11 build and test. Will eventually be switchable.

* CORDA-3696: Filter out the Nashorn warning.

* CORDA-3696: Add JDK11 classifier.

* CORDA-3696: Updated match string to cope with JDK11.

* CORDA-3696: Filtering out SPHINCS256_SHA256 where failing due to JDK11.

* CORDA-3696: Now remove SPHINCS256_SHA256 only if JDK11.

* CORDA-3696: Fix test failure - switch to regex matching.

* CORDA-3696: Hide the illegal access warnings.

* CORDA-3696: Check for Java11 when disabling Java11 warnings.

* CORDA-3696: Fix unneccessary non null check.

* CORDA-3696: Reverting build env to JDK8

* CORDA-3696: Revert hiding of illegal access warnings via Unsafe class.

* CORDA-3696: Remove internal access warnings and new JDK11 version checker.

* CORDA-3696: Updated build file for OS

* CORDA-3696: Removed typo

* CORDA-3696: Fixed space typo.

* CORDA-3696: Open modules to remove the illegal access warnings.

Co-authored-by: Adel El-Beik <adelel-beik@19LDN-MAC108.local>
2020-04-16 10:20:30 +01:00
Dan Newton
896b0ab246
CORDA-3691 delete checkpoint record when complete (#6134)
* CORDA-3691 Delete checkpoint when flow finishes

The checkpoint and its related records in joined tables should be deleted
when a flow finishes.

Keeping these flows around will be completed in the future.

* CORDA-3691 Ignore some flow metadata tests

Ignore tests around recording the finish time of flow metadata records
since we are not currently keeping COMPLETED flows in the database.
2020-04-09 16:57:03 +01:00
Kyriakos Tharrouniatis
501b766e71
CORDA-3604 Store failed and hospitalised errors along with corresponding statuses (#6061)
Flows that are kept for overnight observation:

- Save their Checkpoint.status as 'HOSPITALIZED' in the database
- Save the error that caused the hospitalization in the database

A new Event was added for this reason. Whenever the hospital determines
a flow for hospitalization, it adds this Event in the flow's fiber queue.
When processed it creates a new DB transaction, stores the checkpoint status along with
the error, and it adds a 'FlowContinuation.ProcessEvents' continuation so that the fiber keeps
processing events (effectively since there are no more events in the fiber's channel, the fiber will suspend).

Flows that error:

- Their checkpoints are kept in the database
- Save their Checkpoint.status as 'FAILED'
- Save the error that caused the error in the database

Upon erroring, the flow's Checkpoint.status gets updated('FAILED') and the checkpoint is stored
in the database instead of getting removed. The flow then propagates the error to counterparties,
sets its future with the error and gets removed from memory.
2020-04-07 17:58:00 +01:00
Chris Rankin
39cc5e3403
CORDA-3698: Require no classifier for Open Core and DJVM-related modules. (#6132)
* ENT-4967: Require no classifier for corda-node-djvm, corda-deserializers-djvm.

* Also remove classifiers from core, serialization and finance-contracts.

* Compile corda-serialization-djvm for Java 8 and remove its classifier.
2020-04-06 11:00:40 +01:00
Viktor Kolomeyko
1d7c13276c
ENT-5089: Forcibly free associated with transport resources if it is already closed (#6130) 2020-04-03 17:30:29 +01:00
Stefan Iliev
1718af3a79
Notary logs warning on startup when it's not in the whitelist (#5857) 2020-04-02 10:50:03 +01:00
Chris Rankin
a7358ffa1a
CORDA-3643: Initialise MockNetwork's serialisation environment with all custom cordapps. (#6115) 2020-04-01 17:42:29 +01:00
Chris Rankin
0771a22d5b
ENT-4967: Allow quasar_classifier to be either null or an empty string. (#6119) 2020-03-31 15:32:36 +01:00
williamvigorr3
024d63147d
CORDA-3491 Remove the flow state when a flow finishes (#6083)
Added a new field Completed to the in-memory object FlowState.

FlowState.Completed is corresponds to flow_state=Null in the DB.

This change will save disk space.
2020-03-30 16:56:03 +01:00
Chris Rankin
f1ebaa761b
CORDA-3680: Add CorDapp custom serialisers to Driver's in-process nodes. (#6098)
* Run serialisation tests with both in-process and out-of-process nodes.

* Add custom serialisers and whitelists to Driver's AMQPServerSerializationScheme.
2020-03-27 10:01:30 +00:00
Dan Newton
678fb6eb94
Merge pull request #6108 from corda/dan/os-4.4-to-4.5-merge-2020-03-26
OS 4.4 to 4.5 merge 2020-03-26
2020-03-27 09:39:41 +00:00
Chris Rankin
cccbbe1c80
CORDA-3680: Add CorDapp custom serialisers to Driver's in-process nodes. (#6102)
* Run serialisation tests with both in-process and out-of-process nodes.

* Add custom serialisers and whitelists to Driver's AMQPServerSerializationScheme.
2020-03-27 09:16:31 +00:00
Dimos Raptis
963de40902
[NOTICK] - Enable check in detekt for unused imports (#6106)
* [NOTICK] - Enable check in detekt for unused imports

* Put back accidental removal of used import

* Some more accidental removals
2020-03-26 15:46:33 +00:00
LankyDan
d9c1907c88 Merge branch 'release/os/4.4' into dan/os-4.4-to-4.5-merge-2020-03-26 2020-03-26 14:26:41 +00:00
Kyriakos Tharrouniatis
6baa775e23 Merge branch 'release/os/4.5' into os_4.5-feature_checkpoint_table_improvements-merge 2020-03-26 11:37:00 +00:00
Joseph Zuniga-Daly
b766eff284 Merge remote-tracking branch 'origin/release/os/4.3' into jzd/merge-os4.3-into-os4.4-2020-03-26 2020-03-26 11:01:39 +00:00
Dan Newton
79b36aea8f
CORDA-3601 Record a flow's finish time (#6079)
* CORDA-3601 Record a flow's finish time

Record a flow's finish time by updating its metadata record. It is set
in `updateCheckpoint` by checking the status of the checkpoint. If it is
 `COMPLETED` it will set the `finishInstant` on the metadata object and
 update it.

* CORDA-3601 Record flow finish time for all finished statuses

Update the flow finish time for the following statuses:

- COMPLETED
- KILLED
- FAILED

* CORDA-3601 Use platform clock in `DBCheckpointStorage`
2020-03-25 13:47:00 +00:00
Denis Rekalov
b1180b467b ENT-4659: Remove CryptoServiceFactory from OS 2020-03-25 12:56:33 +00:00
Joseph Zuniga-Daly
2dbf90cafe
ENT-4857: Fix race condition in trackTransaction (#6096)
- Fix issue
- Emit warning if we are inside a DB transaction
- Include a path that does not emit warning
- Add unit tests
2020-03-25 11:53:06 +00:00
Dimos Raptis
b73a498062
[ENT-4754] - Move subflow preparation logic in FlowStateMachine 2020-03-25 09:02:14 +00:00
Ramzi El-Yafi
c697b5850b
[EG-896] Additional uniqueness provider test cases (#6085) 2020-03-24 11:10:27 +00:00
LankyDan
82e068be94 Merge branch 'release/os/4.4' into dan/os-4.4-to-4.5-merge-2020-03-23 2020-03-23 11:46:24 +00:00
Dan Newton
668748b054
CORDA-3669 Do not execute ExecuteAsyncOperation multiple times (#6087)
* CORDA-3669 Do not execute `ExecuteAsyncOperation` multiple times

When a `FlowExternalOperation` or `FlowExternalAsyncOperation` executes
and completes a flag (`isFlowResumed`) is switched to true.

This flag was used inside of `DoRemainingWorkTransition` to decide
whether to skip over the execution of an event.

Since this flag was being switched to true when the external operation's
 future completed, it was possible for _unexpected_ events to be placed
in the fiber's queue that would retrigger the
`FlowIORequest.ExecuteAsyncOperation`, that is held as the checkpoint's
next `FlowIORequest`to process.

By using the existing `StateMachineState.isTransactionTracked` (and
renaming it to `isWaitingForFuture`) we can decide to not process the
`FlowIORequest.ExecuteAsyncOperation` if it has already been called
before. This moves this code path in line with
`FlowIORequest.WaitForLedgerCommit`.

Random `DoRemainingWork` events can now be pushed to the fiber's queue
without causing the `FlowIORequest.ExecuteAsyncOperation` to execute
again.
2020-03-20 19:02:34 +00:00
Kyriakos Tharrouniatis
da320514a5 Merge branch 'release/os/4.5' into os_4.5-feature_checkpoint_table_improvements-merge
# Conflicts:
#	node/src/main/kotlin/net/corda/node/services/statemachine/transitions/StartedFlowTransition.kt
2020-03-19 21:49:55 +00:00
Dan Newton
1c7126afb7
NOTICK Rename rpc_user column to started_by in metadata table (#6081) 2020-03-19 16:07:00 +00:00
Christian Sailer
82d9995717
ENT-5109 Harmonize config-common, make everything compile again and harmonize NetworkParameterOverridesSpec. (#6082) 2020-03-19 15:43:08 +00:00
Dimos Raptis
56067acd20
[CORDA-3628] - Remove overloads for sendAll (#6078) 2020-03-18 14:37:04 +00:00
Chris Rankin
29a36c6b4f
Prevent Quasar from instrumenting classes that belong to AttachmentsClassLoader. (#6077) 2020-03-18 13:59:10 +00:00
Dan Newton
ca23612fe1
CORDA-3596 Record flow metadata (#6067)
* CORDA-3596 Record flow metadata

Record flow metadata during the zero'th checkpoint that occurs before
calling the flow's `call` function.

This required adding an RPC call's arguments to the `InvocationContext`
that gets created. These arguments are then accessible within the
statemachine and from the `Checkpoint` class. The arguments are then
extracted when recording a flow's metadata inside of
`DBCheckpointStorage`.

Updated the size of the started by column to 128 since it was not long
enough to hold the fully qualified class of a service that started a
flow.

* CORDA-3596 Remove arguments from in-memory checkpoint

When executing a flows first real suspend (from flow code) the arguments
 contained in the `InvocationContext` are removed. This saves holding
 these arguments for the whole lifecyle of a flow.

* CORDA-3596 Increase `cordapp_name` column to 128

* CORDA-3596 Join metadata by `flow_id`

Due to changes in where metadata is recorded, there is no need for
having `invocation_id` as the metadata table's primary key. The
`flow_id` is now the primary key of the table and is used to join to the
 main checkpoints table.

The `invocation_id` has been removed from the checkpoints table since it
 is not needed for the join anymore.

* CORDA-3596 Remove `received_time` from metadata table

* CORDA-3596 Remove unused `StartReason` enum

* CORDA-3596 Simple `DBCheckpointStorageTests` for metadata

* CORDA-3596 Truncate really long flow names
2020-03-17 17:28:32 +00:00
Ryan Fowler
349bd5a511
CORDA-3662: Use an INNER JOIN for network map cache queries, (#6062)
- rename add or update function for clarity
- put removal of old nodes after retrieval of new ones to avoid gaps in the map
- plus add a test
2020-03-17 17:02:08 +00:00
Dimos Raptis
eba113621c
[CORDA-3628] - Avoid sending actions to the state machine if no messages are to be sent (#6074) 2020-03-17 16:51:07 +00:00
LankyDan
944fbce740 Merge branch 'feature/checkpoint_table_improvements' into dan/merge-4.5-into-feature-branch 2020-03-16 15:38:14 +00:00
Kyriakos Tharrouniatis
121c789c59
CORDA-3655 Do not run COMPLETED, FAILED or KILLED flows on node startup (#6070)
* CheckpointStorage.getAllCheckpoints will not fetch COMPLETED, FAILED and KILLED flows by default

* Rename getAllCheckpoints to getAllRunnableCheckpoints for clarity

* Fix Detekt issue

* Rename getAllRunnableCheckpoints to getRunnableCheckpoints

* Minor kdoc update

* Bring back in CheckpointStorage.getAllCheckpoints to co-exist with getRunnableCheckpoints
2020-03-16 15:25:36 +00:00
williamvigorr3
1025ee1dee
CORDA-3599 Add progress tracker information to checkpoint (#6063)
* Add progress tracker information to checkpoint

The checkpoint Datebase is updated when the statemachine suspends
with the progress trackers current step name. This is truncated if
it is longer than the Database column.

* Minor rename in statemachine for clarity
2020-03-16 09:30:23 +00:00
LankyDan
8941cf7b8f Merge branch 'feature/checkpoint_table_improvements' into dan/merge-4.5-into-feature-branch
# Conflicts:
#	node/src/test/kotlin/net/corda/node/services/statemachine/FlowFrameworkTests.kt
2020-03-13 14:29:18 +00:00
Christian Sailer
ea22a10b3e
ENT-4494 Harmonize network stack (#6059)
* ENT-4494 harmonize proton wrapper with ENT

* Harmonise Artemis and Bridge implementation

* Move tests across

* detekt changes

* Fix AMQP tests in node
2020-03-13 14:26:24 +00:00
Kyriakos Tharrouniatis
0174d996bd
CORDA-3598 Set Checkpoint.status to RUNNABLE (#6019)
* Set/ Reset Checkpoint.status to RUNNABLE after when suspending

* Removing/ Moving comment as it makes no longer sense to be there since, we now always create a new Checkpoint object in SingleThreadedStateMachineManager.createFlowFromCheckpoint through tryDeserializeCheckpoint

* Set -in memory- Checkpoint.status to RUNNABLE when a flow is retrying from Checkpoint
2020-03-12 17:57:35 +00:00
Dan Newton
3b335ebb00
NOTICK Fix statemachine error handling tests (#6057)
Due to a change in how messaging works, `ActionExecutorImpl
.executeSendInitial` was no longer being called. Changing the byteman
script to throw exception on hits to `ActionExecutorImpl
.executeSendMultiple` allowed the tests to pass.
2020-03-12 12:48:40 +00:00
williamvigorr3
499b6cf17e
CORDA-3603 Save completed flow information (#6034)
When a flow is finished do not delete the checkpoint from the DB.

Instead, the FlowStatus is marked as Completed in the DB.

Updated numerous tests which relied on the flow being removed
when finished.
2020-03-12 11:45:30 +00:00
LankyDan
ac3f880ae4 Merge branch 'release/os/4.5' into dan/merge-4.5-into-feature-branch
# Conflicts:
#	node/src/main/kotlin/net/corda/node/services/statemachine/transitions/StartedFlowTransition.kt
2020-03-12 08:56:59 +00:00
williamvigorr3
51db2c2d7f
CORDA-3600 Add flowIO request to checkpoint (#6017)
* Update Checkpoint DB to update flow io request

* Modify flow monitor to update Checkpoint DB with waiting flows

This happens periodically.

* Refactored code to avoid looping twice and updated tests

* Fix tests after rebasing

* Fix MR comments (non-functional refactor of tests + FlowMonitor).

* Made visible for testing method private in DBCheckpointStorage

This is not needed anymore.

* Explicity check if ioRequestType has changed in update method

* Fix shadowing warning

* Import non deprecated Assert into test

* Use AssertEquals not assert in test

* Address more comments (minor refactor) of DBCheckpointStorage

* Minor fix use it instead of referencing object explicitly

* Add null check to DBCheckpointStorage

* Revert changes to Flow Monitor.

We will instead store the information in the main thread of the
state machine.

* Remove now uneeded API and make statemachine update ioRequest

* Add Integration Test to check statemachine updates DB on Recieve

* Use simpleName in checkpoint storage instead of class.

Hibernate was previously resetting the class field this is now
set to null (when getting checkpoint form DB) and a new method
for getting back the simple name as a string.

* Update StateMachineState to store simple name.

* Fix after rebase broke stuff + renamed test

* Fix Detekt issue

* Remove uneeded null assertion
2020-03-10 15:46:37 +00:00
Christian Sailer
ab6bf315c4 ENT-4494 Harmonize CryptoService interface/base class between OS and ENT, move OS BC implementation to node project. 2020-03-10 12:36:30 +00:00
Joseph Zuniga-Daly
a18c544de1 Don't stop the node inside the unit test
The test node had not been started and is causing tests to fail.
2020-03-09 17:17:36 +00:00
Dimos Raptis
dcf659e643
[CORDA-3628] - Implement sendAll API (#5990)
* [CORDA-3628] - Implement sendAll API

* detekt

* Some minor refactorings and docs

* Eliminate warnings

* Address Rick's comments

* Switch sendAll to use a set
2020-03-09 17:09:13 +00:00
Joseph Zuniga-Daly
b02768aa6e CORDA-3394: Move unit test to correct location 2020-03-09 15:08:40 +00:00
Joseph Zuniga-Daly
217926092d
ENT-4334 Configuration option to exclude packages from Quasar instrumentation (#6005)
* Add quasarExcludePackages configuration option

* Change quasarExcludePackages to quasar.excludePackages

* Add doc comment for QuasarConfiguration

* Use forEach method

* Fix rule violation (TooGenericExceptionThrown)

* Mention new configuration option in changelog

* Remove outer section from new quasarExcludePackages configuration option
2020-03-09 11:40:51 +00:00
Dan Newton
d5e84a4f93
NOTICK Do not cascade checkpoint tables (#6040)
Do not cascade updates to checkpoint error and result tables to hopefully
improve database performance moving forward. Because the joined tables
are no longer being updated by updating the main `DBFlowCheckpoint` entity,
they must be created/updated/deleted manually.

The checkpoint blobs still cascade as they pretty much always evolve in 
tandem with the main checkpoint table.
2020-03-09 10:01:30 +00:00
Joseph Zuniga-Daly
006dde8591 Merge remote-tracking branch 'origin/release/os/4.3' into jzd/os4.3-to-os4.4-merge-2020-03-06 2020-03-06 11:33:26 +00:00
Chris Rankin
493a92ca39 Merge commit '5b50ef49bcca2d1be8bb0604e1be167e3bb5da24' into chrisr3-44-merge 2020-03-06 09:55:00 +00:00
Matthew Nesbit
c65a5cb089
Merge pull request #6030 from corda/chrisr3-44-merge
Merge from OS 4.4 up to bc410b4.
2020-03-06 09:50:39 +00:00
Adel El-Beik
5b50ef49bc
CORDA-3651: addManifest now uses separate files for reading and writing. (#6026)
* CORDA-3651: addManifest now uses separate files for reading and writing.

* CORDA-3651: The jar scanning loader now closes itsself.

Co-authored-by: Adel El-Beik <adelel-beik@19LDN-MAC108.local>
2020-03-06 09:14:36 +00:00
Chris Rankin
2911548a2e Restore Node's missing dependency on jansi. 2020-03-04 20:56:17 +00:00
Matthew Nesbit
9a406839fa
ENT-4494 split nodeapi tests (#6024)
* Split out node-api tests that require test-utils/node-driver

* Add node-api test artefacts to publication list.

* Make test-common a transient dependency - downstream tests assume that it's available.

* Switch dependencies to java-library

* Fix magic package name for cordapp scanning in test
2020-03-04 17:59:15 +00:00
Chris Rankin
ffa2caed32 Merge branch 'release/os/4.4' into chrisr3-44-merge 2020-03-04 16:27:44 +00:00
Andrei Palade
ba68970549
Updated error message to provide consistent logging (#6015) 2020-03-04 11:43:05 +00:00
Chris Rankin
e006b871c8
CORDA-3644: Scan the CorDapp classloader directly for SerializationWhitelist. (#6014)
* CORDA-3644: Scan the CorDapp classloader directly for SerializationWhitelist.

* CORDA-3644: Filter CorDapps from out-of-process node classpaths by their manifest attributes. Also exclude directories and blatant test artifacts.

* Fix IRS Demo - its "tests" artifact had a non-standard classifier of "test".
2020-03-04 10:09:40 +00:00
Joseph Zuniga-Daly
e47e9d546d
Fix the operator used by the notEqual predicate (#6022)
* Fix the operator used by the notEqual predicate

* Add timeouts to tests

* Add change to changelog
2020-03-03 16:06:46 +00:00
Christian Sailer
e38cd9ec63
ENT-5043 separate out test utils (#5998)
* Remove unused dependencies from test-common

* Explicit imports and formatting

* Add core-test-utils project

* Add dependency

* Move Kryo serialization context to node-api (not serialization as we do not want to pull kryo into the serialization lib)

* Move AMQP server serialization scheme to node api

* Move serialization tests to node-api

* Move internal test helpers without further dependencies.

* Move out some types from RPCClientProxyHandler to node-api in preparation for moving the AMQP scheme

* Move client AMQP context to node-api so we can move the test serialization rule out.

* Move InternalSerializationTestHelpers to core-test-utils

* Moved testing.core to core-test-utils

* Make detekt happy

* Add api-scanner to core-test-utils

* Remove inlined package names introduced by IntelliJ refactoring

* Update api-current.txt to account for reordering.

* Add core-test-utils to list of published artifacts.

* Add missing import

* Location of things in api text has moved again (publish name of artefact?)

* Revert all additions to the API, leaving just the reordering

* Code review: fix up core-test-utils build.gradle and introduce kryo version constant.

* Remove OpenSsl flag from ssl config stub (can't be used from node-api)

* Suppress detekt warning

* Move core test util tests to the right module

* Expose kotlin test as a transient dependency - projects have come to rely on that.

* Fix typo in package name
2020-03-03 15:55:17 +00:00
williamvigorr3
ab000e0533
CORDA-3597 Replace old Checkpoint table with new one. (#5992)
* Replace old Checkpoint table with new one.

Adds some of the new fields into the table where needed (I have
guessed this stuff but we can update it as we go along).

* Fix database constraints + name table correctly opps.

* Fixed typos in Liquidbase script

Also corrected constraints and added missed fields in hibernate
checkpoint class and liquibase scripts.

* Update CheckpointStorage to pass in serialization context.

This is cleaner than passing both the checkpoint and the
serialized checkpoint into the methods. Also fixed
CordaPersistanceServiceTests which I accidentally broke.

* Fix detekt problem

* Revert "Update CheckpointStorage to pass in serialization context."

This reverts commit b71e78f202.

* Fix test broken by reverting commit

* CORDA-3597 Update metadata join, timestamp columns and serialization

- Change the metadata join to the checkpoints table to use
  `invocation_id` instead of `flow_id`. There were issues joining
  between the tables because `flow_id` was not the primary key of the
  metadata table. Switching over to `invocation_id` has at least allowed
  us to bypass this issue. The information about the `invocation_id` is
  stored in the `Checkpoint` class which makes it simple to save at
  runtime.
- Some of timestamp columns were nullable when they should always be
  populated, the nullable flags have now been removed.
- Previously the whole checkpoint was being serialized and stored into
  the `checkpoints_blob.checkpoint` column. This meant duplicated saving
  as the `flow_state` was contained in this object. Only the
  `CheckpointState` property of `Checkpoint` is now being serialized and
  saved to this field. Furthermore, it now uses the default
  `STORAGE_CONTEXT` serialization (AMQP) instead of Kryo (which is only
  used for serializing the `flow_state` / flow stack).
- The checkpoint database performance metrics recording has been
  abstracted to its own class.

* CORDA-3597 Make metadata join non optional

Remove the nullable declaration on the metadata field of
`DBFlowCheckpoint`

* CORDA-3597 Rename `node_checkpoints_blobs` to `node_checkpoint_blobs`

* CORDA-3597 Update some kdocs

Co-authored-by: Dan Newton <danknewton@hotmail.com>
2020-03-02 10:04:48 +00:00
williamvigorr3
9d4d128f4e
CORDA-3597 add missed data to checkpoint class (#5995)
* Add extra fields to StateMachineState

* Move structures into checkpoint as this is a more natural place.
2020-02-26 10:48:52 +00:00
Viktor Kolomeyko
275e801094
ENT-4955: Hide usage of FlowHospital inside StateMachineManager (#5997) 2020-02-25 14:10:59 +00:00
Adel El-Beik
fe625d0f37
CORDA-3584: Now cope with 2 contract jars with same hash but different name (#5952)
* CORDA-3484: Now cope with 2 contract jars with same hash but different name, we just select one and use that.

* ENT-3584: Contract jars are now generated on the fly.

* CORDA-3584: Reverted changes to CordappProviderImpl. Exception is raised if node started with multiple jars with same hash.

* ENT-3584: Fixing test failure.

* CORDA-3584: Switch to test extension method instead of reflection to access internal member.

* ENT-3584: Address review comment. Dont fully qualify exception.

* CORDA-3584: Address review comment and converted lazy to a resettable one.

* CORDA-3584: Removed unused logger.

* CORDA-3584: Fixed visibility.

* CORDA-3584: Removed synchronized

* CORDA-3584: Removed CordappResolver

* CORDA-3584: Reverted change in gradle file and fixed test.

* CORDA-3584: Removed V3 from test description as it wasn't actually V3 specific.

* CORDA-3584: Address review comment. Let classes be garbage collected.
2020-02-25 11:18:01 +00:00
Kyriakos Tharrouniatis
2c9c2985c0
CORDA-3381: Errors in vault updates publisher are unsubscribing stopping observers from working (#5912)
* Throw SQLException or PersistenceException plain, that may come out of an unsafe subscriber

* Add explanatory comment about why we changed Observer.tee to use unsafe subscribe

* Introducing not unsubscribing version of Rx.Subscriber

* Wrap PublishSubjects with FlowSafeSubjects in all tests that test Observer.tee

* Minor code formatting

* Make rawUpdates Rx.Observers not unsubscribe when accessed from CordaServices - Do not allow rawUpdates subscribing from flows

* Warning fix: Add else block to when statement

* Revert "Wrap PublishSubjects with FlowSafeSubjects in all tests that test Observer.tee"

This reverts commit e419af86

* Correcting log message

* Improve log message

* Add fiber's id to log message and exception message

* Added test, asserting FlowSafeSubscriber is alive and re-accessed upon flow retry

* Logging flow name instead of flow id at VaultService.rawUpdates subscribing error

* Add kdoc to OnNextFailedException

* Minor text correction

* Update kdocs of FlowSafeSubject/ PreventSubscriptionsSubject

* Moved FlowSafeSubject under package node.internal as it is only used by NodeVaultService

* Add comment and update kdoc explaining how to subscribe with SafeSubscriber to FlowSafeSubject

* Change PreventSubscriptionsSubject#errorAction to be more specific; to return an Exception

* Minor text update

* Update messy comment

* Replace assertThat with assertEquals

* Splitting heartBeat to heartBeat1 and hearBeat2 for more clear asserting

* Correcting comment

* Update messy comment

* Splitting heartBeat into heartBeatOnNext and heartBeatOnError

* Update test name

* Add explanatory comment to test

* Update test name

* Update test and add test comment

* Moving NotarisedTxs from SendStateFlow to VaultObserverExceptionTest inside NodeHandle.getNotarisedTransactionIds

* Moving SubscribingRawUpdatesFlow from ErrorHandling to VaultObserverExceptionTest

* Update kdoc of FlowSafeSubscriber and FlowSafeSubscriber.onNext

* Make kdoc more clear

* Throw exception upon accessing VaultService.rawUpdates from within a flow

* Changing exception thrown when accessing VaultService.rawUpdates from within a flow to a CordaRuntimeException

* Minor kdoc update

* Update test comment

* Update kdoc of FlowSafeSubscriber

* Introducing Observable.flowSafeSubscribe public API method to subscribe with -non unsubscribing- Rx.Subscribers to Observables. It also replaced FlowSafeSubject

* Move CustomSafeSubscriber outside test methods

* Minor text update

* Add timeout to tests

* Update kdoc of flowSafeSubscribe

* Update kdoc of flowSafeSubscribe

* Update kdoc of flowSafeSubscribe

* Move FlowSafeSubscriber and flowSafeSubscribe under their own package

* Fix detekt issue

* Update Detekt baseline

* Revert "Update Detekt baseline"

This reverts commit 793a8ed9

* Fix Detekt issue

* Moved strictMode flag from flowSafeSubscribe to OnFlowSafeSubscribe

Moved OnFlowSafeSubscribe into internal package

Integration tested flowSafeLooseSubscribe

* Suppress Rx Deprecation

* Rename flowSafeSubscribe to flowSafeObservable

* Renaming flowSafeObservable to continueOnError and FlowSafeSubscriber to ResilientSubscriber
2020-02-21 10:02:34 +00:00
Chris Rankin
4c6f15c40c Merge commit '88008b71f03a539749e32e5e76b2dc5ea98f897f' into chrisr3-44-merge 2020-02-19 14:12:10 +00:00
Chris Rankin
afcab51fe3 Update "in-process" test on basis it can be made to work. 2020-02-19 10:42:44 +00:00
Chris Rankin
d89ce6608a Create a unit test for serialization whitelists via driver. 2020-02-19 10:42:34 +00:00
williamvigorr3
8d1b6cf499
CORDA-3432 update structure of checkpoint class (#5983)
* Split StateMachine State into 2 classes

The idea is this better reflects the database structure. Added a
few helper methods to copy and update state.

* Doc + Improve Checkpoint API

* Rename methods to be more clear
2020-02-18 15:31:46 +00:00
Chris Rankin
73e6a40158 Merge commit '21325dfa4fb8151e77f368e9356c16ae89c70aa3' into chrisr4-44-merge 2020-02-15 14:45:50 +00:00
James Higgs
3547b629c3
[NOTICK] Add a detekt rule to catch tests with no timeout (#5959)
* [NOTICK] Add a custom detekt rule for tests with no timeout, and fix remaining missing timeouts

* [NOTICK] Add a test for custom detekt rules and tidying

* add timeout annotation to new test

Co-authored-by: Stefano Franz <roastario@gmail.com>
2020-02-14 17:31:02 +00:00
James Higgs
da192bcf0d
[EG-140] Allow system property paths with multiple keys to be specified in node.conf (#5963)
* [EG-140] Allow system property paths with multiple keys to be specified in node.conf

* [EG-140] Split property paths to remove quotes

* [EG-140] Quote system properties in docs

* [EG-140] Rename path to key
2020-02-14 11:18:21 +00:00
stefano
13c52e4901 Merge remote-tracking branch 'origin/feature/checkpoint_table_improvements' into feature/checkpoint_table_improvements 2020-02-13 12:08:07 +00:00
Schife
a765510cf4 Merge branch 'release/os/4.4' of https://github.com/corda/corda into TM-197-4.5Merge 2020-02-12 14:33:32 +00:00
Razvan Codreanu
c6ab07d92c
TM-197 Fix failing test caused by regression and ignoring flaky DistributedServiceTests (#5934)
* TM-197 Setting bouncy castle provider in order for the test to pass

* TM-197 setting timeout for all builds at 3 hours

* TM-197 ignoring unstable tests
2020-02-12 14:26:06 +00:00
Razvan Codreanu
42eca48a02
TM-197 Build stability changes (#5947)
* TM-197 switching 4.3 to use local k8s instances and also make the maximum duration of builds 3 hours, fix 1 test and ignore 2 flaky ones

* update to use local-k8s version of the plugin

Co-authored-by: Stefano Franz <roastario@gmail.com>
2020-02-12 13:58:30 +00:00
Denis Rekalov
1705299bde ENT-4918: Support symbolic link for certificates directory during node registration 2020-02-11 13:14:34 +00:00
Stefano Franz
4a54374f86
Merge pull request #5940 from corda/release/os/4.4
Merge OS 4.4 -> OS 4.5
2020-02-11 11:55:53 +00:00
Stefano Franz
b23af5f0d2
add timeouts to all tests (#5875) 2020-02-11 10:14:05 +00:00
stefano
9e8ce6473d address dan review comments pt2 2020-02-10 13:22:51 +00:00
stefano
a28c15c2fd address dan review comments 2020-02-10 11:52:54 +00:00
stefano
546166e057 Merge branch 'release/os/4.4' of github.com:corda/corda into new_checkpoint_schema 2020-02-10 11:25:29 +00:00
Viktor Kolomeyko
90df56c173
CORDA-3565: Port ServiceStateSupport from ENT to OS (#5916)
* CORDA-3565: `ServiceStateSupport` and supporting classes

* CORDA-3565:Plug `ServiceLifecycleSupport` into `MessagingService`

* CORDA-3565: Detekt baseline update

* CORDA-3565: React to MessagingServer going up and addition logging for up/down

Co-authored-by: Matthew Nesbit <matthew.nesbit@r3.com>
2020-02-03 09:47:12 +00:00
Kyriakos Tharrouniatis
9ca1dd59da
CORDA-3381 Make internalUtilsKt Observer tee not wrap with SafeSubscriber (#5913)
* Make tee not wrap PublishSubjects in SafeSubscribers, otherwise a non Rx exception from an unsafe observer shuts down all other observers under the same PublishSubject

* Throw SQLException or PersistenceException plain, that may come out of an unsafe subscriber

* Revert "Throw SQLException or PersistenceException plain, that may come out of an unsafe subscriber"

This reverts commit c7b8af3fa6.

* Update Detekt baseline
2020-01-31 12:32:59 +00:00
Florian Friemel
cb6ed6042c
Move jimfs dependency from node to test-utils. (#5911) 2020-01-31 10:55:05 +00:00
stefano
eec9ada6ba persist some nonsense data to see if it breaks the statemachine 2020-01-30 18:18:45 +00:00
jmacmahonr3
9511329e3d
[EG-75] Not handled exception when NetworkMap not accessible (#5903)
* Just passing in the exception was printing the entire stacktrace to the log, now we just pring the exception and message

* Updating exception message
2020-01-30 17:26:22 +00:00
stefano
2b079bd92e working node startup with new tables and entities 2020-01-30 12:17:26 +00:00
Christian Sailer
cac96d81bf Remove unused dependency on IntelliJ forms 2020-01-30 10:48:00 +00:00
Oliver Knowles
286fffd67a
ENT-4368 - Improve error handling when required included config resource cannot be found. Add note in docs encouraging use of required syntax for includes. (#5907) 2020-01-30 10:10:24 +00:00
stefano
7b3da95456 flesh out entities for new checkpointing 2020-01-29 17:10:34 +00:00
Ryan Fowler
d16e1126db
CORDA-3542: Use the config values for reconnecting retry interval and max reconnect attempts (#5869) 2020-01-29 13:08:54 +00:00
stefano
78d83e9583 start adding hibernate entities 2020-01-28 16:55:14 +00:00
stefano
32bdd9728c Merge branch 'release/os/4.4' of github.com:corda/corda into new_checkpoint_schema 2020-01-28 16:05:29 +00:00
Nick Dunstone
a4cada4a2e ENT-4474 O/S Changes to support bulk backchain fetching (#5894)
* Ongoing dev multi backchain fetch

* Final dev pre logging message improvements

* Trace messages

* Trace messages

* Code tidy up

* Code review comments

* Code review comments

* Whitespace removed

* Code review changes

* Code reformatting
2020-01-28 09:04:18 +00:00
Ryan Fowler
e8d6701f87 CORDA-3558: Allow initial registration errors to propagate up so the node exits with a failure code (#5899) 2020-01-27 15:23:07 +00:00
stefano
a3bc624e16 add message to exceptions table 2020-01-27 12:19:14 +00:00
stefano
47c14673c4 address initial review comments 2020-01-27 11:20:13 +00:00
Kyriakos Tharrouniatis
7f62046c2f CORDA-3329 Exceptions thrown in raw vault observers can cause critical issues (#5816)
Observers registered on NodeVaultService#rawUpdates, if they throw an exception when called from serviceHub#recordTransactions and if this exception is not handled by the flow hospital, then this leads to the transaction not being recorded in the local vault. This could get the ledger in an out of sync state.

In the specific case this happens within FinalityFlow#notariseAndRecord this leads to the transaction being notarized but not recorded in the local vault nor broadcasted in any counter party. The -failed to be recorded locally- transaction and its output states are not visible to any vault, and its input states not able to consumed by a new transaction, since they are recorded as consumed within the Notary. In this specific case we need not loose, by any means, the current transaction.

We will handle all cases by catching all exceptions thrown from serviceHub#recordTransactions, wrapping them with a HospitalizeFlowException and throwing it instead. The flow will get to the hospital for observation to be retried from previous checkpoint on next node restart.
2020-01-24 18:21:07 +00:00
Viktor Kolomeyko
a4d63d1329 CORDA-2942: Remove multiple reps from the test (#5893)
... as it is proven to be stable now.
2020-01-24 16:57:31 +00:00
Dan Newton
ecc3b5e330 CORDA-3554 Use flowId from ExternalMessageEvent when failing to init sessions (#5890)
Use `flowId` from `ExternalMessageEvent` when failing to init sessions instead of generating
a new random UUID. The a `flowId` is generated and stored inside the event after the
state machine work that done previously.
2020-01-24 16:34:29 +00:00
stefano
5750c39348 start work on liquibase schema for new checkpoints 2020-01-24 13:24:08 +00:00
Dan Newton
4f1777adb4 NO-TICK Cap the default size of the external operation thread pool (#5878)
Cap the default size of the external operation thread pool to 10 or
the maximum number of available processors, whichever is smaller.

Set the minimum size of the thread pool to 1. Meaning that only a
single thread is used unless the node actually starts to use
`FlowExternalOperation` which consumes threads from this pool.
2020-01-24 12:11:53 +00:00
stefano
0619b4ce90 add 2020-01-24 12:02:16 +00:00
Viktor Kolomeyko
42a2ed98e2 CORDA-2942: Allow exception from CordaService creation to propagate (#5884)
* CORDA-2942: Allow exception from `CordaService` creation to propagate

It will ultimately be thrown from Node's `start()` method terminating the node start-up sequence.

* CORDA-2942: Be lenient when retrievign the name of the Notary

Some tests setup such that they do nto have Notary running.
2020-01-24 10:20:08 +00:00
Christian Sailer
1380779a9c
Remove unused schema options (#5885) 2020-01-24 08:49:59 +00:00
Viktor Kolomeyko
b72f71e7ac CORDA-3549: Re-write flaky CordaServiceLifecycleFatalTests (#5879)
* CORDA-3549: Improve stability of `CordaServiceLifecycleFatalTests`

* CORDA-3549: Bump-up reps count to ensure that test is definitely not flaky when executed by CI
(once proved the number of reps will be reduced)

* CORDA-3549: Making Detekt happier

* CORDA-2942: Ensure `NodeLifecycleEventsDistributor` cleans-up smoothly when node shuts down
2020-01-23 16:35:49 +00:00
James Higgs
9a1d46ddd1 EG-73 - Remove postgres dependency from the node and update docs (#5867) 2020-01-22 09:52:02 +00:00
Dan Newton
4bae045a58 ENT-4601 Public API to run external operations from a flow (#5833)
Deprecate FlowAsyncOperation and reimplement public versions FlowExternalOperation and FlowExternalAsyncOperation.

await added to FlowLogic to allow easy calling from both Java and Kotlin. There are two overrides of await (one for FlowExternalOperation and FlowExternalAsyncOperation).

Implementations of FlowExternalOperation return a result (written as blocking code) from their execute function. This operation will then be executed using a thread provided by the externalOperationExecutor.

Implementations of FlowExternalAsyncOperation return a future from their execute function. This operation must be executed on a newly spawned thread or one provided by a thread pool. It is up to developers to handle threading in this scenario.

The default thread pool (externalOperationExecutor) can be configured through the flowExternalOperationThreadPoolSize node config.

The current implementation leaves FlowAsyncOperation alone, meaning that any developers that have used it (even though it is internal) won't need to change their apps. If this was not concern I would delete it completely and replumb the state machine code. Instead, it has been marked with @DoNotImplement and executeAsync is annotated with @Deprecated
2020-01-22 09:27:17 +00:00
Viktor Kolomeyko
0978500a9a CORDA-2942: Node lifecycle events (#5846)
* CORDA-2942: Port minimal set of changes to make lifecycle events work

... and make codebase compile.

* CORDA-2942: Undo some changes which are not strictly speaking necessary

* CORDA-2942: Make `NodeServicesContext` leaner and delete `extensions-api` module

* CORDA-2942: Reduce even more number of files affected

* CORDA-2942: Integration test fix

* CORDA-2942: Make events `AfterStart` and `BeforeStop` generic w.r.t. `NodeServicesContext`

* CORDA-2942: `NodeLifecycleObserverService` and a set of integration tests.

Public API violations are expected as well as integration tests failing.

* CORDA-2942: Re-work to introduce `ServiceLifecycleObserver`

* CORDA-2942: Explicitly mention a type of exception that may be thrown for some events.

* CORDA-2942: Register `ServiceLifecycleObserver` through `AppServiceHub`

* CORDA-2942: Fix integration test + KDocs update

* CORDA-2942: Detekt and `api-current` update

* CORDA-2942: Improvement to `CordaServiceLifecycleFatalTests`

... or else it has side effects on other tests.

* CORDA-2942: Add an integration test for new API use in Java

Driver test is written in Kotlin, but services definition is written in Java.

Also KDocs improvements.

* CORDA-2942: Documentation and release notes update

* CORDA-2942: First set of changes following review by @mnesbit

* CORDA-2942: Second set of changes following review by @mnesbit

* CORDA-2942: Added multi-threaded test

* CORDA-2942: Fixes

* CORDA-2942: Undo changes to `api-current.txt`

* CORDA-2942: Bare mimimum change to `api-current.txt` for CI gate to pass.

* CORDA-2942: Address review feedback from @rick-r3

* CORDA-2942: Detekt update

* CORDA-2942: Delete `ServiceLifecycleObserverPriority` and replace it with `Int` after discussion with @mnesbit

* CORDA-2942: Introduce more `NodeLifecycleEvent` and switch services to listen for those events

* CORDA-2942: Few more changes after input from @rick-r3

* First stub on integration test
Unfinished - hang on issue and pay

* CORDA-2942: Switch to use out-of-process nodes for the inetgration test

Currently Alice and Notary stuck waiting to hear from each other.

* CORDA-2942: Extra log lines during event distribution

* CORDA-2942: Asynchronously distribute lifecycle events

* CORDA-2942: Await for complete P2P client start-up

Next step: Add vault query to integration test

* CORDA-2942: Asynchronously distribute lifecycle events

Next step: Improve integration test

* CORDA-2942: Fix test broken by recent changes and improve logging

* CORDA-2942: Improvement of the test to be able to monitor actions performed by @CordaService in the remote process

* CORDA-2942: Add node re-start step to the integration test

* CORDA-2942: Remove `CORDAPP_STOPPED` event for now

* CORDA-2942: s/CORDAPP_STARTED/STATE_MACHINE_STARTED/

* CORDA-2942: Inverse the meaning of `priority` as requested by @rick-r3

* CORDA-2942: Register `AppServiceHubImpl` for lifecycle events and put a warning when SMM is not ready.
2020-01-21 13:38:02 +00:00
Chris Rankin
a4d00b79d4 CORDA-3377: Upgrade to DJVM 1.0-RC06. (#5866) 2020-01-21 09:05:23 +00:00
Chris Rankin
49b4f66515 CORDA-3377: Upgrade to DJVM 1.0-RC05 (#5862) 2020-01-17 09:46:15 +00:00
Chris Rankin
01666ed068 CORDA-3523: DJVM custom serializers and whitelists (#5858) 2020-01-16 17:48:15 +00:00
Chris Rankin
f027fd5c77 ENT-4652: Add unit tests for parsing attachment fix-up rules. (#5859) 2020-01-16 17:19:58 +00:00
Chris Rankin
a7147c1ffd ENT-4652: Provide an "attachment fixup" mechanism to repair broken transactions. (#5825)
* Do not register cordapp custom serialisers when using attachment classloader.

* Record the URLs of CorDapp JARs that contain custom serialisers. Include these JARs as extra attachments if we discover that we're missing a custom serialiser during transaction verification.

* Check for disabled serializer when explicitly requesting a custom serializer.
Refactor test case to force use of a custom serializer.

* Tidy up basic custom serializer test.

* Also test that TransactionBuilder rejects missing custom serializers.

* Remove test whitelists, which should not be needed with custom serialisers.

* Add changelog entry. Also align TestCordappImpl.findRoots() with OS backports.

* Second approach based around CorDapps inside AttachmentStorage - report missing type descriptor or any non-composable types.

* Initial implementation of Corda-Fixup rules inside a CorDapp jar.

* Replace original "automatic attachment fixing" mechanism completely.

* First review comments: restore "missing class" logic to TransactionBuilder.

* Restore "missing class" mechanism as fallback for SignedTransaction too.
2020-01-14 15:18:51 +00:00
Ryan Fowler
4669a699c0 CORDA-3507: Use the config value for connectionRetryInterval (#5850)
* CORDA-3507: Use the config value for connectionRetryInterval rather than a hardcoded value

* CORDA-3507: Use the config value for connectionRetryInterval rather than a hardcoded value
2020-01-14 09:57:11 +00:00
Denis Rekalov
73a0782f5d CORDA-3452: Node: Configure the input of custom string in CSR (#5844)
* CORDA-3452: Node: Configure the input of custom string in CSR to be used by Identity Service

* CORDA-3452: Remove unused import

* CORDA-3452: Add test for networkServices configuration
2020-01-13 09:52:51 +00:00
James Higgs
d0543d7270 [CORDA-3436] Allow CorDapps access to node diagnostic information (#5834)
* [CORDA-3436] Allow CorDapps access to node diagnostic information

* [CORDA-3436] Fix API breakages

* [CORDA-3436] Improve documentation around diagnostics service

* [CORDA-3436] Remove CorDapps from the diagnostics information

* [CORDA-3436] Silence detekt warning
2020-01-09 11:18:32 +00:00
Ryan Fowler
e0eac8fa0d Corda 3513 rpc flow without permission (#5828)
* CORDA-3513: Don't try to reconnect for PermissionExceptions

* CORDA-3513: Don't try to reconnect for PermissionExceptions

* CORDA-3513: Add test for not reconnecting for PermissionExceptions

* CORDA-3513: Update exception message and test
2020-01-07 11:35:45 +00:00
Thomas Schroeter
5df5d01f14
[ENT-4489] Add notary doctor (#5836) 2020-01-06 12:21:50 +00:00
Viktor Kolomeyko
7f4c2ca974 CORDA-2942: Improve tests coverage for Checkpoint dumper (#5830)
* CORDA-2942: Switch to use predictable timestamp

* CORDA-2942: Validate content of dumped checkpoint

* CORDA-2942: First stub on the integration test
(no checkpoints dumped for some reason using RPC)

* CORDA-2942: Reduce checkpointing code to bare minimum

* CORDA-2942: Minor refactoring

* CORDA-2942: Verify dump checkpoint content
2019-12-30 10:43:59 +00:00
Denis Rekalov
bc96bea24a ENT-4628: Harmonize CryptoService and BCCryptoService between OS and ENT (#5822) 2019-12-18 16:54:39 +00:00
Denis Rekalov
8d5781db43 ENT-4628: Harmonize net.corda.nodeapi.internal.crypto between OS and ENT (#5820)
* ENT-4628: Harmonize net.corda.nodeapi.internal.crypto between OS and ENT

* ENT-4628: Fix detekt
2019-12-18 13:59:30 +00:00
Chris Rankin
545a463a5b NOTICK: Remove buildscript block from Node's build.gradle. (#5809) 2019-12-16 16:24:47 +00:00
Viktor Kolomeyko
91cdffd429 ENT-4382: Core module changes to support multi-RPC work (#5814)
* ENT-4382: Move `InvocationHandlerTemplate` into `core`

This is an internal helper which is general enough and does not have any Node specific code.

* ENT-4382: Make @CordaInternal applicable to classes

And apply it on `AttachmentTrustCalculator` which is `core/internal` interface anyway.
2019-12-16 16:03:48 +00:00
Christian Sailer
4a35b99283 CORDA-3479: Added timestamp to the node_transactions table. (#2660) (#5812)
* ENT-4237: Added timestamp to the node_transactions table.

* ENT-4237: Clock for timestamp now retrieved from ServiceHub. And now record verification time as well.

* ENT-4237: Fixed tests. Also enabled stream output in allParallelIntegrationTest.

* ENT-4237: Changed timestamp to a val.

* ENT-4237: Changed streamOutput to false for allParallelIntegrationTest

* ENT-4237: Unit tests added for new timestamp column. Also now passing a clock into DBTransactionStorage.

* ENT-4237: Added more unit tests to check timestamp

* ENT-4237: Fix test to actually change clock time when testing transaction time does not change.
2019-12-13 15:14:21 +00:00
Kyriakos Tharrouniatis
b0903efa50 CORDA-3202 Add a specific exception for flows to hospitalise themselves (#5767)
* Introducing a new type of exception and a new hospital staff member to pause flows by immediately hospitalising them.

* Renaming exception to "HospitalizeFlowException".

* Making HospitalizeFlowException an open class.

* Overloading constructors of HospitalizeFlowException to be available in Java.

* Using Throwable#mentionsThrowable.

* Moving HospitalizeFlowException in its own file.

* Update kdocs for HospitalizeFlowException and StaffedFlowHospital#SedationNurse.

* Added tests, testing various HospitalizeFlowException types thrown.

* Fix Detekt issues.

* Imports optimizing.

* Add safe casting.

* Update api-flows and node-flow-hospital docs.

* Minor code comment change.

* Add DOCSTART-DOCEND signs in HospitalizeFlowException for makeDocs. It is referenced by api-flows.rst.

* Minor change in note.

* Code formatting.

* Remove comment.

* Remove if statement that makes example worse.

* Remove redundant comment.

* Moving 'Internal Corda errors' at the bottom.

* Changing node-flow-hospital.rst as per review.

* Change HospitalizeFlowException description as per review.

* Adding an example for FlowException.

* Minor indentation fix.

* Update FlowException example label as per review.

* Correcting handling of custom exception.
2019-12-11 10:35:58 +00:00
Christian Sailer
14050826e9 ENT-4595 harmonize core and serialization (#5792)
* Harmonize serialization/core and deterministic counterparts

* Fix test for changed private alias key behaviour

* Detekt errors

* roll back project.xml
2019-12-09 14:17:48 +00:00
Viktor Kolomeyko
43205e1f1a CORDA-3471: Create CordaTransactionSupport and make it accessible through AppServiceHub (#5768)
* CORDA-3471: Create `CordaTransactionSupport` and use wherever possible instead of `CordaPersistence`

* CORDA-3471: Address comments by @mnesbit
- Relocate `CordaTransactionSupport` to `core`
- Create a lighter version of transaction - `VaultTransaction` that gives access to `session` object only.

* CORDA-3471: More changes after discussion with  @mnesbit
- Rename `VaultTransaction` into `SessionScope`.

* CORDA-3471: Revert changes to most of the files after conversation with @mnesbit and @rick-r3

* CORDA-3471: Introduce `CordaTransactionSupportImpl` and make it accessible via `AppServiceHub`.

* CORDA-3471: Minor change (comment).

* CORDA-3471: Address input from @mnesbit

* CORDA-3471: Address input from @rick-r3

* CORDA-3471: Make Detekt happier

* CORDA-3471: Add a new test that proves transactions can be started from client threads

As requested by @mnesbit

* CORDA-3471: Change log and documentation update.

As requested by @mnesbit
2019-12-04 17:18:40 +00:00
Chris Rankin
5a41ec9b82 CORDA-3464: Also scan attachment:// URLs for custom serializers. (#5769)
* CORDA-3464: Also scan attachment:// URLs for custom serializers.

* Only scan the given classloader - ignore this classloader's parents.

* Upgrade to ClassGraph 4.8.58 - for "robustness fixes".

* Register the attachment:// URL scheme using AttachmentsClassLoader.

* Add integration test for custom serializer in contract state.

* Rename Currancy -> Currantsy, just to make the point.
2019-12-03 15:50:21 +00:00
Nick Dunstone
e6f9b46584 https://r3-cev.atlassian.net/browse/ENT-3829 (#5763)
Open Source
2019-12-03 10:28:00 +00:00
Rick Parker
6e467f20b3
CORDA-3470 Fix flow async operations (#5780) 2019-12-02 14:40:07 +00:00
Christian Sailer
06f97cfed5 ENT-4610 Move tx signing to the Uniqueness provider (#5773)
* ENT-4610 Move tx signing to the Uniqueness provider

* Make detekt happy

* Remove unused imports

* Address review comment
2019-11-29 17:30:33 +00:00
Dan Newton
45d6d3ead4 CORDA-3356 Subflow ledger consistency tests + move statemachine tests to slow integration tests (#5722)
* CORDA-3356 Subflow ledger consistency tests + move statemachine tests to slow integration tests

Add tests for subflows that fail during transitions.

Split out `StatemachineErrorHandlingTest` into a series of smaller tests.

Move these tests into the `integration-test-slow` category so they are
not run against every PR.

* CORDA-3356 Fix detekt issue

* CORDA-3356 Tidy test names
2019-11-25 17:11:54 +00:00
Kyriakos Tharrouniatis
21a075b727 CORDA-3197 Fix flow has been waiting message (#5660)
* Added a timestamp property to Checkpoint getting a new Instant.now() value at every Checkpoint instantiation/ copy instantiation. FlowMonitor is now using this new property (Checkpoint#timestamp) and StateMachineState#isFlowResumed to determine which flows are actually suspended. It leaves out flows that are doing work in their FlowLogic#call method.

* Cleaner comment

* Broke FlowMonitor#logFlowsWaitingForParty into logFlowsWaitingForParty and waitingFlowsToDurations. This way waitingFlowsToDurations is modular and can be tested.

Made FlowMonitor constructor get StateMachineManager instead of the retrieveFlows lamda. This way FlowMonitor is more consistent as a service, and entire flow filtering process is now being done in FlowMonitor#waitingFlowsToDurations.

Removed "smm as? StateMachineManagerInternal" in AbstractNode#start as it made no sense.

Updated CheckpointDumper to mention the Checkpoint#timestamp when writing the checkpoint as json.

* Added tests for FlowMonitor service.

* Remove old comment

* 1. FLowMonitor#waitingFlowDurations now returns a Sequence to have an iteration less.
It used to be, one iteration from returning a Set from FLowMonitor#waitingFlowDurations plus one iteration from FlowMonitor#logFlowsWaitingForParty.

2. Code reformattings

* 1. Remove constructor keyword from FlowMonitor

2. Code reformattings

3. Update detekt baseline

* Resolve conflict in Detekt baseline
2019-11-25 16:49:35 +00:00
Ed Prosser
e692ec292a Merge branch 'release/os/4.3' of https://github.com/corda/corda into EdP/CORDA-3446-4.4 2019-11-20 10:36:38 +00:00
Stefan Iliev
926429647d CORDA-3307: Fix for underscore variables (#5682)
* Revert "Revert "CORDA-3307 - add support for environment variables in linux (#5523)" (#5643)"

This reverts commit 03ab258fc2.

* Env variables with underscore are now validated using schema validation and checking for unknown key errors.

* Resolving comments from PR review.

* Fix for deprecated import.

* Reworked logic according to PR review.

* Resolved bad string parsing problems where the json structure could be broken if some symbols were included in the key or value.
2019-11-19 17:51:52 +00:00
Chris Rankin
ce8ce2ba7c CORDA-2876: Upgrade to deterministic-rt 1.0-RC01. (#5738) 2019-11-19 15:02:35 +00:00
Denis Rekalov
a791e38319 CORDA-3443: Do not log warning when appending duplicated network parameters (#5739) 2019-11-19 15:02:07 +00:00
Stefano Franz
221576d94a
move - responding flow from suffix to prefix of test name (#5731) 2019-11-18 15:41:15 +00:00
Chris Rankin
c68f8161fc CORDA-2876: Upgrade DJVM to 1.0-RC03. 2019-11-16 16:51:24 +00:00
Oliver Knowles
4a8c73bf4a CORDA-2744 - Prevent ClassCastException if error during initial registration (#5712)
* Prevent class cast exception if error during initial registration

* Appeasing detekt
2019-11-14 16:13:08 +00:00
Jonathan Locke
c31cddaffc Fixed merge conflicts 2019-11-14 10:40:12 +00:00
Chris Rankin
987415dd83 CORDA-3407: Install external byte-code cache for sandboxed CorDapps. (#5699) 2019-11-14 09:09:57 +00:00
Jonathan Locke
86f1f9e371
CORDA-3393: Lazy load MessageDeduplicationHandler.flowId (#5679)
CORDA-3393: Lazy load `MessageDeduplicationHandler.flowId` (#5679)
2019-11-07 09:44:03 +00:00
Dan Newton
b15db200e3 NOTICK: Quick and dirty change to stop "Unable to start notaries." error message (#5686)
* Quick and dirty change to stop "Unable to start notaries." error message (#5686)

"Unable to start notaries. A required port might be bound already" is
returned whenever a startup error occurs while starting the notary nodes
in driver tests. This hides the real error.

This change prints the actual error to std_err and read from file
at a later point. This means the real error is not lost and will be
shown in failed builds.

* Suppress detekt warnings
2019-11-07 09:43:29 +00:00
stefano
5d6fc34f88 fix merge issue with exception thrown in RpcExceptionHandlingTest 2019-11-06 12:28:05 +00:00
LankyDan
91f35d5967 CORDA-3393 Lazy load MessageDeduplicationHandler.flowId
This is to potentially help with debugging in the future as the
`flowId` could become confusing for received messages where the `flowId`
has nothing to do with the current flow.
2019-11-06 11:26:25 +00:00
stefano
03d8009c7d fix temp folder rules 2019-11-05 17:56:06 +00:00
stefano
d8ed3492b9 fix merge conflicts due to changes in Signed test cordapp creation 2019-11-05 17:42:38 +00:00
stefano
6a3a9eded3 Merge remote-tracking branch 'origin/release/os/4.3' into my_merge_branch
# Conflicts:
#	core/src/main/kotlin/net/corda/core/internal/TransactionVerifierServiceInternal.kt
#	node/src/integration-test/kotlin/net/corda/node/services/rpc/RpcExceptionHandlingTest.kt
#	testing/node-driver/src/main/kotlin/net/corda/testing/node/internal/DriverDSLImpl.kt
2019-11-05 17:19:18 +00:00
Chris Rankin
f226ddc4f2 CORDA-2876: Integrate the DJVM with the Corda Node. (#5633)
* * CORDA-2876: Migrate DJVM serialization modules into Corda.
* Pre-generate Corda classes for DJVM sandbox when node boots in production mode.
* Ensure that all DJVM test contract CorDapps are signed.
* Test examining attachments within DJVM sandbox.
* Test Contract.verify() using cryptographic verify function.
* Add test cases for more non-determinism in Contract.verify().
* Update node-driver to support testing nodes with DJVM support.
* Modify Node to allow alternative DJVM configurations for testing.
* Refactor DeterministicVerifierFactoryService for default use-case.
* Small whitespace and code-style refactors.
* Create and activate a DJVM execution profile for the Node.
* Revert making Verifier implement AutoCloseable.
* Allow the node to cache sandboxed Corda byte-code for reuse.
* Use updated Quasar agent that knows not to touch DJVM classloaders.
* Fix Quasar's package exclusions globs for DJVM.
* Deserialise LedgerTransaction into the sandbox for Contract.verify().
* Add the DJVM's serialisation modules to the Corda node.
* Update the node for the latest DJVM API, and preserve the ConstructorForDeserialization annotation on user contract classes.
* Add corda-dev to repositories while DJVM is SNAPSHOT.
* Migrate DJVM specialisation into AbstractNode's ServiceHubInternalImpl.
* Exclude sandbox.** and shaded djvm.** classes from Quasar agent.
* Add the corda-dev repository to :node for the deterministic runtime.
* Turn Verifier into an abstract base class that is specialised by BasicVerifier and DeterministicVerifier.
* Add the Corda deterministic libraries to the Node, and split the DJVM sandbox across two SandboxClassLoader instances.
* Add DJVM to contract verification path inside Corda Node.
* Minor lambda simplifications and removing unused import.
* CORDA-2871: Remove @CordaSerializable from LedgerTransaction.
* CORDA-2871: Add a callback to ServicesForResolution to allow the Node to modify a LedgerTransaction object.
* CORDA-2871: Refactor the contract verification code into a separate class,
 and allow LedgerTransaction to choose different Verifier objects.
* Update DJVM to use Corda 4.4-SNAPSHOT. (#95)
* CORDA-3330: Allow DJVM to preload / pregenerate classes from selected jars. (#92)
* Add support for SourceClassLoader.getResources() to DJVM.
* Allow a SandboxConfiguration to preload sandbox byte-code for all classes inside jars containing META-INF/DJVM-preload.
* CORDA-3309: Remove explicit try-catch in favour of UncaughtExceptionHandler. (#91)
* CORDA-3309: Install UncaughtExceptionHandler for DJVM tasks. (#88)
* Fix tests broken by Windows line endings. (#82)
* CORDA-3292: Reimplement ExecutionProfile as a data class. (#80)
* CORDA-2877: Refactor how we create child SandboxConfiguration objects. (#76)
* CORDA-2877: Load bytecode from a persistent cache to prevent repeated rewriting. (#75)
* Refactor byte-code cache to SandboxConfiguration instead of AnalysisConfiguration. We cannot "mix and match" byte-code generated by different sets of rules.
* CORDA-3137: Enhance annotation handling so that we can allow some annotations to be mapped into the sandbox without also needing to be stitched. (#72)
* CORDA-2871: Minor cosmetic fixes. (#69)
* CORDA-3218: Align DJVM with internal Corda Serialisation API. (#68)
* Ensure we get the latest SNAPSHOT of the serialisation code.
* CORDA-2871: Refactor SourceClassLoader to define source classes. (#66)
* Rewrite SourceClassLoader to support parent/child relationships.
* Revert catching TypNotPresebtException - it was a symptom of a bigger problem.
* Remove AutoCloseable from AnalysisConfiguration and SourceClassLoader.
* SourceClassLoader.getResource() must delegate to its parent first.
* CORDA-2871: Ensure ClassLoader.loadClass() throws ClassNotFoundException for all cases where the class cannot be found. (#64)
* CORDA-2871: Modify sandbox tasks to implement both java.Function and sandbox.Function (#62)
* Make TaskExecutors implement BiFunction to make them composable.
* Create ImportTask to wrap a java.Function inside a sandbox.Function.
* Add createExecutor() and createRawExecutor() APIs to SandboxClassLoader.
* Update serialization to use SandboxClassLoader.toSandboxClass().
* Remove a layer of lambdas from the serialisation code.
* Update SandboxExecutor and SandboxRawExecutor.
* Rename Executor to TaskFactory.
* Rename dangling executor -> taskFactory.
* CORDA-2871: Sanity fixes! (#63)
* Improve message for SandboxClassLoadingException.
* Fix serialisation API for using sandboxed environment.
* CORDA-3174: Extend serialisation to include InputStream and OpaqueBytesSubSequence. (#60)
* Update DJVM Example project for serialisation.
* Add serializers for InputStream and OpaqueBytesSubSequence.
* Support ZIP Inflater and CRC32 inside the sandbox.
* Allow the DJVM to wrap java.io.InputStream as sandbox.java.io.InputStream.
* Configure tests also to preserve @DeprecatedConstructorForDeserialization.
* CORDA-3174: Implement Corda serialization modules. (#59)
* Create DJVM serialization modules.
* Create test cases for Array<T>, List<T> and List<Array<T>>.
* Refactor SandboxPrimiveSerializer for all primitive types.
* Implement SandboxCollectionSerializer to support Collection types.
* Implement SandboxMapSerializer to support Map types.
* Attempt to fix infinite loop when computing Collection and Map fingerprints.
* Apply special handling when deserialising sandbox.java.lang.Character.
* Remap Java primitive types to sandbox Java object types to deter evolution.
* Use Class.getPackage().getName() to determine sandbox package name.
* Implement SandboxEnumSerializer to support Enum types.
* Implement SandboxPublicKeySerializer to support Java security keys.
* Add serialization projects to the composite example project.
* Implement serializers for BigInteger, BigDecimal, Currency and StringBuffer.
* Test that deserialising does not instantiate the untrusted user classes.
* Implement serializers for java.time.* types.
* Add serialiser for BitSet - currently disabled until BitSet itself is supported.
* Add serialisers for EnumSet and Class.
* Include support for EnumMap in the SandboxMapSerializer.
* Ensure the DJVM Example project's tests preserve @CordaSerializable.
* Add support for UUID as a primitive type.
* Use common abortReadOnly() method for declaring serialization as unsupported.
* Streamline the API for deserialising into the sandbox.
* Add preliminary support for deserialising X.509 certificates.
* Implement serializer for java.util.Optional.
* Refactor configuration of the sandbox serialization scheme.
* Add tests for deserialising arrays of basic types.
* Include method annotations in annotation stitching. This ensures that `@ConstructorForDeserialization` is not dropped.
* Enable test for SandboxBitSetSerializer.
* Enable tests for X.509 serializers.
* Implement serializers for ProtonJ primitive types.
* Serialize java.util.Date as a primitive type.
* Add the bintray Gradle plugin to the serialisation modules.
* Do not publish serialisation modules - they will become part of Corda itself.

* CORDA-2876: Only apply DJVM sources to Node Driver when devMode=true.

* Resolve DeteKT warnings.

* Require Node's JVM to set -Dnet.corda.djvm=true in order to enable DJVM.

* Enable DJVM for DemoBench nodes.

* Disable Quasar instrumentation verification for DemoBench nodes.

* Upgrade to DJVM 1.0-RC01.

* Try to modify DriverParameters in a more "ABI friendly" way.

* Refactor and simplify sandbox deserialisation of primitive objects.

* Review fixes.

* Update EvolutionSerializerFactory to handle sandboxed primitive boxed types.
2019-11-05 13:44:18 +00:00
LankyDan
532e4989d6 @Ignore P2PMessagingTest - communicating with a distributed service which we're part of 2019-11-05 11:52:32 +00:00
Dimos Raptis
485feb2d6c CORDA-3350: Increase size of constraints column (#5639)
* CORDA-3350: Increase size of constraints column (#5639)

* Detekt

* Update api file with new threshold

* Add check in transaction builder

* Revert "Add check in transaction builder"

This reverts commit ca3128f44c.

* Add check for max number of keys

* Update api file

* Address Tudor's comments

* Remove check for pre-5 and add test for EC keys

* fix typo and rename liquibase script

* updated docs with measurement numbers for composite keys

* Make detekt happy again
2019-11-05 11:00:26 +00:00
stefano
5ac1e50135 Merge remote-tracking branch 'origin/release/os/4.3' into my_merge_branch 2019-11-04 17:48:15 +00:00
LankyDan
54394f6747 Release state soft locks when a flow is killed via killFlow 2019-11-04 13:21:06 +00:00
LankyDan
3c0631a26a Fix VaultObserverExceptionTest due to differences between ENT and OS 2019-11-04 09:42:41 +00:00
LankyDan
be2e34b33a Fix error from warning from detekt 2019-11-01 16:00:09 +00:00
LankyDan
3aaddb47ea Attempt to fix detekt issues in DriverDSLImpl again 2019-11-01 14:36:47 +00:00
LankyDan
b05bd76a77 Fix error in VaultObserverExceptionTest due to bad merge 2019-11-01 13:38:42 +00:00
LankyDan
e737b01184 Port ledger integrity work to SingleThreadedStateMachineManager
- Port ledger integrity work to `SingleThreadedStateMachineManager`
- Fix `StatemachineErrorHandlingTest`
- Fix compile errors in `RetryFlowMockTest` + `VaultObserverExceptionTest`
- Add method to `StaffedFlowHospital` that was missed during original merge
2019-11-01 11:50:16 +00:00
Christian Sailer
f670cf774b Suppress/fix more warnings 2019-11-01 11:48:07 +00:00
Christian Sailer
1f82213827 Suppress/Baseline detekt issues that were highlighted freshly because of the merge. 2019-11-01 11:48:07 +00:00
Christian Sailer
5b9c5a6b83 Fix and or suppress detekt warnings 2019-11-01 11:48:07 +00:00
Christian Sailer
119f939ee1 Fix and or suppress detekt warnings 2019-11-01 11:48:07 +00:00
Christian Sailer
48aa2f2faa Update compiler.xml for new projects (and code style settings) 2019-11-01 11:48:07 +00:00
LankyDan
c5124689ca CORDA-3194 Update TODOs with jira tickets 2019-11-01 11:48:07 +00:00
Christian Sailer
8f2923c7a3 CID-843 code review fixes (#2681)
* Layout fix

* Fix test to use proper hook in flow hospital

* Fix comment
2019-11-01 11:48:07 +00:00