Commit Graph

2761 Commits

Author SHA1 Message Date
James Higgs
1956bdb5d4 [MERGE] Fix detekt issues 2020-04-29 11:57:37 +01:00
James Higgs
adbe030a2c Merge branch 'release/os/4.5' into jamesh/error-reporting-sync-29-04-20
# Conflicts:
#	node/src/main/kotlin/net/corda/node/internal/AbstractNode.kt
2020-04-29 11:47:14 +01:00
James Higgs
ab95aa57a2
[EG-440] Add some error codes and the error resource generation tool (#6192)
* [EG-438] First commit of error code interface

* [EG-438] Implement error reporter and a few error codes

* [EG-438] Add unit tests and default properties files

* [EG-438] Add the error table builder

* [EG-438] Update initial properties files

* [EG-438] Add some Irish tests and the build.gradle

* [EG-438] Fall back for aliases and use different resource strategy

* [EG-438] Define the URL using a project-specific context

* [EG-438] Tidy up initialization code

* [EG-438] Add testing to generator and tidy up

* [EG-438] Remove direct dependency on core and add own logging config

* [EG-438] Fix compiler warnings and tidy up logging

* [EG-438] Fix detekt warnings

* [EG-438] Improve error messages

* [EG-438] Address first set of review comments

* [EG-438] Use enums and a builder for the reporter

* [EG-438] Address first set of review comments

* [EG-438] Use enums and a builder for the reporter

* [EG-438] Add kdocs for error resource static methods

* [EG-440] Add error code for duplicate CorDapp loading

* [EG-438] Handle enums defined with underscores

* [EG-440] Add errors for some CorDapp loading scenarios

* [EG-440] Finish adding errors for CorDapp loading

* [EG-440] Fix up errors in properties files

* [EG-440] Start change to error code definition

* [EG-440] Update error code definition and add resource generation tool

* [EG-440] Tidy up error resource generation tool frontend

* [EG-440] Small refactorings and add kdocs

* [EG-440] Generate all missing resources

* [EG-440] Some refactoring and start writing a test

* [EG-440] Update unit test for resource generator

* [EG-440] Renaming of various parts of the error tool

* [EG-440] Add testing for errors and fix an issue in resource generation

* [EG-440] Add a kdoc for context provider API

* [EG-440] Remove old code from repository

* [EG-440] Address some review comments
2020-04-29 11:21:50 +01:00
Dan Newton
297e504740
CORDA-3291 isKilled flag and session errors for killed flows (#6170)
* CORDA-3291 `isKilled` flag and session errors for killed flows

## Summary

Two major improvements have been worked on:

- A new flag named `isKilled` has been added to `FlowLogic` to allow
developers to break out of loops without suspension points.
- Killed flows now send session errors to their counter parties allowing
their flows to also terminate without further coordination.

Achieving these changes required a __fundamental__ change to how flows are
killed as well as how they sleep.

## `isKilled` flag

The addition of `FlowLogic.isKilled` allows flows to check if the
current flow has been killed. They can then throw an exception to lead
to the flow's termination (following the standard error pathway). They
can also perform some extra logic or not throw an exception if they
really wanted to.

No matter what, once the flag is set, the flow will terminate. Due to
timing, a killed flow might successfully process its next suspension
event, but it will then process a killed transition and terminate.

## Send session errors when killing a flow

A flow will now send session errors to all of its counter parties. They
are transferred as `UnexpectedFlowEndException`s. This allows initiated
flows to handle these errors as they see fit, although they should
probably just terminate.

## How flows are killed

### Before

Originally we were relying on Quasar to interrupt a flow's fiber, we
could then handle the resulting `InterruptedException`. The problem with
this solution is that it only worked when a flow was already suspended
or when a flow moved into suspension. Flows stuck in loops did not work.

### After

We now *do not* use Quasar to interrupt a flow's fiber. Instead, we
switch `FlowStateMachine.isKilled` to true and schedule a new event.
Any event that is processed after switching this flag will now cause a
`KilledFlowTransition`. This transition follows similar logic to how
error propagation works. Note, the extra event allows a suspended flow
to be killed without waiting for the event that it was _really_ waiting
for.

This allows a lot of the tidy up code in `StateMachineManager.killFlow`
to be removed as tidy up is executed as part of removing a flow.
Deleting a flow's checkpoint and releasing related soft locks is still
handled manually in case of infinite loops but also triggered as part
of the actions executed in a transition.

This required flow sleeping to be changed as we no longer rely on
quasar.

## How flows now sleep

The reliance on Quasar to make a flow sleep has been removed.

Instead, when a flow sleeps we create a `ScheduledFuture` that is
delayed for the requested sleep duration. When the future executes it
schedules a `WakeUpFromSleep` event that wakes up the flow... Duh.

`FlowSleepScheduler` handles the future logic. It also uses the same
scheduled thread pool that timed flows uses.

A future field was added to `StateMachineState`. This removes the 
need for concurrency control around flow sleeps as the code path does
not need to touch any concurrent data structures.

To achieve this:

- `StateMachineState.future` added as a `var`
- When the `ScheduledFuture` is created to wake up the flow the passed
in `StateMachineState` has its `future` value changed
- When resumed `future` and `isWaitingForFuture` are set to `null` and
`false` respectively
- When cancelling a sleeping flow, the `future` is cancelled and nulled
out. `isWaitingForFuture` is not changed since the flow is ending anyway
so really the value of the field is not important.
2020-04-28 15:53:44 +01:00
James Higgs
ab43238420
[EG-438] Error Reporting Framework (#6125)
* [EG-438] First commit of error code interface

* [EG-438] Implement error reporter and a few error codes

* [EG-438] Add unit tests and default properties files

* [EG-438] Add the error table builder

* [EG-438] Update initial properties files

* [EG-438] Add some Irish tests and the build.gradle

* [EG-438] Fall back for aliases and use different resource strategy

* [EG-438] Define the URL using a project-specific context

* [EG-438] Tidy up initialization code

* [EG-438] Add testing to generator and tidy up

* [EG-438] Remove direct dependency on core and add own logging config

* [EG-438] Fix compiler warnings and tidy up logging

* [EG-438] Fix detekt warnings

* [EG-438] Improve error messages

* [EG-438] Address first set of review comments

* [EG-438] Use enums and a builder for the reporter

* [EG-438] Add kdocs for error resource static methods

* [EG-438] Handle enums defined with underscores

* [EG-438] Slight refactoring of startup code

* [EG-438] Port changes to error reporting code from future branch

* [EG-438] Also port test changes

* [EG-438] Suppress a deliberately unused parameter
2020-04-28 14:07:50 +01:00
Dan Newton
9a2ae8ae19
CORDA-3722 withEntityManager can rollback its session (#6187)
* CORDA-3722 withEntityManager can rollback its session

## Summary

Improve the handling of database transactions when using
`withEntityManager` inside a flow.

Extra changes have been included to improve the safety and
correctness of Corda around handling database transactions.

This focuses on allowing flows to catch errors that occur inside an
entity manager and handle them accordingly.

Errors can be caught in two places:

- Inside `withEntityManager`
- Outside `withEntityManager`

Further changes have been included to ensure that transactions are
rolled back correctly.

## Catching errors inside `withEntityManager`

Errors caught inside `withEntityManager` require the flow to manually
`flush` the current session (the entity manager's individual session).
By manually flushing the session, a `try-catch` block can be placed
around the `flush` call, allowing possible exceptions to be caught.

Once an error is thrown from a call to `flush`, it is no longer possible
to use the same entity manager to trigger any database operations. The
only possible option is to rollback the changes from that session.
The flow can continue executing updates within the same session but they
will never be committed. What happens in this situation should be handled
by the flow. Explicitly restricting the scenario requires a lot of effort
and code. Instead, we should rely on the developer to control complex
workflows.

To continue updating the database after an error like this occurs, a new
`withEntityManager` block should be used (after catching the previous
error).

## Catching errors outside `withEntityManager`

Exceptions can be caught around `withEntityManager` blocks. This allows
errors to be handled in the same way as stated above, except the need to
manually `flush` the session is removed. `withEntityManager` will
automatically `flush` a session if it has not been marked for rollback
due to an earlier error.

A `try-catch` can then be placed around the whole of the
`withEntityManager` block, allowing the error to be caught while not
committing any changes to the underlying database transaction.

## Savepoints / Transactionality

To make `withEntityManager` blocks work like mini database transactions,
save points have been utilised. A new savepoint is created when opening
a `withEntityManager` block (along with a new session). It is then used
as a reference point to rollback to if the session errors and needs to
roll back. The savepoint is then released (independently from
completing successfully or failing).

Using save points means, that either all the statements inside the
entity manager are executed, or none of them are.

## Some implementation details

- A new session is created every time an entity manager is requested,
but this does not replace the flow's main underlying database session.
- `CordaPersistence.transaction` can now determine whether it needs
to execute its extra error handling code. This is needed to allow errors
escape `withEntityManager` blocks while allowing some of our exception
handling around subscribers (in `NodeVaultService`) to continue to work.
2020-04-28 11:20:00 +01:00
nikinagy
ec96a844bd
CORDA-3659 - adding kdoc for RestrictedEntityManager and RestrictedConnection (#6179)
* adding kdoc for RestrictedEntityManager and RestrictedConnection

* adding kdoc for RestrictedEntityManager and RestrictedConnection
2020-04-28 09:35:47 +01:00
Matthijs van den Bos
a298a94960
Load CordaServices before NotaryService (#6173)
On node start, load CordaServices before starting the NotaryService,
so that the NotaryService can check that the services it requires are
available when starting.

Resolves #6172.
2020-04-24 09:15:38 +01:00
nikinagy
335372cfc1 Merge branch 'release/os/4.3' into nnagy-os-4.3-os-4.4-20200423 2020-04-23 16:03:17 +01:00
Joseph Zuniga-Daly
69a4f80cd2
ENT-5141: Fix ConcurrentModificationException in FetchDataFlow (#6176)
* ENT-5141: Fix ConcurrentModificationException in FetchDataFlow

* Make detekt happy

* Make CheckpointSerializationEnvironmentRule inheritable
2020-04-23 13:11:23 +01:00
Viktor Kolomeyko
257026f606
CORDA-3726: Remove memory leaks when many InProcess nodes started (#6169)
* CORDA-3762: Integration test exposing the problem reported

* CORDA-3726: Additional logging

* CORDA-3726: Prevent thread leaks

* CORDA-3726: New `journalBufferTimeout` parameter

* CORDA-3726: Override `journalBufferTimeout` parameter

* CORDA-3726: Making Detekt happier

* CORDA-3276: Account for extra thread user in MockNetwork

For real node this does not matter as `shutdown` can safely be called multiple times, which is not true for server thread provided by MockNetwork

* CORDA-3276: Do not make SMM shutdown "executor" as it belongs to AbstractNode

* CORDA-3276: Address input from @rick-r3

* CORDA-3276: Fix test after rebase
2020-04-23 08:53:51 +01:00
Denis Rekalov
0d441c3760
ENT-4912: Enable CRL checking with embedded Artemis (#6154) 2020-04-22 10:35:58 +01:00
nikinagy
2bcaa2ac80
CORDA-3569 - Add RestrictedConnection and more blocked methods to RestrictedEntityManager (#6129)
* adding blocked functions ro RestrictedEntityManager and creating RestrictedConnection class

* adding flow tests and fixing issues regarding the review

* adding quasar util to gradle

* updating flow tests

* adding space before } at .isThrownBy()

* adding spaces
2020-04-21 14:39:41 +01:00
Dimos Raptis
8faf72f7b5
[ENT-5210] - Whitelist SNAPPY encoding (#6163)
* [ENT-5210] - Whitelist SNAPPY encoding

* Remove unused imports
2020-04-20 08:09:38 +01:00
Ramzi El-Yafi
45b43f116d
[EG-503] Test cases for exposed notaryService (#6161) 2020-04-17 16:51:27 +01:00
Ramzi El-Yafi
6a07284324
[EG-503] Spent state audit tool (#6107)
* [EG-503] Spent state audit tool

Fixes

* Refinements to notary query interfaces. Feature complete.

* EG-503: Introduce optional `notaryService` in `ServiceHubCoreInternal`

* Remove redundant logic following change to use extensions API

Co-authored-by: Viktor Kolomeyko <viktor.kolomeyko@r3.com>
2020-04-16 16:05:21 +01:00
Adel El-Beik
27ea570fbb
CORDA-3696: JDK 11 Testing branch (#6131)
* CORDA-3696: Temporary update to enable JDK11 build and test. Will eventually be switchable.

* CORDA-3696: Filter out the Nashorn warning.

* CORDA-3696: Add JDK11 classifier.

* CORDA-3696: Updated match string to cope with JDK11.

* CORDA-3696: Filtering out SPHINCS256_SHA256 where failing due to JDK11.

* CORDA-3696: Now remove SPHINCS256_SHA256 only if JDK11.

* CORDA-3696: Fix test failure - switch to regex matching.

* CORDA-3696: Hide the illegal access warnings.

* CORDA-3696: Check for Java11 when disabling Java11 warnings.

* CORDA-3696: Fix unneccessary non null check.

* CORDA-3696: Reverting build env to JDK8

* CORDA-3696: Revert hiding of illegal access warnings via Unsafe class.

* CORDA-3696: Remove internal access warnings and new JDK11 version checker.

* CORDA-3696: Updated build file for OS

* CORDA-3696: Removed typo

* CORDA-3696: Fixed space typo.

* CORDA-3696: Open modules to remove the illegal access warnings.

Co-authored-by: Adel El-Beik <adelel-beik@19LDN-MAC108.local>
2020-04-16 10:20:30 +01:00
Chris Rankin
39cc5e3403
CORDA-3698: Require no classifier for Open Core and DJVM-related modules. (#6132)
* ENT-4967: Require no classifier for corda-node-djvm, corda-deserializers-djvm.

* Also remove classifiers from core, serialization and finance-contracts.

* Compile corda-serialization-djvm for Java 8 and remove its classifier.
2020-04-06 11:00:40 +01:00
Viktor Kolomeyko
1d7c13276c
ENT-5089: Forcibly free associated with transport resources if it is already closed (#6130) 2020-04-03 17:30:29 +01:00
Stefan Iliev
1718af3a79
Notary logs warning on startup when it's not in the whitelist (#5857) 2020-04-02 10:50:03 +01:00
Chris Rankin
a7358ffa1a
CORDA-3643: Initialise MockNetwork's serialisation environment with all custom cordapps. (#6115) 2020-04-01 17:42:29 +01:00
Chris Rankin
0771a22d5b
ENT-4967: Allow quasar_classifier to be either null or an empty string. (#6119) 2020-03-31 15:32:36 +01:00
Chris Rankin
f1ebaa761b
CORDA-3680: Add CorDapp custom serialisers to Driver's in-process nodes. (#6098)
* Run serialisation tests with both in-process and out-of-process nodes.

* Add custom serialisers and whitelists to Driver's AMQPServerSerializationScheme.
2020-03-27 10:01:30 +00:00
Dan Newton
678fb6eb94
Merge pull request #6108 from corda/dan/os-4.4-to-4.5-merge-2020-03-26
OS 4.4 to 4.5 merge 2020-03-26
2020-03-27 09:39:41 +00:00
Chris Rankin
cccbbe1c80
CORDA-3680: Add CorDapp custom serialisers to Driver's in-process nodes. (#6102)
* Run serialisation tests with both in-process and out-of-process nodes.

* Add custom serialisers and whitelists to Driver's AMQPServerSerializationScheme.
2020-03-27 09:16:31 +00:00
Dimos Raptis
963de40902
[NOTICK] - Enable check in detekt for unused imports (#6106)
* [NOTICK] - Enable check in detekt for unused imports

* Put back accidental removal of used import

* Some more accidental removals
2020-03-26 15:46:33 +00:00
LankyDan
d9c1907c88 Merge branch 'release/os/4.4' into dan/os-4.4-to-4.5-merge-2020-03-26 2020-03-26 14:26:41 +00:00
Joseph Zuniga-Daly
b766eff284 Merge remote-tracking branch 'origin/release/os/4.3' into jzd/merge-os4.3-into-os4.4-2020-03-26 2020-03-26 11:01:39 +00:00
Denis Rekalov
b1180b467b ENT-4659: Remove CryptoServiceFactory from OS 2020-03-25 12:56:33 +00:00
Joseph Zuniga-Daly
2dbf90cafe
ENT-4857: Fix race condition in trackTransaction (#6096)
- Fix issue
- Emit warning if we are inside a DB transaction
- Include a path that does not emit warning
- Add unit tests
2020-03-25 11:53:06 +00:00
Dimos Raptis
b73a498062
[ENT-4754] - Move subflow preparation logic in FlowStateMachine 2020-03-25 09:02:14 +00:00
Ramzi El-Yafi
c697b5850b
[EG-896] Additional uniqueness provider test cases (#6085) 2020-03-24 11:10:27 +00:00
LankyDan
82e068be94 Merge branch 'release/os/4.4' into dan/os-4.4-to-4.5-merge-2020-03-23 2020-03-23 11:46:24 +00:00
Dan Newton
668748b054
CORDA-3669 Do not execute ExecuteAsyncOperation multiple times (#6087)
* CORDA-3669 Do not execute `ExecuteAsyncOperation` multiple times

When a `FlowExternalOperation` or `FlowExternalAsyncOperation` executes
and completes a flag (`isFlowResumed`) is switched to true.

This flag was used inside of `DoRemainingWorkTransition` to decide
whether to skip over the execution of an event.

Since this flag was being switched to true when the external operation's
 future completed, it was possible for _unexpected_ events to be placed
in the fiber's queue that would retrigger the
`FlowIORequest.ExecuteAsyncOperation`, that is held as the checkpoint's
next `FlowIORequest`to process.

By using the existing `StateMachineState.isTransactionTracked` (and
renaming it to `isWaitingForFuture`) we can decide to not process the
`FlowIORequest.ExecuteAsyncOperation` if it has already been called
before. This moves this code path in line with
`FlowIORequest.WaitForLedgerCommit`.

Random `DoRemainingWork` events can now be pushed to the fiber's queue
without causing the `FlowIORequest.ExecuteAsyncOperation` to execute
again.
2020-03-20 19:02:34 +00:00
Christian Sailer
82d9995717
ENT-5109 Harmonize config-common, make everything compile again and harmonize NetworkParameterOverridesSpec. (#6082) 2020-03-19 15:43:08 +00:00
Dimos Raptis
56067acd20
[CORDA-3628] - Remove overloads for sendAll (#6078) 2020-03-18 14:37:04 +00:00
Chris Rankin
29a36c6b4f
Prevent Quasar from instrumenting classes that belong to AttachmentsClassLoader. (#6077) 2020-03-18 13:59:10 +00:00
Ryan Fowler
349bd5a511
CORDA-3662: Use an INNER JOIN for network map cache queries, (#6062)
- rename add or update function for clarity
- put removal of old nodes after retrieval of new ones to avoid gaps in the map
- plus add a test
2020-03-17 17:02:08 +00:00
Dimos Raptis
eba113621c
[CORDA-3628] - Avoid sending actions to the state machine if no messages are to be sent (#6074) 2020-03-17 16:51:07 +00:00
Christian Sailer
ea22a10b3e
ENT-4494 Harmonize network stack (#6059)
* ENT-4494 harmonize proton wrapper with ENT

* Harmonise Artemis and Bridge implementation

* Move tests across

* detekt changes

* Fix AMQP tests in node
2020-03-13 14:26:24 +00:00
Dan Newton
3b335ebb00
NOTICK Fix statemachine error handling tests (#6057)
Due to a change in how messaging works, `ActionExecutorImpl
.executeSendInitial` was no longer being called. Changing the byteman
script to throw exception on hits to `ActionExecutorImpl
.executeSendMultiple` allowed the tests to pass.
2020-03-12 12:48:40 +00:00
Christian Sailer
ab6bf315c4 ENT-4494 Harmonize CryptoService interface/base class between OS and ENT, move OS BC implementation to node project. 2020-03-10 12:36:30 +00:00
Joseph Zuniga-Daly
a18c544de1 Don't stop the node inside the unit test
The test node had not been started and is causing tests to fail.
2020-03-09 17:17:36 +00:00
Dimos Raptis
dcf659e643
[CORDA-3628] - Implement sendAll API (#5990)
* [CORDA-3628] - Implement sendAll API

* detekt

* Some minor refactorings and docs

* Eliminate warnings

* Address Rick's comments

* Switch sendAll to use a set
2020-03-09 17:09:13 +00:00
Joseph Zuniga-Daly
b02768aa6e CORDA-3394: Move unit test to correct location 2020-03-09 15:08:40 +00:00
Joseph Zuniga-Daly
217926092d
ENT-4334 Configuration option to exclude packages from Quasar instrumentation (#6005)
* Add quasarExcludePackages configuration option

* Change quasarExcludePackages to quasar.excludePackages

* Add doc comment for QuasarConfiguration

* Use forEach method

* Fix rule violation (TooGenericExceptionThrown)

* Mention new configuration option in changelog

* Remove outer section from new quasarExcludePackages configuration option
2020-03-09 11:40:51 +00:00
Joseph Zuniga-Daly
006dde8591 Merge remote-tracking branch 'origin/release/os/4.3' into jzd/os4.3-to-os4.4-merge-2020-03-06 2020-03-06 11:33:26 +00:00
Chris Rankin
493a92ca39 Merge commit '5b50ef49bcca2d1be8bb0604e1be167e3bb5da24' into chrisr3-44-merge 2020-03-06 09:55:00 +00:00
Matthew Nesbit
c65a5cb089
Merge pull request #6030 from corda/chrisr3-44-merge
Merge from OS 4.4 up to bc410b4.
2020-03-06 09:50:39 +00:00
Adel El-Beik
5b50ef49bc
CORDA-3651: addManifest now uses separate files for reading and writing. (#6026)
* CORDA-3651: addManifest now uses separate files for reading and writing.

* CORDA-3651: The jar scanning loader now closes itsself.

Co-authored-by: Adel El-Beik <adelel-beik@19LDN-MAC108.local>
2020-03-06 09:14:36 +00:00
Chris Rankin
2911548a2e Restore Node's missing dependency on jansi. 2020-03-04 20:56:17 +00:00
Matthew Nesbit
9a406839fa
ENT-4494 split nodeapi tests (#6024)
* Split out node-api tests that require test-utils/node-driver

* Add node-api test artefacts to publication list.

* Make test-common a transient dependency - downstream tests assume that it's available.

* Switch dependencies to java-library

* Fix magic package name for cordapp scanning in test
2020-03-04 17:59:15 +00:00
Chris Rankin
ffa2caed32 Merge branch 'release/os/4.4' into chrisr3-44-merge 2020-03-04 16:27:44 +00:00
Andrei Palade
ba68970549
Updated error message to provide consistent logging (#6015) 2020-03-04 11:43:05 +00:00
Chris Rankin
e006b871c8
CORDA-3644: Scan the CorDapp classloader directly for SerializationWhitelist. (#6014)
* CORDA-3644: Scan the CorDapp classloader directly for SerializationWhitelist.

* CORDA-3644: Filter CorDapps from out-of-process node classpaths by their manifest attributes. Also exclude directories and blatant test artifacts.

* Fix IRS Demo - its "tests" artifact had a non-standard classifier of "test".
2020-03-04 10:09:40 +00:00
Joseph Zuniga-Daly
e47e9d546d
Fix the operator used by the notEqual predicate (#6022)
* Fix the operator used by the notEqual predicate

* Add timeouts to tests

* Add change to changelog
2020-03-03 16:06:46 +00:00
Christian Sailer
e38cd9ec63
ENT-5043 separate out test utils (#5998)
* Remove unused dependencies from test-common

* Explicit imports and formatting

* Add core-test-utils project

* Add dependency

* Move Kryo serialization context to node-api (not serialization as we do not want to pull kryo into the serialization lib)

* Move AMQP server serialization scheme to node api

* Move serialization tests to node-api

* Move internal test helpers without further dependencies.

* Move out some types from RPCClientProxyHandler to node-api in preparation for moving the AMQP scheme

* Move client AMQP context to node-api so we can move the test serialization rule out.

* Move InternalSerializationTestHelpers to core-test-utils

* Moved testing.core to core-test-utils

* Make detekt happy

* Add api-scanner to core-test-utils

* Remove inlined package names introduced by IntelliJ refactoring

* Update api-current.txt to account for reordering.

* Add core-test-utils to list of published artifacts.

* Add missing import

* Location of things in api text has moved again (publish name of artefact?)

* Revert all additions to the API, leaving just the reordering

* Code review: fix up core-test-utils build.gradle and introduce kryo version constant.

* Remove OpenSsl flag from ssl config stub (can't be used from node-api)

* Suppress detekt warning

* Move core test util tests to the right module

* Expose kotlin test as a transient dependency - projects have come to rely on that.

* Fix typo in package name
2020-03-03 15:55:17 +00:00
Viktor Kolomeyko
275e801094
ENT-4955: Hide usage of FlowHospital inside StateMachineManager (#5997) 2020-02-25 14:10:59 +00:00
Adel El-Beik
fe625d0f37
CORDA-3584: Now cope with 2 contract jars with same hash but different name (#5952)
* CORDA-3484: Now cope with 2 contract jars with same hash but different name, we just select one and use that.

* ENT-3584: Contract jars are now generated on the fly.

* CORDA-3584: Reverted changes to CordappProviderImpl. Exception is raised if node started with multiple jars with same hash.

* ENT-3584: Fixing test failure.

* CORDA-3584: Switch to test extension method instead of reflection to access internal member.

* ENT-3584: Address review comment. Dont fully qualify exception.

* CORDA-3584: Address review comment and converted lazy to a resettable one.

* CORDA-3584: Removed unused logger.

* CORDA-3584: Fixed visibility.

* CORDA-3584: Removed synchronized

* CORDA-3584: Removed CordappResolver

* CORDA-3584: Reverted change in gradle file and fixed test.

* CORDA-3584: Removed V3 from test description as it wasn't actually V3 specific.

* CORDA-3584: Address review comment. Let classes be garbage collected.
2020-02-25 11:18:01 +00:00
Kyriakos Tharrouniatis
2c9c2985c0
CORDA-3381: Errors in vault updates publisher are unsubscribing stopping observers from working (#5912)
* Throw SQLException or PersistenceException plain, that may come out of an unsafe subscriber

* Add explanatory comment about why we changed Observer.tee to use unsafe subscribe

* Introducing not unsubscribing version of Rx.Subscriber

* Wrap PublishSubjects with FlowSafeSubjects in all tests that test Observer.tee

* Minor code formatting

* Make rawUpdates Rx.Observers not unsubscribe when accessed from CordaServices - Do not allow rawUpdates subscribing from flows

* Warning fix: Add else block to when statement

* Revert "Wrap PublishSubjects with FlowSafeSubjects in all tests that test Observer.tee"

This reverts commit e419af86

* Correcting log message

* Improve log message

* Add fiber's id to log message and exception message

* Added test, asserting FlowSafeSubscriber is alive and re-accessed upon flow retry

* Logging flow name instead of flow id at VaultService.rawUpdates subscribing error

* Add kdoc to OnNextFailedException

* Minor text correction

* Update kdocs of FlowSafeSubject/ PreventSubscriptionsSubject

* Moved FlowSafeSubject under package node.internal as it is only used by NodeVaultService

* Add comment and update kdoc explaining how to subscribe with SafeSubscriber to FlowSafeSubject

* Change PreventSubscriptionsSubject#errorAction to be more specific; to return an Exception

* Minor text update

* Update messy comment

* Replace assertThat with assertEquals

* Splitting heartBeat to heartBeat1 and hearBeat2 for more clear asserting

* Correcting comment

* Update messy comment

* Splitting heartBeat into heartBeatOnNext and heartBeatOnError

* Update test name

* Add explanatory comment to test

* Update test name

* Update test and add test comment

* Moving NotarisedTxs from SendStateFlow to VaultObserverExceptionTest inside NodeHandle.getNotarisedTransactionIds

* Moving SubscribingRawUpdatesFlow from ErrorHandling to VaultObserverExceptionTest

* Update kdoc of FlowSafeSubscriber and FlowSafeSubscriber.onNext

* Make kdoc more clear

* Throw exception upon accessing VaultService.rawUpdates from within a flow

* Changing exception thrown when accessing VaultService.rawUpdates from within a flow to a CordaRuntimeException

* Minor kdoc update

* Update test comment

* Update kdoc of FlowSafeSubscriber

* Introducing Observable.flowSafeSubscribe public API method to subscribe with -non unsubscribing- Rx.Subscribers to Observables. It also replaced FlowSafeSubject

* Move CustomSafeSubscriber outside test methods

* Minor text update

* Add timeout to tests

* Update kdoc of flowSafeSubscribe

* Update kdoc of flowSafeSubscribe

* Update kdoc of flowSafeSubscribe

* Move FlowSafeSubscriber and flowSafeSubscribe under their own package

* Fix detekt issue

* Update Detekt baseline

* Revert "Update Detekt baseline"

This reverts commit 793a8ed9

* Fix Detekt issue

* Moved strictMode flag from flowSafeSubscribe to OnFlowSafeSubscribe

Moved OnFlowSafeSubscribe into internal package

Integration tested flowSafeLooseSubscribe

* Suppress Rx Deprecation

* Rename flowSafeSubscribe to flowSafeObservable

* Renaming flowSafeObservable to continueOnError and FlowSafeSubscriber to ResilientSubscriber
2020-02-21 10:02:34 +00:00
Chris Rankin
4c6f15c40c Merge commit '88008b71f03a539749e32e5e76b2dc5ea98f897f' into chrisr3-44-merge 2020-02-19 14:12:10 +00:00
Chris Rankin
afcab51fe3 Update "in-process" test on basis it can be made to work. 2020-02-19 10:42:44 +00:00
Chris Rankin
d89ce6608a Create a unit test for serialization whitelists via driver. 2020-02-19 10:42:34 +00:00
Chris Rankin
73e6a40158 Merge commit '21325dfa4fb8151e77f368e9356c16ae89c70aa3' into chrisr4-44-merge 2020-02-15 14:45:50 +00:00
James Higgs
3547b629c3
[NOTICK] Add a detekt rule to catch tests with no timeout (#5959)
* [NOTICK] Add a custom detekt rule for tests with no timeout, and fix remaining missing timeouts

* [NOTICK] Add a test for custom detekt rules and tidying

* add timeout annotation to new test

Co-authored-by: Stefano Franz <roastario@gmail.com>
2020-02-14 17:31:02 +00:00
James Higgs
da192bcf0d
[EG-140] Allow system property paths with multiple keys to be specified in node.conf (#5963)
* [EG-140] Allow system property paths with multiple keys to be specified in node.conf

* [EG-140] Split property paths to remove quotes

* [EG-140] Quote system properties in docs

* [EG-140] Rename path to key
2020-02-14 11:18:21 +00:00
Schife
a765510cf4 Merge branch 'release/os/4.4' of https://github.com/corda/corda into TM-197-4.5Merge 2020-02-12 14:33:32 +00:00
Razvan Codreanu
c6ab07d92c
TM-197 Fix failing test caused by regression and ignoring flaky DistributedServiceTests (#5934)
* TM-197 Setting bouncy castle provider in order for the test to pass

* TM-197 setting timeout for all builds at 3 hours

* TM-197 ignoring unstable tests
2020-02-12 14:26:06 +00:00
Razvan Codreanu
42eca48a02
TM-197 Build stability changes (#5947)
* TM-197 switching 4.3 to use local k8s instances and also make the maximum duration of builds 3 hours, fix 1 test and ignore 2 flaky ones

* update to use local-k8s version of the plugin

Co-authored-by: Stefano Franz <roastario@gmail.com>
2020-02-12 13:58:30 +00:00
Denis Rekalov
1705299bde ENT-4918: Support symbolic link for certificates directory during node registration 2020-02-11 13:14:34 +00:00
Stefano Franz
4a54374f86
Merge pull request #5940 from corda/release/os/4.4
Merge OS 4.4 -> OS 4.5
2020-02-11 11:55:53 +00:00
Stefano Franz
b23af5f0d2
add timeouts to all tests (#5875) 2020-02-11 10:14:05 +00:00
Viktor Kolomeyko
90df56c173
CORDA-3565: Port ServiceStateSupport from ENT to OS (#5916)
* CORDA-3565: `ServiceStateSupport` and supporting classes

* CORDA-3565:Plug `ServiceLifecycleSupport` into `MessagingService`

* CORDA-3565: Detekt baseline update

* CORDA-3565: React to MessagingServer going up and addition logging for up/down

Co-authored-by: Matthew Nesbit <matthew.nesbit@r3.com>
2020-02-03 09:47:12 +00:00
Kyriakos Tharrouniatis
9ca1dd59da
CORDA-3381 Make internalUtilsKt Observer tee not wrap with SafeSubscriber (#5913)
* Make tee not wrap PublishSubjects in SafeSubscribers, otherwise a non Rx exception from an unsafe observer shuts down all other observers under the same PublishSubject

* Throw SQLException or PersistenceException plain, that may come out of an unsafe subscriber

* Revert "Throw SQLException or PersistenceException plain, that may come out of an unsafe subscriber"

This reverts commit c7b8af3fa6.

* Update Detekt baseline
2020-01-31 12:32:59 +00:00
Florian Friemel
cb6ed6042c
Move jimfs dependency from node to test-utils. (#5911) 2020-01-31 10:55:05 +00:00
jmacmahonr3
9511329e3d
[EG-75] Not handled exception when NetworkMap not accessible (#5903)
* Just passing in the exception was printing the entire stacktrace to the log, now we just pring the exception and message

* Updating exception message
2020-01-30 17:26:22 +00:00
Christian Sailer
cac96d81bf Remove unused dependency on IntelliJ forms 2020-01-30 10:48:00 +00:00
Oliver Knowles
286fffd67a
ENT-4368 - Improve error handling when required included config resource cannot be found. Add note in docs encouraging use of required syntax for includes. (#5907) 2020-01-30 10:10:24 +00:00
Ryan Fowler
d16e1126db
CORDA-3542: Use the config values for reconnecting retry interval and max reconnect attempts (#5869) 2020-01-29 13:08:54 +00:00
Nick Dunstone
a4cada4a2e ENT-4474 O/S Changes to support bulk backchain fetching (#5894)
* Ongoing dev multi backchain fetch

* Final dev pre logging message improvements

* Trace messages

* Trace messages

* Code tidy up

* Code review comments

* Code review comments

* Whitespace removed

* Code review changes

* Code reformatting
2020-01-28 09:04:18 +00:00
Ryan Fowler
e8d6701f87 CORDA-3558: Allow initial registration errors to propagate up so the node exits with a failure code (#5899) 2020-01-27 15:23:07 +00:00
Kyriakos Tharrouniatis
7f62046c2f CORDA-3329 Exceptions thrown in raw vault observers can cause critical issues (#5816)
Observers registered on NodeVaultService#rawUpdates, if they throw an exception when called from serviceHub#recordTransactions and if this exception is not handled by the flow hospital, then this leads to the transaction not being recorded in the local vault. This could get the ledger in an out of sync state.

In the specific case this happens within FinalityFlow#notariseAndRecord this leads to the transaction being notarized but not recorded in the local vault nor broadcasted in any counter party. The -failed to be recorded locally- transaction and its output states are not visible to any vault, and its input states not able to consumed by a new transaction, since they are recorded as consumed within the Notary. In this specific case we need not loose, by any means, the current transaction.

We will handle all cases by catching all exceptions thrown from serviceHub#recordTransactions, wrapping them with a HospitalizeFlowException and throwing it instead. The flow will get to the hospital for observation to be retried from previous checkpoint on next node restart.
2020-01-24 18:21:07 +00:00
Viktor Kolomeyko
a4d63d1329 CORDA-2942: Remove multiple reps from the test (#5893)
... as it is proven to be stable now.
2020-01-24 16:57:31 +00:00
Dan Newton
ecc3b5e330 CORDA-3554 Use flowId from ExternalMessageEvent when failing to init sessions (#5890)
Use `flowId` from `ExternalMessageEvent` when failing to init sessions instead of generating
a new random UUID. The a `flowId` is generated and stored inside the event after the
state machine work that done previously.
2020-01-24 16:34:29 +00:00
Dan Newton
4f1777adb4 NO-TICK Cap the default size of the external operation thread pool (#5878)
Cap the default size of the external operation thread pool to 10 or
the maximum number of available processors, whichever is smaller.

Set the minimum size of the thread pool to 1. Meaning that only a
single thread is used unless the node actually starts to use
`FlowExternalOperation` which consumes threads from this pool.
2020-01-24 12:11:53 +00:00
Viktor Kolomeyko
42a2ed98e2 CORDA-2942: Allow exception from CordaService creation to propagate (#5884)
* CORDA-2942: Allow exception from `CordaService` creation to propagate

It will ultimately be thrown from Node's `start()` method terminating the node start-up sequence.

* CORDA-2942: Be lenient when retrievign the name of the Notary

Some tests setup such that they do nto have Notary running.
2020-01-24 10:20:08 +00:00
Christian Sailer
1380779a9c
Remove unused schema options (#5885) 2020-01-24 08:49:59 +00:00
Viktor Kolomeyko
b72f71e7ac CORDA-3549: Re-write flaky CordaServiceLifecycleFatalTests (#5879)
* CORDA-3549: Improve stability of `CordaServiceLifecycleFatalTests`

* CORDA-3549: Bump-up reps count to ensure that test is definitely not flaky when executed by CI
(once proved the number of reps will be reduced)

* CORDA-3549: Making Detekt happier

* CORDA-2942: Ensure `NodeLifecycleEventsDistributor` cleans-up smoothly when node shuts down
2020-01-23 16:35:49 +00:00
James Higgs
9a1d46ddd1 EG-73 - Remove postgres dependency from the node and update docs (#5867) 2020-01-22 09:52:02 +00:00
Dan Newton
4bae045a58 ENT-4601 Public API to run external operations from a flow (#5833)
Deprecate FlowAsyncOperation and reimplement public versions FlowExternalOperation and FlowExternalAsyncOperation.

await added to FlowLogic to allow easy calling from both Java and Kotlin. There are two overrides of await (one for FlowExternalOperation and FlowExternalAsyncOperation).

Implementations of FlowExternalOperation return a result (written as blocking code) from their execute function. This operation will then be executed using a thread provided by the externalOperationExecutor.

Implementations of FlowExternalAsyncOperation return a future from their execute function. This operation must be executed on a newly spawned thread or one provided by a thread pool. It is up to developers to handle threading in this scenario.

The default thread pool (externalOperationExecutor) can be configured through the flowExternalOperationThreadPoolSize node config.

The current implementation leaves FlowAsyncOperation alone, meaning that any developers that have used it (even though it is internal) won't need to change their apps. If this was not concern I would delete it completely and replumb the state machine code. Instead, it has been marked with @DoNotImplement and executeAsync is annotated with @Deprecated
2020-01-22 09:27:17 +00:00
Viktor Kolomeyko
0978500a9a CORDA-2942: Node lifecycle events (#5846)
* CORDA-2942: Port minimal set of changes to make lifecycle events work

... and make codebase compile.

* CORDA-2942: Undo some changes which are not strictly speaking necessary

* CORDA-2942: Make `NodeServicesContext` leaner and delete `extensions-api` module

* CORDA-2942: Reduce even more number of files affected

* CORDA-2942: Integration test fix

* CORDA-2942: Make events `AfterStart` and `BeforeStop` generic w.r.t. `NodeServicesContext`

* CORDA-2942: `NodeLifecycleObserverService` and a set of integration tests.

Public API violations are expected as well as integration tests failing.

* CORDA-2942: Re-work to introduce `ServiceLifecycleObserver`

* CORDA-2942: Explicitly mention a type of exception that may be thrown for some events.

* CORDA-2942: Register `ServiceLifecycleObserver` through `AppServiceHub`

* CORDA-2942: Fix integration test + KDocs update

* CORDA-2942: Detekt and `api-current` update

* CORDA-2942: Improvement to `CordaServiceLifecycleFatalTests`

... or else it has side effects on other tests.

* CORDA-2942: Add an integration test for new API use in Java

Driver test is written in Kotlin, but services definition is written in Java.

Also KDocs improvements.

* CORDA-2942: Documentation and release notes update

* CORDA-2942: First set of changes following review by @mnesbit

* CORDA-2942: Second set of changes following review by @mnesbit

* CORDA-2942: Added multi-threaded test

* CORDA-2942: Fixes

* CORDA-2942: Undo changes to `api-current.txt`

* CORDA-2942: Bare mimimum change to `api-current.txt` for CI gate to pass.

* CORDA-2942: Address review feedback from @rick-r3

* CORDA-2942: Detekt update

* CORDA-2942: Delete `ServiceLifecycleObserverPriority` and replace it with `Int` after discussion with @mnesbit

* CORDA-2942: Introduce more `NodeLifecycleEvent` and switch services to listen for those events

* CORDA-2942: Few more changes after input from @rick-r3

* First stub on integration test
Unfinished - hang on issue and pay

* CORDA-2942: Switch to use out-of-process nodes for the inetgration test

Currently Alice and Notary stuck waiting to hear from each other.

* CORDA-2942: Extra log lines during event distribution

* CORDA-2942: Asynchronously distribute lifecycle events

* CORDA-2942: Await for complete P2P client start-up

Next step: Add vault query to integration test

* CORDA-2942: Asynchronously distribute lifecycle events

Next step: Improve integration test

* CORDA-2942: Fix test broken by recent changes and improve logging

* CORDA-2942: Improvement of the test to be able to monitor actions performed by @CordaService in the remote process

* CORDA-2942: Add node re-start step to the integration test

* CORDA-2942: Remove `CORDAPP_STOPPED` event for now

* CORDA-2942: s/CORDAPP_STARTED/STATE_MACHINE_STARTED/

* CORDA-2942: Inverse the meaning of `priority` as requested by @rick-r3

* CORDA-2942: Register `AppServiceHubImpl` for lifecycle events and put a warning when SMM is not ready.
2020-01-21 13:38:02 +00:00
Chris Rankin
a4d00b79d4 CORDA-3377: Upgrade to DJVM 1.0-RC06. (#5866) 2020-01-21 09:05:23 +00:00
Chris Rankin
49b4f66515 CORDA-3377: Upgrade to DJVM 1.0-RC05 (#5862) 2020-01-17 09:46:15 +00:00
Chris Rankin
01666ed068 CORDA-3523: DJVM custom serializers and whitelists (#5858) 2020-01-16 17:48:15 +00:00
Chris Rankin
f027fd5c77 ENT-4652: Add unit tests for parsing attachment fix-up rules. (#5859) 2020-01-16 17:19:58 +00:00
Chris Rankin
a7147c1ffd ENT-4652: Provide an "attachment fixup" mechanism to repair broken transactions. (#5825)
* Do not register cordapp custom serialisers when using attachment classloader.

* Record the URLs of CorDapp JARs that contain custom serialisers. Include these JARs as extra attachments if we discover that we're missing a custom serialiser during transaction verification.

* Check for disabled serializer when explicitly requesting a custom serializer.
Refactor test case to force use of a custom serializer.

* Tidy up basic custom serializer test.

* Also test that TransactionBuilder rejects missing custom serializers.

* Remove test whitelists, which should not be needed with custom serialisers.

* Add changelog entry. Also align TestCordappImpl.findRoots() with OS backports.

* Second approach based around CorDapps inside AttachmentStorage - report missing type descriptor or any non-composable types.

* Initial implementation of Corda-Fixup rules inside a CorDapp jar.

* Replace original "automatic attachment fixing" mechanism completely.

* First review comments: restore "missing class" logic to TransactionBuilder.

* Restore "missing class" mechanism as fallback for SignedTransaction too.
2020-01-14 15:18:51 +00:00
Ryan Fowler
4669a699c0 CORDA-3507: Use the config value for connectionRetryInterval (#5850)
* CORDA-3507: Use the config value for connectionRetryInterval rather than a hardcoded value

* CORDA-3507: Use the config value for connectionRetryInterval rather than a hardcoded value
2020-01-14 09:57:11 +00:00
Denis Rekalov
73a0782f5d CORDA-3452: Node: Configure the input of custom string in CSR (#5844)
* CORDA-3452: Node: Configure the input of custom string in CSR to be used by Identity Service

* CORDA-3452: Remove unused import

* CORDA-3452: Add test for networkServices configuration
2020-01-13 09:52:51 +00:00
James Higgs
d0543d7270 [CORDA-3436] Allow CorDapps access to node diagnostic information (#5834)
* [CORDA-3436] Allow CorDapps access to node diagnostic information

* [CORDA-3436] Fix API breakages

* [CORDA-3436] Improve documentation around diagnostics service

* [CORDA-3436] Remove CorDapps from the diagnostics information

* [CORDA-3436] Silence detekt warning
2020-01-09 11:18:32 +00:00
Ryan Fowler
e0eac8fa0d Corda 3513 rpc flow without permission (#5828)
* CORDA-3513: Don't try to reconnect for PermissionExceptions

* CORDA-3513: Don't try to reconnect for PermissionExceptions

* CORDA-3513: Add test for not reconnecting for PermissionExceptions

* CORDA-3513: Update exception message and test
2020-01-07 11:35:45 +00:00