Commit Graph

910 Commits

Author SHA1 Message Date
Ross Nicoll
88a205600d
INFRA-749 Shut down and delete test H2 databases (#6739) 2020-09-22 14:32:01 +01:00
Ryan Fowler
ed9f6f0ce1 Merge branch 'release/os/4.6' into rfowler-os-4.6-ent-4.6-20200922 2020-09-22 11:42:00 +01:00
Dan Newton
317b59ab9d
NOTICK Fix kill flow standalone rpc client test (#6721)
Test was flaky because the flow can finish below it can be killed,
therefore failing the test. Sleep for 1 minute instead to give plenty of
time for the test.
2020-09-18 15:15:44 +01:00
Denis Rekalov
5c6acb0909
CORDA-3968: Prepare keystore handling for certificate rotation [OS] (#6698)
* CORDA-3968: Move keystore logic to KeyStoreHandler.

* CORDA-3967: Stop generating node legal identity in runtime.

* CORDA-3969: Remove Node CA access at node normal runtime.

* CORDA-3968: Fix handling of absent keystore file and wrong password.
2020-09-17 10:15:46 +01:00
Christian Sailer
b746d5cb22
ENT-5759 Give the failing test CorDapp a different package name so it does not get added automatically to all tests. (#6712) 2020-09-15 16:52:08 +01:00
Christian Sailer
ea423215c2
ENT-5759 check for app schema migration (#6701)
* Fix behaviour for missing schema outside devMode plus test
2020-09-14 11:25:18 +01:00
Rick Parker
fd341c7e48
INFRA-670 Stop shutting down the database before some services that use it (#6690)
* INFRA-670 Stop shutting down the database before some services that use it.

* Test fixes
2020-09-08 08:34:19 +01:00
Yash Nabar
d1735b8c42
EG-3456: Make network parameters path configurable (#6531)
* Update docker image

* EG-3456: Make network parameters path configurable

* EG-3456: Remove filename from network parameters path property

* EG-3456: Add tests for network parameters and node configuration

* Revert docker image update

* EG-3456: Resolve network parameters path against base directory

* EG-3456: Add tests for network parameters path, increase test timeout limits

* EG-3456: Replace baseDirectory argument with networkParametersPath in NetworkParametersReader

* EG-3456: Refactor tests to reuse node configuration object

* EG-3456: Pass networkParametersPath as argument instead of baseDirectory
2020-09-07 09:49:37 +01:00
Ross Nicoll
0c16957622 Merge remote-tracking branch 'origin/release/os/4.6' into rni/notick/46-47-merge-2020-09-04 2020-09-04 14:15:47 +01:00
Jonathan Locke
e562c5828b
INFRA-604: Switch tests to use database snapshots (#6671)
Created a database snapshot of a clean Corda OS 4.5.1 database, which can now be used for testing by both the node driver and mock network.

The MockNetwork was changed from using an in memory database to using an on disk database, and makes use of the snapshot to speed up setup times.

The Node Driver was changed from defaulting to an in-memory database to defaulting to an on-disk database. Tests that do not specify the type of database to use will thus use an on-disk database. Tests that opt in for an in-memory database will continue to use an in-memory database as before.

The database snapshots are copied to the node directory inside the build folder, therefore, they should be cleaned up after a build.

Co-authored-by: Ross Nicoll <ross.nicoll@r3.com>
2020-09-03 14:43:55 +01:00
Adel El-Beik
9962c9085d
INFRA-424: Merge openj9 updates into 4.6 (#6683)
* INFRA-424 linux1 jenkinsfile

* INFRA-424 full run

* INFRA-424 bigger heap size

* Upgraded DJVM to handle BC - latest version of BC is a multirelease JAR.
When reading JKS keystore if a BC EdDSAPrivateKey is returned then swap for a net.i2p EdDSA private key.

* Temporary downgrade of BC

* Removed the BC EdDSA conversion

* INFRA-424 bigger heap size

* Upgrading Quasar to handle openJ9 different fields.

* INFRA-424: Handle lack of SUPPRESSED_SENTINEL in openj9.

* INFRA-424: If BCEdDSA public or private key is generated convert to net.i2p EdDSA form.

* INFRA-424 bigger heap size

* INFRA-424: On openJ9 only getting upto milli resolution.

* INFRA-424: Handle keystore returning a BCEdDSAPrivateKey.

* INFRA-424: Disable test on JDK11, as it requires the custom cordapp to generate JDK8 contract code, which we now check for.

* INFRA-424: Truncated time test to resolution of millis for openj9.

* INFRA-424 disabling log intensive tests until a fix is developed

* INFRA-424 one more test disabled

* INFRA-424: Disabled a couple of tests failing on openj9.

* INFRA-424: Disabling failing openj9 tests.

* INFRA-424: Disabling test failing on openj9.

* INFRA-424: Ignoring another flaky sleep test on openj9.

* INFRA-424 run integrationTests

* INFRA-424 set timeout to 4 hours

* INFRA-424: Cope with exception message from openj9.

* INFRA-424: Handle the coloured text characters openj9 adds.

* INFRA-424: Disabling test as it is generating JDK11 contract code under JDK11. Currently on JDK8 contract code allowed.

* INFRA-424: Commenting test out for openj9. Output of the processs thats read by the test is sometimes garbled.

* INFRA-424 switching to smoke tests

* INFRA-424 switching to slow integration tests

* INFRA-424 full run

* INFRA-424 moving jenkinsfile

* INFRA-424 removing references

* INFRA-424: Created common IS_OPENJ9 func for ignoring tests.

Co-authored-by: Schife <razvan.codreanu@r3.com>
2020-09-02 14:35:30 +01:00
Nikolett Nagy
6113cbbd39
verify with NETWORK_PARAMETERS role and test (#6628) 2020-09-02 09:48:01 +01:00
Ross Nicoll
b1e2d6a48c
NOTICK Reformat DriverDSLImpl.kt (#6685)
Reformat DriverDSLImpl to standard IntelliJ style
2020-09-01 19:40:39 +01:00
Ross Nicoll
d928c29019 Merge remote-tracking branch 'origin/release/os/4.5' into rni/notick/os-45-46-docs-db 2020-09-01 17:00:00 +01:00
Adel El-Beik
14e23430c0
CORDA-4003: Now support + in CorDapp filenames (#6673)
* CORDA-4003: Now cope with file: prefix not being in class path element.

* CORDA-4003: Switched to new URL type filter.

* CORDA-4003: Switched to a URL comparison. In the string comparison the scheme was removed in latest version of classgraph.

* CORDA-4003: Moved to latest version of classgraph that has support for + in filenames.

* CORDA-4003: Switched to accept version of the deprecated classgraph methods.
2020-09-01 10:30:49 +01:00
Viktor Kolomeyko
9fc896beb3
CORDA-3960: Port MultiRPCClient to OS (#6644)
* CORDA-3960: Port MultiRPCClient to OS

* CORDA-3960: Carefully restore serialisation environment in `MultiRpcClientTest` to eliminate side effects on other tests

* CORDA-3960: Move ThreadDumpUtils.kt to `core-utils`
2020-08-18 10:35:19 +01:00
Viktor Kolomeyko
b81eb1559d
CORDA-3959: Eliminate InternalCordaRPCOps (#6600)
* CORDA-3959: Make `ThreadContextAdjustingRpcOpsProxy` flexible for `RPCOps` it uses

* CORDA-3959: More changes towards supporting multiple `RPCOps` implementations

* CORDA-3959: Detekt baseline update

* CORDA-3959: Integration test compilation fix

* CORDA-3959: Introduce `CheckpointRPCOpsImpl` and wire it on

* CORDA-3959: Use multiple RPCOps interfaces in the shell commands

* CORDA-3959: Detekt baseline update

* CORDA-3959: Update RPCPermissionsTests

* CORDA-3959: Update RPCSecurityManagerTest

* CORDA-3959: Remove deprecated marker and rename the property

* CORDA-3959: Detekt baseline

* CORDA-3959: Introduce AttachmentTrustInfoRPCOpsImpl and wire it on

* CORDA-3959: Delete `InternalCordaRPCOps`

* CORDA-3959: Detekt baseline update

* CORDA-3959: Rename `CheckpointRPCOps` to `FlowManagerRPCOps`
2020-08-13 14:41:52 +01:00
Alexey Kadyrov
0b6b69bbda
CORDA-3657/5459 inspect waiting flows (#6540)
* CORDA-3657 Extract information from state machine

`FlowReadOperations` interface provides functions that extract
information about flows from the state machine manager.

`FlowOperator` implements this interface (along with another currenly
empty interface).

* CORDA-3657 Rename function and use set

* initial test is passing

* wip

* done tests

* additional tests to cover more FlowIORequest variations

* completed tests

* The quasar.jar should nat have been changed

* Fixed issues reported by detekt

* got rid of sync objects, instead relying on nodes being offline

* Added extra grouping test and minor simplification

* Hospital test must use online node which fails on otherside

* Added additional information required for the ENT

* Added tests to cover SEND FlowIORequests

* using node name constants from the core testing module

* Changed flow operator to the query pattern

* made query fields mutable to simply building query

* fixed detekt issue

* Fixed test which had dependency on the order int the result (failed for windows)

* Fixed recommendations in PR

* Moved WrappedFlowExternalOperation and  WrappedFlowExternalAsyncOperation to FlowExternalOperation.kt as per PR comment

* Moved extension to FlowAsyncOperation

* removed unnecessarily brackets

Co-authored-by: LankyDan <danknewton@hotmail.com>
2020-08-12 10:14:05 +01:00
Kyriakos Tharrouniatis
2afedeabb4 Merge branch 'release/os/4.6' into os_4.6-feature_pass_in_client_id_when_starting_a_flow 2020-08-06 13:21:45 +01:00
Christian Sailer
89759510ae Merge remote-tracking branch 'origin/release/os/4.6' into christians/ENT-5273-update-from-os-4.6-20200803 2020-08-03 16:28:20 +01:00
Tamas Veingartner
85be50779b
CORDA-3663 MockServices crashes when two of the provided packages to … (#6472)
* CORDA-3663 MockServices crashes when two of the provided packages to scan are deemed empty in 4.4 RC05

this happends when a given package is not found on the classpath. Now it is handled and an exception is thrown

* replace dummy package names in tests with valid ones

* allow empty package list for CustomCordapps and exclude those from the created jars

* detekt fix

* always true logic fix

* fix to check for empty packages instead of empty classes

* fix for classes and fixups

* logic refactor because of detekt stupidity

* PR related minor refactors
2020-08-03 09:19:48 +01:00
James Higgs
250ed8a21a
Merge pull request #6533 from corda/eg-177merge
EG-177   Network Parameter Hotloading
2020-07-31 09:45:23 +01:00
Christian Sailer
81d68abe7e Merge remote-tracking branch 'origin/release/os/4.6' into christians/ENT-5273-update-fb-from-os.4.6 2020-07-30 18:39:04 +01:00
Peter Nemeth
777be6c11a Merge branch 'feature/EG-177' of https://github.com/corda/corda into feature/EG-177 2020-07-30 16:20:01 +01:00
pnemeth
2a27f3ac37
EG-2055 Implement network parameters hotloading (#6517)
* Implement network parameters hotloading

* fixed failing unit test

* PR comments

* PR comments

* added integr tests+ renamed updater to hotloader

* moved exiting logic out of hotloader

* extra tests

* latest PR comments

* refactor

* address detekt/suppress if more significant refactoring needed

* extra tests

* addressed PR comments

* detekt

* formatting
2020-07-30 16:02:33 +01:00
Kyriakos Tharrouniatis
8a37108a58 Merge branch 'release/os/4.6' into os_4.6-feature_pass_in_client_id_when_starting_a_flow-merge 2020-07-30 10:34:03 +01:00
Yiftach Kaplan
0bedbd8c75
INFRA-530: Start notary node in process (#6521) 2020-07-29 15:47:45 +01:00
Dan Newton
c2fd8253ea
CORDA-3777 Reload after every checkpoint (#6494)
Enable reloading of a flow after every checkpoint is saved. This
includes reloading the checkpoint from the database and recreating the
fiber.

When a flow and its `StateMachineState` is created it checks the node's
config to see if the `reloadCheckpointAfterSuspend` is set to true. If it is
it initialises `StateMachineState.reloadCheckpointAfterSuspendCount`
with the value 0. Otherwise, it remains `null`.

This count represents how many times the flow has reloaded from its
checkpoint (not the same as retrying). It is incremented every time the
flow is reloaded.

When a flow suspends, it processes the suspend event like usual, but
it will now also check if `reloadCheckpointAfterSuspendCount` is not
`null` (that it is activated) and process a 
`ReloadFlowFromCheckpointAfterSuspend`event, if and only if 
`reloadCheckpointAfterSuspendCount` is greater than
`CheckpointState.numberOfSuspends`.

This means idempotent flows can reload from the start and not reload
again until reaching a new suspension point.

Flows that skip checkpoints can reload from a previously saved
checkpoint (or from the initial checkpoint) and will continue reloading
on reaching the next new suspension point (not the suspension point that
it skipped saving).

If the flow fails to deserialize the checkpoint from the database upon
reloading a `ReloadFlowFromCheckpointException` is throw. This causes
the flow to be kept for observation.
2020-07-28 16:27:51 +01:00
Dimitris Gounaris
1e6be340eb
CORDA-3844: bulk node infos request (#6411)
* CORDA-3844: Add new functions to network map client

* CORDA-3844: Apply new fetch logic to nm updater

* CORDA-3844: Fix base url and warnings

* CORDA-3844: Change response object and response validation

In order to make sure that the returned node infos are not maliciously modified, either a signed list response
or a signed reference object would need to be provided. As providing a signed list requires a lot of effort from NM and Signer services,
the signed network map is provided instead, allowing nodes to validate that the list provided conforms to the entries of the signed network map.

* CORDA-3844: Add clarifications and comments

* CORDA-3844: Add error handling for bulk request

* CORDA-3844: Enhance testing

* CORDA-3844: Fix detekt issues

* EG-3844: Apply pr suggestions
2020-07-28 15:02:53 +01:00
Christian Sailer
c4748e0588 Postpone notary configuration in driver DSL (matching original behaviour) to get around the service name check in registration. 2020-07-24 10:18:45 +01:00
Ryan Fowler
2ca10464b7
CORDA-3845: Update BC, log4j, slf4j (#6464)
* CORDA-3845: Update BC to 1.64

* CORDA-3845: Upgraded log4j to 2.13.3

* We can remove the use of Manifests from the logging package so that when _it_ logs it doesn't error on the fact the stream was already closed by the default Java logger.

* Some more tidy up

* Remove the logging package as a plugin

* latest BC version

* Remove old test

* fix up

* Fix some rebased changes to log file handling

* Fix some rebased changes to log file handling

* Update slf4j too

Co-authored-by: Adel El-Beik <adel.el-beik@r3.com>
2020-07-23 16:37:29 +01:00
Christian Sailer
db94f65d8a Merge remote-tracking branch 'origin/release/os/4.6' into christians/ENT-5273-update-from-os-4.6 2020-07-23 14:02:18 +01:00
Joseph Zuniga-Daly
c33720c73d
CORDA-3717: Apply custom serializers to checkpoints (#6392)
* CORDA-3717: Apply custom serializers to checkpoints

* Remove try/catch to fix TooGenericExceptionCaught detekt rule

* Rename exception

* Extract method

* Put calls to the userSerializer on their own lines to improve readability

* Remove unused constructors from exception

* Remove unused proxyType field

* Give field a descriptive name

* Explain why we are looking for two type parameters when we only use one

* Tidy up the fetching of types

* Use 0 seconds when forcing a flow checkpoint inside test

* Add test to check references are restored correctly

* Add CheckpointCustomSerializer interface

* Wire up the new CheckpointCustomSerializer interface

* Use kryo default for abstract classes

* Remove unused imports

* Remove need for external library in tests

* Make file match original to remove from diff

* Remove maySkipCheckpoint from calls to sleep

* Add newline to end of file

* Test custom serializers mapped to interfaces

* Test serializer configured with abstract class

* Move test into its own package

* Rename test

* Move flows and serializers into their own source file

* Move broken map into its own source file

* Delete comment now source file is simpler

* Rename class to have a shorter name

* Add tests that run the checkpoint serializer directly

* Check serialization of final classes

* Register as default unless the target class is final

* Test PublicKey serializer has not been overridden

* Add a broken serializer for EdDSAPublicKey to make test more robust

* Split serializer registration into default and non-default registrations. Run registrations at the right time to preserve Cordas own custom serializers.

* Check for duplicate custom checkpoint serializers

* Add doc comments

* Add doc comments to CustomSerializerCheckpointAdaptor

* Add test to check duplicate serializers are logged

* Do not log the duplicate serializer warning when the duplicate is the same class

* Update doc comment for CheckpointCustomSerializer

* Sort serializers by classname so we are not registering in an unknown or random order

* Add test to serialize a class that references itself

* Store custom serializer type in the Kryo stream so we can spot when a different serializer is being used to deserialize

* Testing has shown that registering custom serializers as default is more robust when adding new cordapps

* Remove new line character

* Remove unused imports

* Add interface net.corda.core.serialization.CheckpointCustomSerializer to api-current.txt

* Remove comment

* Update comment on exception

* Make CustomSerializerCheckpointAdaptor internal

* Revert "Add interface net.corda.core.serialization.CheckpointCustomSerializer to api-current.txt"

This reverts commit b835de79bd.

* Restore "Add interface net.corda.core.serialization.CheckpointCustomSerializer to api-current.txt""

This reverts commit 718873a4e9.

* Pass the class loader instead of the context

* Do less work in test setup

* Make the serialization context unique for CustomCheckpointSerializerTest so we get a new Kryo pool for the test

* Rebuild the Kryo pool for the given context when we change custom serializers

* Rebuild all Kryo pools on serializer change to keep serializer list consistent

* Move the custom serializer list into CheckpointSerializationContext to reduce scope from global to a serialization context

* Remove unused imports

* Make the new checkpointCustomSerializers property default to the empty list

* Delegate implementation using kotlin language feature
2020-07-22 17:31:59 +01:00
Christian Sailer
377c3f9d78 Merge remote-tracking branch 'origin/release/os/4.6' into christians/ENT-5273-update-from-os-4.6 2020-07-21 14:17:27 +01:00
Dimos Raptis
7261fa690f
CORDA-3506 - Implement session close operations (#6357) 2020-07-21 13:26:11 +01:00
Viktor Kolomeyko
234ac2c246 Merge remote-tracking branch 'origin/release/os/4.5' into vkolomeyko/os-4.5-os-4.6-merge 2020-07-21 10:29:07 +01:00
Viktor Kolomeyko
1c48418904
NOTICK: Remove relaxedThoroughness variable (#6483)
It is:
a. No longer works in Jenkins
b. May result spawning 100s of process that kill Windows build. Evidence: https://ci02.dev.r3.com/blue/organizations/jenkins/Corda-Enterprise%2FCorda-ENT-MS-Win-Compatibility%2Fenterprise/detail/PR-3553/25/pipeline/
2020-07-21 09:58:29 +01:00
Viktor Kolomeyko
3e9a8cdfcd Merge remote-tracking branch 'origin/release/os/4.5' into vkolomeyko/os-4.5-os-4.6-merge 2020-07-20 15:16:33 +01:00
Viktor Kolomeyko
4491abd7ba Merge remote-tracking branch 'origin/release/os/4.4' into vkolomeyko/os-4.4-os-4.5-merge 2020-07-20 11:51:08 +01:00
Kyriakos Tharrouniatis
22d92d5ef0
CORDA-3809 Expose client side unique RPC ID for flow starts (#6307)
Introducing a new flow start method (`startFlowDynamicWithClientId`) passing in a `clientId`.

Once `startFlowDynamicWithClientId` gets called, the `clientId` gets injected into `InvocationContext` and also pushed to the logging context.

If a new flow starts with this method, then a < `clientId` to flow > pair is kept on node side, even after the flow's lifetime. If `startFlowDynamicWithClientId` is called again with the same `clientId` then the node identifies that this `clientId` refers to an existing < `clientId` to flow > pair and returns back to the rpc client a `FlowStateMachineHandle` future, created out of that pair.

`FlowStateMachineHandle` interface was introduced as a thinner `FlowStateMachine`. All `FlowStateMachine` properties used by call sites are moved into this new interface along with `clientId` and then `FlowStateMachine` extends it.

Introducing an acknowledgement method (`removeClientId`). Calling this method removes the < `clientId` to flow > pair on the node side and frees resources.
2020-07-16 10:52:08 +01:00
Viktor Kolomeyko
fad8999ef4
CORDA-3862: Make SharedMemoryIncremental public (#6467) 2020-07-16 10:51:23 +01:00
Adel El-Beik
2fa6b5a208
CORDA-3769: Switched attachments class loader cache to use caffeine (#6326)
* CORDA-3769: Switched attachments class loader cache to use caffeine with original implementation used by determinstic core.

* CORDA-3769: Removed default ctor arguments.

* CORDA-3769: Switched mapping function to Function type to avoid synthetic method being generated.

* CORDA-3769: Now using a cache created from NamedCacheFactory for the attachments class loader cache.

* CORDA-3769: Making detekt happy.

* CORDA-3769: The finality tests now check for UntrustedAttachmentsException which will actually happen in reality.

* CORDA-3769: Refactored after review comments.

* CORDA-3769: Removed the AttachmentsClassLoaderSimpleCacheImpl as DJVM does not need it. Also updated due to review comments.

* CORDA-3769: Removed the generic parameters from AttachmentsClassLoader.

* CORDA-3769: Removed unused imports.

* CORDA-3769: Updates from review comments.

* CORDA-3769: Updated following review comments. MigrationServicesForResolution now uses cache factory. Ctor updated for AttachmentsClassLoaderSimpleCacheImpl.

* CORDA-3769: Reduced max class loader cache size

* CORDA-3769: Fixed the attachments class loader cache size to a fixed default

* CORDA-3769: Switched attachments class loader size to be reduced by fixed value.
2020-07-16 09:58:36 +01:00
Yiftach Kaplan
79b75ff1ec
INFRA-429: Wait for log file to indicate that the process is running (#6433)
Wait for log file to indicate that a driver node process is running, instead of trying to open a socket on the port the node is attempting to binding on. This means the driver is more responsive to the node being ready, as well as removing a race condition where the binding test could block the node from starting.

As sometimes nodes do not log this information, after 20 seconds we presume the node is up.
2020-07-13 11:42:30 +01:00
Christian Sailer
057a8d8ae9
NOTICK fix smoke tests and slow integration tests (#6422)
* Add schema migration to smoke tests
* Fix driver to work correctly for out-of-proc node with persistent database.
Co-authored-by: Ross Nicoll <ross.nicoll@r3.com>
2020-07-09 15:13:20 +01:00
LankyDan
fdae04fc28 Merge branch 'release/os/4.5' into dan/os-4.5-to-4.6-merge-2020-07-08 2020-07-08 10:44:47 +01:00
Chris Rankin
16eb9dfc08
NOTICK: Fix some Gradle technical debt. (#6430) 2020-07-07 09:08:58 +01:00
Ross Nicoll
6aa19723e6
INFRA-417 Improve driver DSL test stability (#6415)
* Move log messages that are not useful in typical usage from info to debug level to reduce log spam.
* Add node startup check before attempting to connect.
2020-07-03 20:42:29 +01:00
Jonathan Locke
f8d86c0617
INFRA-415: Add empty javadoc to node-driver (#6429)
Maven central will not allow the node-driver to be published without a Javadoc `.jar` file, even if it is empty. This adds an empty Javadoc `.jar` to the output.
2020-07-02 16:17:34 +01:00
Dan Newton
6bc2c79e23
NOTICK NodeBasedTest take in cordapps (#6424)
In enterprise, `AuthDBTests` picked up a schema from a unit test and
included it in the cordapp it builds. This schema does not have a
migration and therefore fails the integration tests.

`NodeBasedTest` now lets cordapps to be defined and passed in to avoid
this issue. It defaults to making a cordapp from the tests base
directory if none are provided.
2020-07-02 16:14:51 +01:00
Dan Newton
efd633c7b9 CORDA-3722 withEntityManager can rollback its session (#6187)
* CORDA-3722 withEntityManager can rollback its session

Improve the handling of database transactions when using
`withEntityManager` inside a flow.

Extra changes have been included to improve the safety and
correctness of Corda around handling database transactions.

This focuses on allowing flows to catch errors that occur inside an
entity manager and handle them accordingly.

Errors can be caught in two places:

- Inside `withEntityManager`
- Outside `withEntityManager`

Further changes have been included to ensure that transactions are
rolled back correctly.

Errors caught inside `withEntityManager` require the flow to manually
`flush` the current session (the entity manager's individual session).
By manually flushing the session, a `try-catch` block can be placed
around the `flush` call, allowing possible exceptions to be caught.

Once an error is thrown from a call to `flush`, it is no longer possible
to use the same entity manager to trigger any database operations. The
only possible option is to rollback the changes from that session.
The flow can continue executing updates within the same session but they
will never be committed. What happens in this situation should be handled
by the flow. Explicitly restricting the scenario requires a lot of effort
and code. Instead, we should rely on the developer to control complex
workflows.

To continue updating the database after an error like this occurs, a new
`withEntityManager` block should be used (after catching the previous
error).

Exceptions can be caught around `withEntityManager` blocks. This allows
errors to be handled in the same way as stated above, except the need to
manually `flush` the session is removed. `withEntityManager` will
automatically `flush` a session if it has not been marked for rollback
due to an earlier error.

A `try-catch` can then be placed around the whole of the
`withEntityManager` block, allowing the error to be caught while not
committing any changes to the underlying database transaction.

To make `withEntityManager` blocks work like mini database transactions,
save points have been utilised. A new savepoint is created when opening
a `withEntityManager` block (along with a new session). It is then used
as a reference point to rollback to if the session errors and needs to
roll back. The savepoint is then released (independently from
completing successfully or failing).

Using save points means, that either all the statements inside the
entity manager are executed, or none of them are.

- A new session is created every time an entity manager is requested,
but this does not replace the flow's main underlying database session.
- `CordaPersistence.transaction` can now determine whether it needs
to execute its extra error handling code. This is needed to allow errors
escape `withEntityManager` blocks while allowing some of our exception
handling around subscribers (in `NodeVaultService`) to continue to work.
2020-06-30 11:54:16 +01:00
Razvan Codreanu
e021022d0d
INFRA-357 Publish build tag (#6375) 2020-06-20 11:08:52 +01:00
Waldemar Zurowski
1a4efbac7f Merge branch 'release/os/4.5' into merge-45-to-46 2020-06-19 19:48:08 +01:00
Waldemar Zurowski
f1fcce4247 Merge branch 'release/os/4.4' into merge-44-to-45 2020-06-19 19:42:19 +01:00
Razvan Codreanu
ad21e381dc
INFRA-357 Adding JDK 11 Regression and Build tag jenkinsfile (#6374) 2020-06-19 14:10:31 +01:00
Adel El-Beik
debc962d1d Merge remote-tracking branch 'origin/release/os/4.3' into adel/merge-4.3-to-4.4-17June 2020-06-17 10:06:34 +01:00
Tamas Veingartner
26d4bfb89f
CORDA-3578 add corda prefix to conf file names as original issue was … (#6322)
* CORDA-3578 add corda prefix to conf file names as original issue was having non-corda reference.conf files on classpath causes DriverDSLImp failure

it is better to have this naming convention and avoid further conflicts of conf files.

* fixed weird duplicates

* revert renaming changes for web-reference.conf and loadtest-reference.conf
2020-06-16 09:15:51 +01:00
Christian Sailer
836dd559e8
ENT-5316 split schema migration
* ENT-5273 Split schema migration into separate core and app schema migration, with separate command line flags
2020-06-15 15:52:31 +01:00
Christian Sailer
98f62f60f1 Merge branch 'release/os/4.6' into christians/update-feat-20200502 2020-06-02 18:15:32 +01:00
williamvigorr3
0554c98d18
NOTICK Restrict to shell commands (#6303)
Remove shell command for pausing flows
2020-06-02 16:04:10 +01:00
Christian Sailer
0ed6307577 Merge remote-tracking branch 'origin/release/os/4.6' into christians/update-feat-20200502 2020-06-02 09:03:11 +01:00
Rick Parker
9f2bd1dcae
Merge pull request #6295 from corda/feature/checkpoint_table_improvements
CORDA-3432 Feature/checkpoint table improvements
2020-06-01 11:31:13 +01:00
Adel El-Beik
a33309a31b
CORDA-3755: Backport AttachmentURLStreamHandlerFactory memory leak (#6274)
* CORDA-3755: Switched attachments map to a WeakHashMap (#6214)

* CORDA-3772: Now specify source and target of 8 when compiling contract classes.

* CORDA-3651: addManifest now uses separate files for reading and writing. (#6026)

* CORDA-3651: addManifest now uses separate files for reading and writing.

* CORDA-3651: The jar scanning loader now closes itsself.

Co-authored-by: Adel El-Beik <adelel-beik@19LDN-MAC108.local>

Co-authored-by: Adel El-Beik <adelel-beik@19LDN-MAC108.local>
2020-05-27 11:35:15 +01:00
Denis Rekalov
5afdb63c94 CORDA-3818: Synchronize OS implementation of PublicKeyToOwningIdentityCache with CE 2020-05-26 16:10:14 +01:00
Christian Sailer
70f1ea0a9d
ENT-5258 db schema set-up only via command line flag (#6280)
Removing the ability to initialise schema from the node config, and add a new sub-command to initialise the schema (that does not do anything else and exits afterwards).
Also adding a command line flag that allow app schema to be maintained by hibernate for legacy cordapps, tests or rapid development.
Patching up mock net and driver test frameworks so they create the required schemas for tests to work, defaulting schema migration and hibernate schema management to true to match pre-existing behaviour.
Modified network bootstrapper to run an initial schema set-up so it can register nodes.
2020-05-22 16:27:10 +01:00
williamvigorr3
eb52de1b40
CORDA-3490 Add option to start node without starting checkpointed flows (#6136)
Added command-line option: `--pause-all-flows` to the Node to control this.
This mode causes all checkpoints to be set to status PAUSED when the
state machine starts up (in StartMode.Safe mode). 

Changed the state machine so that PAUSED checkpoints are loaded into
memory (the checkpoint is deserialised but the flow state is left serialised)
but not started.

Messages from peers are queued whilst the flow is paused and processed
once the flow is resumed.
2020-05-19 16:27:41 +01:00
James Higgs
06467d04b1
EG-1980 - Remove stale docs links from OS (#6251)
Remove hardcoded docs references, push docs link to gradle
2020-05-14 12:57:59 +01:00
Adel El-Beik
74080e7cb6
CORDA-3772: Now specify source and target of 8 when compiling contract classes. (#6249) 2020-05-13 15:11:25 +01:00
Adel El-Beik
1547efb093
CORDA-3755: Switched attachments map to a WeakHashMap (#6214)
* Bump OS release version 4.6

* CORDA-3755: Switched attachments map to a WeakHashMap

* CORDA-3755: Added explicit strong references to map key.

* CORDA-3755: Keeping detekt happy.

* CORDA-3755: Test a gc in verify.

* CORDA-3755: Making detekt happy.

* CORDA-3755: Suppress warnings for weak reference test.

* CORDA-3755: Fixing build failure with attachments.

* CORDA-3755: Rewrite based on Ricks input - now handles attachment already existing in map!

* CORDA-3755: Refactor WeakReference behaviour into AttachmentsHolderImpl and provide alternate version of this class for core-deterministic.

* CORDA-3755: Added more tests for WeakHashMap.

* CORDA-3755: Ignore the tests using System.gc keep for local testing only

* CORDA-3755: Adding comment to explain the ignored tests.

* Make AttachmentsHolderImpl package-private inside core-deterministic, just like it is inside core.

* CORDA-3755: Update assertions following review comments.

* CORDA-3755: Removing import

* CORDA-3755: Removed unused var.

* CORDA-3755: Reverting files that somehow got changed in rebase.

Co-authored-by: nargas-ritu <ritu.gupta@r3.com>
Co-authored-by: Chris Rankin <chris.rankin@r3.com>
2020-05-12 09:51:12 +01:00
LankyDan
4ccd0fd3df Merge branch 'release/os/4.6' into dan/4.6-into-checkpoint-feature-branch-2020-05-05
# Conflicts:
#	node/src/main/kotlin/net/corda/node/services/statemachine/Event.kt
#	node/src/main/kotlin/net/corda/node/services/statemachine/FlowStateMachineImpl.kt
#	node/src/main/kotlin/net/corda/node/services/statemachine/SingleThreadedStateMachineManager.kt
#	node/src/main/kotlin/net/corda/node/services/statemachine/transitions/TopLevelTransition.kt
2020-05-05 17:05:23 +01:00
Ryan Fowler
ebdd40049c CORDA-3662: Use an INNER JOIN for network map cache queries, (#6062)
- rename add or update function for clarity
- put removal of old nodes after retrieval of new ones to avoid gaps in the map
- plus add a test
2020-05-04 11:10:21 +01:00
Dan Newton
66f241efe8
CORDA-3291 Support for new flow sleep in mock networks (#6200)
The new flow sleep made `NotaryWhitelistTests` flaky which highlighted
an issue with the change to how a flow sleeps. Messages were being
pumped while the flow was sleeping which lead to inconsistent behaviour.
Messages are no longer pumped while a flow sleeps.
2020-04-30 10:18:45 +01:00
Matthew Nesbit
75d10fe99c
Merge pull request #6190 from corda/nikinagy/CORDA-3747-publish-sources-for-node-driver
CORDA-3747 - switch on the publishing of sources
2020-04-29 17:27:19 +01:00
nikinagy
274b958dad switch on the publishing of sources 2020-04-28 11:43:07 +01:00
Dan Newton
9a2ae8ae19
CORDA-3722 withEntityManager can rollback its session (#6187)
* CORDA-3722 withEntityManager can rollback its session

## Summary

Improve the handling of database transactions when using
`withEntityManager` inside a flow.

Extra changes have been included to improve the safety and
correctness of Corda around handling database transactions.

This focuses on allowing flows to catch errors that occur inside an
entity manager and handle them accordingly.

Errors can be caught in two places:

- Inside `withEntityManager`
- Outside `withEntityManager`

Further changes have been included to ensure that transactions are
rolled back correctly.

## Catching errors inside `withEntityManager`

Errors caught inside `withEntityManager` require the flow to manually
`flush` the current session (the entity manager's individual session).
By manually flushing the session, a `try-catch` block can be placed
around the `flush` call, allowing possible exceptions to be caught.

Once an error is thrown from a call to `flush`, it is no longer possible
to use the same entity manager to trigger any database operations. The
only possible option is to rollback the changes from that session.
The flow can continue executing updates within the same session but they
will never be committed. What happens in this situation should be handled
by the flow. Explicitly restricting the scenario requires a lot of effort
and code. Instead, we should rely on the developer to control complex
workflows.

To continue updating the database after an error like this occurs, a new
`withEntityManager` block should be used (after catching the previous
error).

## Catching errors outside `withEntityManager`

Exceptions can be caught around `withEntityManager` blocks. This allows
errors to be handled in the same way as stated above, except the need to
manually `flush` the session is removed. `withEntityManager` will
automatically `flush` a session if it has not been marked for rollback
due to an earlier error.

A `try-catch` can then be placed around the whole of the
`withEntityManager` block, allowing the error to be caught while not
committing any changes to the underlying database transaction.

## Savepoints / Transactionality

To make `withEntityManager` blocks work like mini database transactions,
save points have been utilised. A new savepoint is created when opening
a `withEntityManager` block (along with a new session). It is then used
as a reference point to rollback to if the session errors and needs to
roll back. The savepoint is then released (independently from
completing successfully or failing).

Using save points means, that either all the statements inside the
entity manager are executed, or none of them are.

## Some implementation details

- A new session is created every time an entity manager is requested,
but this does not replace the flow's main underlying database session.
- `CordaPersistence.transaction` can now determine whether it needs
to execute its extra error handling code. This is needed to allow errors
escape `withEntityManager` blocks while allowing some of our exception
handling around subscribers (in `NodeVaultService`) to continue to work.
2020-04-28 11:20:00 +01:00
Viktor Kolomeyko
257026f606
CORDA-3726: Remove memory leaks when many InProcess nodes started (#6169)
* CORDA-3762: Integration test exposing the problem reported

* CORDA-3726: Additional logging

* CORDA-3726: Prevent thread leaks

* CORDA-3726: New `journalBufferTimeout` parameter

* CORDA-3726: Override `journalBufferTimeout` parameter

* CORDA-3726: Making Detekt happier

* CORDA-3276: Account for extra thread user in MockNetwork

For real node this does not matter as `shutdown` can safely be called multiple times, which is not true for server thread provided by MockNetwork

* CORDA-3276: Do not make SMM shutdown "executor" as it belongs to AbstractNode

* CORDA-3276: Address input from @rick-r3

* CORDA-3276: Fix test after rebase
2020-04-23 08:53:51 +01:00
Ramzi El-Yafi
6a07284324
[EG-503] Spent state audit tool (#6107)
* [EG-503] Spent state audit tool

Fixes

* Refinements to notary query interfaces. Feature complete.

* EG-503: Introduce optional `notaryService` in `ServiceHubCoreInternal`

* Remove redundant logic following change to use extensions API

Co-authored-by: Viktor Kolomeyko <viktor.kolomeyko@r3.com>
2020-04-16 16:05:21 +01:00
Adel El-Beik
27ea570fbb
CORDA-3696: JDK 11 Testing branch (#6131)
* CORDA-3696: Temporary update to enable JDK11 build and test. Will eventually be switchable.

* CORDA-3696: Filter out the Nashorn warning.

* CORDA-3696: Add JDK11 classifier.

* CORDA-3696: Updated match string to cope with JDK11.

* CORDA-3696: Filtering out SPHINCS256_SHA256 where failing due to JDK11.

* CORDA-3696: Now remove SPHINCS256_SHA256 only if JDK11.

* CORDA-3696: Fix test failure - switch to regex matching.

* CORDA-3696: Hide the illegal access warnings.

* CORDA-3696: Check for Java11 when disabling Java11 warnings.

* CORDA-3696: Fix unneccessary non null check.

* CORDA-3696: Reverting build env to JDK8

* CORDA-3696: Revert hiding of illegal access warnings via Unsafe class.

* CORDA-3696: Remove internal access warnings and new JDK11 version checker.

* CORDA-3696: Updated build file for OS

* CORDA-3696: Removed typo

* CORDA-3696: Fixed space typo.

* CORDA-3696: Open modules to remove the illegal access warnings.

Co-authored-by: Adel El-Beik <adelel-beik@19LDN-MAC108.local>
2020-04-16 10:20:30 +01:00
Chris Rankin
a7358ffa1a
CORDA-3643: Initialise MockNetwork's serialisation environment with all custom cordapps. (#6115) 2020-04-01 17:42:29 +01:00
Chris Rankin
0771a22d5b
ENT-4967: Allow quasar_classifier to be either null or an empty string. (#6119) 2020-03-31 15:32:36 +01:00
Chris Rankin
f1ebaa761b
CORDA-3680: Add CorDapp custom serialisers to Driver's in-process nodes. (#6098)
* Run serialisation tests with both in-process and out-of-process nodes.

* Add custom serialisers and whitelists to Driver's AMQPServerSerializationScheme.
2020-03-27 10:01:30 +00:00
Dan Newton
678fb6eb94
Merge pull request #6108 from corda/dan/os-4.4-to-4.5-merge-2020-03-26
OS 4.4 to 4.5 merge 2020-03-26
2020-03-27 09:39:41 +00:00
Chris Rankin
cccbbe1c80
CORDA-3680: Add CorDapp custom serialisers to Driver's in-process nodes. (#6102)
* Run serialisation tests with both in-process and out-of-process nodes.

* Add custom serialisers and whitelists to Driver's AMQPServerSerializationScheme.
2020-03-27 09:16:31 +00:00
Dimos Raptis
963de40902
[NOTICK] - Enable check in detekt for unused imports (#6106)
* [NOTICK] - Enable check in detekt for unused imports

* Put back accidental removal of used import

* Some more accidental removals
2020-03-26 15:46:33 +00:00
LankyDan
d9c1907c88 Merge branch 'release/os/4.4' into dan/os-4.4-to-4.5-merge-2020-03-26 2020-03-26 14:26:41 +00:00
Joseph Zuniga-Daly
b766eff284 Merge remote-tracking branch 'origin/release/os/4.3' into jzd/merge-os4.3-into-os4.4-2020-03-26 2020-03-26 11:01:39 +00:00
Joseph Zuniga-Daly
2dbf90cafe
ENT-4857: Fix race condition in trackTransaction (#6096)
- Fix issue
- Emit warning if we are inside a DB transaction
- Include a path that does not emit warning
- Add unit tests
2020-03-25 11:53:06 +00:00
Kyriakos Tharrouniatis
da320514a5 Merge branch 'release/os/4.5' into os_4.5-feature_checkpoint_table_improvements-merge
# Conflicts:
#	node/src/main/kotlin/net/corda/node/services/statemachine/transitions/StartedFlowTransition.kt
2020-03-19 21:49:55 +00:00
Dimos Raptis
56067acd20
[CORDA-3628] - Remove overloads for sendAll (#6078) 2020-03-18 14:37:04 +00:00
Chris Rankin
29a36c6b4f
Prevent Quasar from instrumenting classes that belong to AttachmentsClassLoader. (#6077) 2020-03-18 13:59:10 +00:00
Ryan Fowler
349bd5a511
CORDA-3662: Use an INNER JOIN for network map cache queries, (#6062)
- rename add or update function for clarity
- put removal of old nodes after retrieval of new ones to avoid gaps in the map
- plus add a test
2020-03-17 17:02:08 +00:00
LankyDan
8941cf7b8f Merge branch 'feature/checkpoint_table_improvements' into dan/merge-4.5-into-feature-branch
# Conflicts:
#	node/src/test/kotlin/net/corda/node/services/statemachine/FlowFrameworkTests.kt
2020-03-13 14:29:18 +00:00
williamvigorr3
499b6cf17e
CORDA-3603 Save completed flow information (#6034)
When a flow is finished do not delete the checkpoint from the DB.

Instead, the FlowStatus is marked as Completed in the DB.

Updated numerous tests which relied on the flow being removed
when finished.
2020-03-12 11:45:30 +00:00
Christian Sailer
ab6bf315c4 ENT-4494 Harmonize CryptoService interface/base class between OS and ENT, move OS BC implementation to node project. 2020-03-10 12:36:30 +00:00
Chris Rankin
35f7cd349f CORDA-3644: Add Corda-Testing tag to core-test-utils manifest. 2020-03-09 21:26:53 +00:00
Joseph Zuniga-Daly
217926092d
ENT-4334 Configuration option to exclude packages from Quasar instrumentation (#6005)
* Add quasarExcludePackages configuration option

* Change quasarExcludePackages to quasar.excludePackages

* Add doc comment for QuasarConfiguration

* Use forEach method

* Fix rule violation (TooGenericExceptionThrown)

* Mention new configuration option in changelog

* Remove outer section from new quasarExcludePackages configuration option
2020-03-09 11:40:51 +00:00
Chris Rankin
493a92ca39 Merge commit '5b50ef49bcca2d1be8bb0604e1be167e3bb5da24' into chrisr3-44-merge 2020-03-06 09:55:00 +00:00
Matthew Nesbit
c65a5cb089
Merge pull request #6030 from corda/chrisr3-44-merge
Merge from OS 4.4 up to bc410b4.
2020-03-06 09:50:39 +00:00
Adel El-Beik
5b50ef49bc
CORDA-3651: addManifest now uses separate files for reading and writing. (#6026)
* CORDA-3651: addManifest now uses separate files for reading and writing.

* CORDA-3651: The jar scanning loader now closes itsself.

Co-authored-by: Adel El-Beik <adelel-beik@19LDN-MAC108.local>
2020-03-06 09:14:36 +00:00
Rick Parker
e5a8888232
CORDA-3644: Add Corda-Testing tag to test artifacts' MANIFEST.MF. (#6032) 2020-03-05 17:39:55 +00:00
Stefano Franz
c3a59f5293
EG-464 Corda returns incorrect exit code in case if node is started with unknown/missing option (#6010)
* EG-464

Corda returns incorrect exit code in case if node is started with unknown/missing option

* fixed empty else block

* We should not use the parent's exception handler as it will quit and we have our own one

* SampleCordaCliWrapper implemented and tests to verify error handling.

* addressing code review comments
2020-03-05 15:38:56 +00:00
Matthew Nesbit
9a406839fa
ENT-4494 split nodeapi tests (#6024)
* Split out node-api tests that require test-utils/node-driver

* Add node-api test artefacts to publication list.

* Make test-common a transient dependency - downstream tests assume that it's available.

* Switch dependencies to java-library

* Fix magic package name for cordapp scanning in test
2020-03-04 17:59:15 +00:00
Chris Rankin
ffa2caed32 Merge branch 'release/os/4.4' into chrisr3-44-merge 2020-03-04 16:27:44 +00:00
Chris Rankin
e006b871c8
CORDA-3644: Scan the CorDapp classloader directly for SerializationWhitelist. (#6014)
* CORDA-3644: Scan the CorDapp classloader directly for SerializationWhitelist.

* CORDA-3644: Filter CorDapps from out-of-process node classpaths by their manifest attributes. Also exclude directories and blatant test artifacts.

* Fix IRS Demo - its "tests" artifact had a non-standard classifier of "test".
2020-03-04 10:09:40 +00:00
Christian Sailer
e38cd9ec63
ENT-5043 separate out test utils (#5998)
* Remove unused dependencies from test-common

* Explicit imports and formatting

* Add core-test-utils project

* Add dependency

* Move Kryo serialization context to node-api (not serialization as we do not want to pull kryo into the serialization lib)

* Move AMQP server serialization scheme to node api

* Move serialization tests to node-api

* Move internal test helpers without further dependencies.

* Move out some types from RPCClientProxyHandler to node-api in preparation for moving the AMQP scheme

* Move client AMQP context to node-api so we can move the test serialization rule out.

* Move InternalSerializationTestHelpers to core-test-utils

* Moved testing.core to core-test-utils

* Make detekt happy

* Add api-scanner to core-test-utils

* Remove inlined package names introduced by IntelliJ refactoring

* Update api-current.txt to account for reordering.

* Add core-test-utils to list of published artifacts.

* Add missing import

* Location of things in api text has moved again (publish name of artefact?)

* Revert all additions to the API, leaving just the reordering

* Code review: fix up core-test-utils build.gradle and introduce kryo version constant.

* Remove OpenSsl flag from ssl config stub (can't be used from node-api)

* Suppress detekt warning

* Move core test util tests to the right module

* Expose kotlin test as a transient dependency - projects have come to rely on that.

* Fix typo in package name
2020-03-03 15:55:17 +00:00
Adel El-Beik
b86ae0d6a6
Web server fails to connect to node if node takes too long to startup (#6012)
* CORDA-3557: The web server attempts to reconnect to the client for a minute if initial attempt fails.

* CORDA-3557: Fixing detekt issues
2020-03-03 12:10:04 +00:00
Chris Rankin
d2568b9077
NOTICK: Fix potential infinite loop when a SecurityManager is installed. (#5993) 2020-02-24 09:36:16 +00:00
Kyriakos Tharrouniatis
2c9c2985c0
CORDA-3381: Errors in vault updates publisher are unsubscribing stopping observers from working (#5912)
* Throw SQLException or PersistenceException plain, that may come out of an unsafe subscriber

* Add explanatory comment about why we changed Observer.tee to use unsafe subscribe

* Introducing not unsubscribing version of Rx.Subscriber

* Wrap PublishSubjects with FlowSafeSubjects in all tests that test Observer.tee

* Minor code formatting

* Make rawUpdates Rx.Observers not unsubscribe when accessed from CordaServices - Do not allow rawUpdates subscribing from flows

* Warning fix: Add else block to when statement

* Revert "Wrap PublishSubjects with FlowSafeSubjects in all tests that test Observer.tee"

This reverts commit e419af86

* Correcting log message

* Improve log message

* Add fiber's id to log message and exception message

* Added test, asserting FlowSafeSubscriber is alive and re-accessed upon flow retry

* Logging flow name instead of flow id at VaultService.rawUpdates subscribing error

* Add kdoc to OnNextFailedException

* Minor text correction

* Update kdocs of FlowSafeSubject/ PreventSubscriptionsSubject

* Moved FlowSafeSubject under package node.internal as it is only used by NodeVaultService

* Add comment and update kdoc explaining how to subscribe with SafeSubscriber to FlowSafeSubject

* Change PreventSubscriptionsSubject#errorAction to be more specific; to return an Exception

* Minor text update

* Update messy comment

* Replace assertThat with assertEquals

* Splitting heartBeat to heartBeat1 and hearBeat2 for more clear asserting

* Correcting comment

* Update messy comment

* Splitting heartBeat into heartBeatOnNext and heartBeatOnError

* Update test name

* Add explanatory comment to test

* Update test name

* Update test and add test comment

* Moving NotarisedTxs from SendStateFlow to VaultObserverExceptionTest inside NodeHandle.getNotarisedTransactionIds

* Moving SubscribingRawUpdatesFlow from ErrorHandling to VaultObserverExceptionTest

* Update kdoc of FlowSafeSubscriber and FlowSafeSubscriber.onNext

* Make kdoc more clear

* Throw exception upon accessing VaultService.rawUpdates from within a flow

* Changing exception thrown when accessing VaultService.rawUpdates from within a flow to a CordaRuntimeException

* Minor kdoc update

* Update test comment

* Update kdoc of FlowSafeSubscriber

* Introducing Observable.flowSafeSubscribe public API method to subscribe with -non unsubscribing- Rx.Subscribers to Observables. It also replaced FlowSafeSubject

* Move CustomSafeSubscriber outside test methods

* Minor text update

* Add timeout to tests

* Update kdoc of flowSafeSubscribe

* Update kdoc of flowSafeSubscribe

* Update kdoc of flowSafeSubscribe

* Move FlowSafeSubscriber and flowSafeSubscribe under their own package

* Fix detekt issue

* Update Detekt baseline

* Revert "Update Detekt baseline"

This reverts commit 793a8ed9

* Fix Detekt issue

* Moved strictMode flag from flowSafeSubscribe to OnFlowSafeSubscribe

Moved OnFlowSafeSubscribe into internal package

Integration tested flowSafeLooseSubscribe

* Suppress Rx Deprecation

* Rename flowSafeSubscribe to flowSafeObservable

* Renaming flowSafeObservable to continueOnError and FlowSafeSubscriber to ResilientSubscriber
2020-02-21 10:02:34 +00:00
Chris Rankin
e9c2af661d NOTICK: Tidy up driver shutdown so that resources are always released on error. 2020-02-19 16:45:41 +00:00
Chris Rankin
4c6f15c40c Merge commit '88008b71f03a539749e32e5e76b2dc5ea98f897f' into chrisr3-44-merge 2020-02-19 14:12:10 +00:00
Chris Rankin
d89ce6608a Create a unit test for serialization whitelists via driver. 2020-02-19 10:42:34 +00:00
Chris Rankin
73e6a40158 Merge commit '21325dfa4fb8151e77f368e9356c16ae89c70aa3' into chrisr4-44-merge 2020-02-15 14:45:50 +00:00
James Higgs
3547b629c3
[NOTICK] Add a detekt rule to catch tests with no timeout (#5959)
* [NOTICK] Add a custom detekt rule for tests with no timeout, and fix remaining missing timeouts

* [NOTICK] Add a test for custom detekt rules and tidying

* add timeout annotation to new test

Co-authored-by: Stefano Franz <roastario@gmail.com>
2020-02-14 17:31:02 +00:00
Stefano Franz
4a54374f86
Merge pull request #5940 from corda/release/os/4.4
Merge OS 4.4 -> OS 4.5
2020-02-11 11:55:53 +00:00
Stefano Franz
b23af5f0d2
add timeouts to all tests (#5875) 2020-02-11 10:14:05 +00:00
Viktor Kolomeyko
90df56c173
CORDA-3565: Port ServiceStateSupport from ENT to OS (#5916)
* CORDA-3565: `ServiceStateSupport` and supporting classes

* CORDA-3565:Plug `ServiceLifecycleSupport` into `MessagingService`

* CORDA-3565: Detekt baseline update

* CORDA-3565: React to MessagingServer going up and addition logging for up/down

Co-authored-by: Matthew Nesbit <matthew.nesbit@r3.com>
2020-02-03 09:47:12 +00:00
Florian Friemel
cb6ed6042c
Move jimfs dependency from node to test-utils. (#5911) 2020-01-31 10:55:05 +00:00
Ryan Fowler
d0faf4d7cb NOTICK - allow resolveJar errors to be quieter (#5898) 2020-01-27 14:08:08 +00:00
Kyriakos Tharrouniatis
7f62046c2f CORDA-3329 Exceptions thrown in raw vault observers can cause critical issues (#5816)
Observers registered on NodeVaultService#rawUpdates, if they throw an exception when called from serviceHub#recordTransactions and if this exception is not handled by the flow hospital, then this leads to the transaction not being recorded in the local vault. This could get the ledger in an out of sync state.

In the specific case this happens within FinalityFlow#notariseAndRecord this leads to the transaction being notarized but not recorded in the local vault nor broadcasted in any counter party. The -failed to be recorded locally- transaction and its output states are not visible to any vault, and its input states not able to consumed by a new transaction, since they are recorded as consumed within the Notary. In this specific case we need not loose, by any means, the current transaction.

We will handle all cases by catching all exceptions thrown from serviceHub#recordTransactions, wrapping them with a HospitalizeFlowException and throwing it instead. The flow will get to the hospital for observation to be retried from previous checkpoint on next node restart.
2020-01-24 18:21:07 +00:00
Dan Newton
4bae045a58 ENT-4601 Public API to run external operations from a flow (#5833)
Deprecate FlowAsyncOperation and reimplement public versions FlowExternalOperation and FlowExternalAsyncOperation.

await added to FlowLogic to allow easy calling from both Java and Kotlin. There are two overrides of await (one for FlowExternalOperation and FlowExternalAsyncOperation).

Implementations of FlowExternalOperation return a result (written as blocking code) from their execute function. This operation will then be executed using a thread provided by the externalOperationExecutor.

Implementations of FlowExternalAsyncOperation return a future from their execute function. This operation must be executed on a newly spawned thread or one provided by a thread pool. It is up to developers to handle threading in this scenario.

The default thread pool (externalOperationExecutor) can be configured through the flowExternalOperationThreadPoolSize node config.

The current implementation leaves FlowAsyncOperation alone, meaning that any developers that have used it (even though it is internal) won't need to change their apps. If this was not concern I would delete it completely and replumb the state machine code. Instead, it has been marked with @DoNotImplement and executeAsync is annotated with @Deprecated
2020-01-22 09:27:17 +00:00
Viktor Kolomeyko
0978500a9a CORDA-2942: Node lifecycle events (#5846)
* CORDA-2942: Port minimal set of changes to make lifecycle events work

... and make codebase compile.

* CORDA-2942: Undo some changes which are not strictly speaking necessary

* CORDA-2942: Make `NodeServicesContext` leaner and delete `extensions-api` module

* CORDA-2942: Reduce even more number of files affected

* CORDA-2942: Integration test fix

* CORDA-2942: Make events `AfterStart` and `BeforeStop` generic w.r.t. `NodeServicesContext`

* CORDA-2942: `NodeLifecycleObserverService` and a set of integration tests.

Public API violations are expected as well as integration tests failing.

* CORDA-2942: Re-work to introduce `ServiceLifecycleObserver`

* CORDA-2942: Explicitly mention a type of exception that may be thrown for some events.

* CORDA-2942: Register `ServiceLifecycleObserver` through `AppServiceHub`

* CORDA-2942: Fix integration test + KDocs update

* CORDA-2942: Detekt and `api-current` update

* CORDA-2942: Improvement to `CordaServiceLifecycleFatalTests`

... or else it has side effects on other tests.

* CORDA-2942: Add an integration test for new API use in Java

Driver test is written in Kotlin, but services definition is written in Java.

Also KDocs improvements.

* CORDA-2942: Documentation and release notes update

* CORDA-2942: First set of changes following review by @mnesbit

* CORDA-2942: Second set of changes following review by @mnesbit

* CORDA-2942: Added multi-threaded test

* CORDA-2942: Fixes

* CORDA-2942: Undo changes to `api-current.txt`

* CORDA-2942: Bare mimimum change to `api-current.txt` for CI gate to pass.

* CORDA-2942: Address review feedback from @rick-r3

* CORDA-2942: Detekt update

* CORDA-2942: Delete `ServiceLifecycleObserverPriority` and replace it with `Int` after discussion with @mnesbit

* CORDA-2942: Introduce more `NodeLifecycleEvent` and switch services to listen for those events

* CORDA-2942: Few more changes after input from @rick-r3

* First stub on integration test
Unfinished - hang on issue and pay

* CORDA-2942: Switch to use out-of-process nodes for the inetgration test

Currently Alice and Notary stuck waiting to hear from each other.

* CORDA-2942: Extra log lines during event distribution

* CORDA-2942: Asynchronously distribute lifecycle events

* CORDA-2942: Await for complete P2P client start-up

Next step: Add vault query to integration test

* CORDA-2942: Asynchronously distribute lifecycle events

Next step: Improve integration test

* CORDA-2942: Fix test broken by recent changes and improve logging

* CORDA-2942: Improvement of the test to be able to monitor actions performed by @CordaService in the remote process

* CORDA-2942: Add node re-start step to the integration test

* CORDA-2942: Remove `CORDAPP_STOPPED` event for now

* CORDA-2942: s/CORDAPP_STARTED/STATE_MACHINE_STARTED/

* CORDA-2942: Inverse the meaning of `priority` as requested by @rick-r3

* CORDA-2942: Register `AppServiceHubImpl` for lifecycle events and put a warning when SMM is not ready.
2020-01-21 13:38:02 +00:00
Chris Rankin
a7147c1ffd ENT-4652: Provide an "attachment fixup" mechanism to repair broken transactions. (#5825)
* Do not register cordapp custom serialisers when using attachment classloader.

* Record the URLs of CorDapp JARs that contain custom serialisers. Include these JARs as extra attachments if we discover that we're missing a custom serialiser during transaction verification.

* Check for disabled serializer when explicitly requesting a custom serializer.
Refactor test case to force use of a custom serializer.

* Tidy up basic custom serializer test.

* Also test that TransactionBuilder rejects missing custom serializers.

* Remove test whitelists, which should not be needed with custom serialisers.

* Add changelog entry. Also align TestCordappImpl.findRoots() with OS backports.

* Second approach based around CorDapps inside AttachmentStorage - report missing type descriptor or any non-composable types.

* Initial implementation of Corda-Fixup rules inside a CorDapp jar.

* Replace original "automatic attachment fixing" mechanism completely.

* First review comments: restore "missing class" logic to TransactionBuilder.

* Restore "missing class" mechanism as fallback for SignedTransaction too.
2020-01-14 15:18:51 +00:00
James Higgs
d0543d7270 [CORDA-3436] Allow CorDapps access to node diagnostic information (#5834)
* [CORDA-3436] Allow CorDapps access to node diagnostic information

* [CORDA-3436] Fix API breakages

* [CORDA-3436] Improve documentation around diagnostics service

* [CORDA-3436] Remove CorDapps from the diagnostics information

* [CORDA-3436] Silence detekt warning
2020-01-09 11:18:32 +00:00
Denis Rekalov
bc96bea24a ENT-4628: Harmonize CryptoService and BCCryptoService between OS and ENT (#5822) 2019-12-18 16:54:39 +00:00
Viktor Kolomeyko
43205e1f1a CORDA-3471: Create CordaTransactionSupport and make it accessible through AppServiceHub (#5768)
* CORDA-3471: Create `CordaTransactionSupport` and use wherever possible instead of `CordaPersistence`

* CORDA-3471: Address comments by @mnesbit
- Relocate `CordaTransactionSupport` to `core`
- Create a lighter version of transaction - `VaultTransaction` that gives access to `session` object only.

* CORDA-3471: More changes after discussion with  @mnesbit
- Rename `VaultTransaction` into `SessionScope`.

* CORDA-3471: Revert changes to most of the files after conversation with @mnesbit and @rick-r3

* CORDA-3471: Introduce `CordaTransactionSupportImpl` and make it accessible via `AppServiceHub`.

* CORDA-3471: Minor change (comment).

* CORDA-3471: Address input from @mnesbit

* CORDA-3471: Address input from @rick-r3

* CORDA-3471: Make Detekt happier

* CORDA-3471: Add a new test that proves transactions can be started from client threads

As requested by @mnesbit

* CORDA-3471: Change log and documentation update.

As requested by @mnesbit
2019-12-04 17:18:40 +00:00
Chris Rankin
5a41ec9b82 CORDA-3464: Also scan attachment:// URLs for custom serializers. (#5769)
* CORDA-3464: Also scan attachment:// URLs for custom serializers.

* Only scan the given classloader - ignore this classloader's parents.

* Upgrade to ClassGraph 4.8.58 - for "robustness fixes".

* Register the attachment:// URL scheme using AttachmentsClassLoader.

* Add integration test for custom serializer in contract state.

* Rename Currancy -> Currantsy, just to make the point.
2019-12-03 15:50:21 +00:00
Stefan Iliev
926429647d CORDA-3307: Fix for underscore variables (#5682)
* Revert "Revert "CORDA-3307 - add support for environment variables in linux (#5523)" (#5643)"

This reverts commit 03ab258fc2.

* Env variables with underscore are now validated using schema validation and checking for unknown key errors.

* Resolving comments from PR review.

* Fix for deprecated import.

* Reworked logic according to PR review.

* Resolved bad string parsing problems where the json structure could be broken if some symbols were included in the key or value.
2019-11-19 17:51:52 +00:00
stefano
b11fd02f6e Merge remote-tracking branch 'origin/release/os/4.3' into my_merge_branch 2019-11-15 15:44:45 +00:00
Razvan Codreanu
deed4e9763 Enable test retry in distributed testing (#5702)
* TM-88 only write the test to the file if it passes otherwise retry

* TM-88 1 failing test for testing purposes

* TM-88 fail an integration test instead of unit test

* TM-88 include failed test for retry

* TM-88 new logic for retrying failed tests

* TM-88 refactored to use retry class when dealing with test failures + copy results even after pod failure

* TM-88 remaining pods as set to prevent duplicates, limit retries to prevent crashed build and a bit more logging

* TM-88 atomic int instead of int array

* TM-88 moving atomic int outside retry loops

* TM-88 removing second retry as it is not needed

* TM-88 small fix to final copy

* TM-88 while loop to prevent destruction of results on test retry

* TM-88 removing shutdown hook

* TM-88 reverting to previous implementation

* TM-88 copying xml files before retry

* TM-88 removing fail
2019-11-15 15:07:23 +00:00
Jonathan Locke
612fd83893 Fixed node driver issue 2019-11-14 11:25:56 +00:00
Jonathan Locke
c31cddaffc Fixed merge conflicts 2019-11-14 10:40:12 +00:00
Dan Newton
b15db200e3 NOTICK: Quick and dirty change to stop "Unable to start notaries." error message (#5686)
* Quick and dirty change to stop "Unable to start notaries." error message (#5686)

"Unable to start notaries. A required port might be bound already" is
returned whenever a startup error occurs while starting the notary nodes
in driver tests. This hides the real error.

This change prints the actual error to std_err and read from file
at a later point. This means the real error is not lost and will be
shown in failed builds.

* Suppress detekt warnings
2019-11-07 09:43:29 +00:00
stefano
6a3a9eded3 Merge remote-tracking branch 'origin/release/os/4.3' into my_merge_branch
# Conflicts:
#	core/src/main/kotlin/net/corda/core/internal/TransactionVerifierServiceInternal.kt
#	node/src/integration-test/kotlin/net/corda/node/services/rpc/RpcExceptionHandlingTest.kt
#	testing/node-driver/src/main/kotlin/net/corda/testing/node/internal/DriverDSLImpl.kt
2019-11-05 17:19:18 +00:00
Chris Rankin
02ed8e91b3 CORDA-2876: Ensure NodeConfig does not have devModeOptions outside of devMode. (#5676) 2019-11-05 16:59:40 +00:00
Chris Rankin
f226ddc4f2 CORDA-2876: Integrate the DJVM with the Corda Node. (#5633)
* * CORDA-2876: Migrate DJVM serialization modules into Corda.
* Pre-generate Corda classes for DJVM sandbox when node boots in production mode.
* Ensure that all DJVM test contract CorDapps are signed.
* Test examining attachments within DJVM sandbox.
* Test Contract.verify() using cryptographic verify function.
* Add test cases for more non-determinism in Contract.verify().
* Update node-driver to support testing nodes with DJVM support.
* Modify Node to allow alternative DJVM configurations for testing.
* Refactor DeterministicVerifierFactoryService for default use-case.
* Small whitespace and code-style refactors.
* Create and activate a DJVM execution profile for the Node.
* Revert making Verifier implement AutoCloseable.
* Allow the node to cache sandboxed Corda byte-code for reuse.
* Use updated Quasar agent that knows not to touch DJVM classloaders.
* Fix Quasar's package exclusions globs for DJVM.
* Deserialise LedgerTransaction into the sandbox for Contract.verify().
* Add the DJVM's serialisation modules to the Corda node.
* Update the node for the latest DJVM API, and preserve the ConstructorForDeserialization annotation on user contract classes.
* Add corda-dev to repositories while DJVM is SNAPSHOT.
* Migrate DJVM specialisation into AbstractNode's ServiceHubInternalImpl.
* Exclude sandbox.** and shaded djvm.** classes from Quasar agent.
* Add the corda-dev repository to :node for the deterministic runtime.
* Turn Verifier into an abstract base class that is specialised by BasicVerifier and DeterministicVerifier.
* Add the Corda deterministic libraries to the Node, and split the DJVM sandbox across two SandboxClassLoader instances.
* Add DJVM to contract verification path inside Corda Node.
* Minor lambda simplifications and removing unused import.
* CORDA-2871: Remove @CordaSerializable from LedgerTransaction.
* CORDA-2871: Add a callback to ServicesForResolution to allow the Node to modify a LedgerTransaction object.
* CORDA-2871: Refactor the contract verification code into a separate class,
 and allow LedgerTransaction to choose different Verifier objects.
* Update DJVM to use Corda 4.4-SNAPSHOT. (#95)
* CORDA-3330: Allow DJVM to preload / pregenerate classes from selected jars. (#92)
* Add support for SourceClassLoader.getResources() to DJVM.
* Allow a SandboxConfiguration to preload sandbox byte-code for all classes inside jars containing META-INF/DJVM-preload.
* CORDA-3309: Remove explicit try-catch in favour of UncaughtExceptionHandler. (#91)
* CORDA-3309: Install UncaughtExceptionHandler for DJVM tasks. (#88)
* Fix tests broken by Windows line endings. (#82)
* CORDA-3292: Reimplement ExecutionProfile as a data class. (#80)
* CORDA-2877: Refactor how we create child SandboxConfiguration objects. (#76)
* CORDA-2877: Load bytecode from a persistent cache to prevent repeated rewriting. (#75)
* Refactor byte-code cache to SandboxConfiguration instead of AnalysisConfiguration. We cannot "mix and match" byte-code generated by different sets of rules.
* CORDA-3137: Enhance annotation handling so that we can allow some annotations to be mapped into the sandbox without also needing to be stitched. (#72)
* CORDA-2871: Minor cosmetic fixes. (#69)
* CORDA-3218: Align DJVM with internal Corda Serialisation API. (#68)
* Ensure we get the latest SNAPSHOT of the serialisation code.
* CORDA-2871: Refactor SourceClassLoader to define source classes. (#66)
* Rewrite SourceClassLoader to support parent/child relationships.
* Revert catching TypNotPresebtException - it was a symptom of a bigger problem.
* Remove AutoCloseable from AnalysisConfiguration and SourceClassLoader.
* SourceClassLoader.getResource() must delegate to its parent first.
* CORDA-2871: Ensure ClassLoader.loadClass() throws ClassNotFoundException for all cases where the class cannot be found. (#64)
* CORDA-2871: Modify sandbox tasks to implement both java.Function and sandbox.Function (#62)
* Make TaskExecutors implement BiFunction to make them composable.
* Create ImportTask to wrap a java.Function inside a sandbox.Function.
* Add createExecutor() and createRawExecutor() APIs to SandboxClassLoader.
* Update serialization to use SandboxClassLoader.toSandboxClass().
* Remove a layer of lambdas from the serialisation code.
* Update SandboxExecutor and SandboxRawExecutor.
* Rename Executor to TaskFactory.
* Rename dangling executor -> taskFactory.
* CORDA-2871: Sanity fixes! (#63)
* Improve message for SandboxClassLoadingException.
* Fix serialisation API for using sandboxed environment.
* CORDA-3174: Extend serialisation to include InputStream and OpaqueBytesSubSequence. (#60)
* Update DJVM Example project for serialisation.
* Add serializers for InputStream and OpaqueBytesSubSequence.
* Support ZIP Inflater and CRC32 inside the sandbox.
* Allow the DJVM to wrap java.io.InputStream as sandbox.java.io.InputStream.
* Configure tests also to preserve @DeprecatedConstructorForDeserialization.
* CORDA-3174: Implement Corda serialization modules. (#59)
* Create DJVM serialization modules.
* Create test cases for Array<T>, List<T> and List<Array<T>>.
* Refactor SandboxPrimiveSerializer for all primitive types.
* Implement SandboxCollectionSerializer to support Collection types.
* Implement SandboxMapSerializer to support Map types.
* Attempt to fix infinite loop when computing Collection and Map fingerprints.
* Apply special handling when deserialising sandbox.java.lang.Character.
* Remap Java primitive types to sandbox Java object types to deter evolution.
* Use Class.getPackage().getName() to determine sandbox package name.
* Implement SandboxEnumSerializer to support Enum types.
* Implement SandboxPublicKeySerializer to support Java security keys.
* Add serialization projects to the composite example project.
* Implement serializers for BigInteger, BigDecimal, Currency and StringBuffer.
* Test that deserialising does not instantiate the untrusted user classes.
* Implement serializers for java.time.* types.
* Add serialiser for BitSet - currently disabled until BitSet itself is supported.
* Add serialisers for EnumSet and Class.
* Include support for EnumMap in the SandboxMapSerializer.
* Ensure the DJVM Example project's tests preserve @CordaSerializable.
* Add support for UUID as a primitive type.
* Use common abortReadOnly() method for declaring serialization as unsupported.
* Streamline the API for deserialising into the sandbox.
* Add preliminary support for deserialising X.509 certificates.
* Implement serializer for java.util.Optional.
* Refactor configuration of the sandbox serialization scheme.
* Add tests for deserialising arrays of basic types.
* Include method annotations in annotation stitching. This ensures that `@ConstructorForDeserialization` is not dropped.
* Enable test for SandboxBitSetSerializer.
* Enable tests for X.509 serializers.
* Implement serializers for ProtonJ primitive types.
* Serialize java.util.Date as a primitive type.
* Add the bintray Gradle plugin to the serialisation modules.
* Do not publish serialisation modules - they will become part of Corda itself.

* CORDA-2876: Only apply DJVM sources to Node Driver when devMode=true.

* Resolve DeteKT warnings.

* Require Node's JVM to set -Dnet.corda.djvm=true in order to enable DJVM.

* Enable DJVM for DemoBench nodes.

* Disable Quasar instrumentation verification for DemoBench nodes.

* Upgrade to DJVM 1.0-RC01.

* Try to modify DriverParameters in a more "ABI friendly" way.

* Refactor and simplify sandbox deserialisation of primitive objects.

* Review fixes.

* Update EvolutionSerializerFactory to handle sandboxed primitive boxed types.
2019-11-05 13:44:18 +00:00
Dimos Raptis
485feb2d6c CORDA-3350: Increase size of constraints column (#5639)
* CORDA-3350: Increase size of constraints column (#5639)

* Detekt

* Update api file with new threshold

* Add check in transaction builder

* Revert "Add check in transaction builder"

This reverts commit ca3128f44c.

* Add check for max number of keys

* Update api file

* Address Tudor's comments

* Remove check for pre-5 and add test for EC keys

* fix typo and rename liquibase script

* updated docs with measurement numbers for composite keys

* Make detekt happy again
2019-11-05 11:00:26 +00:00
Jonathan Locke
c193aa46f0
CID-1154: reliable finality merge to OS (#5658)
CID-1154: reliable finality merge to OS (#5658)
2019-11-05 10:48:00 +00:00
Stefano Franz
e4e920eee9
multiprocess port allocator is no longer used, so we can remove the tests as they add a significant amount of time to run (2-3 min) (#5663) 2019-11-04 11:56:38 +00:00
LankyDan
3c0631a26a Fix VaultObserverExceptionTest due to differences between ENT and OS 2019-11-04 09:42:41 +00:00
Barry
91e6c9783f TM-51 Read and write test results to artifactory. (#5597)
* TM-51  Prep for reading and writing test results to artifactory.

* TM-51  Tests from target branch if no tests for current branch

* TM-51  Placeholder for test averaging over runs.

* TM-51  Replace slashes in branch names used as tags.

* TM-51  More placeholder work for the mean duration work.

* TM-51  Write out average tests results as as csv.

The csv file should grow and be updated on each run.  This includes whether or not we are running unit tests, integration tests and so on.

* TM-51  Comment out old junit test archiving, add more comments.

* TM-51  Zip task needs to depend on a csv creation task.

If there isn't a csv file present, then the zip task doesn't run due to 'NO-SOURCE'

* TM-51  Zip task should ignore empty dirs

* TM-51  Fix up loading of test results.

We were looking for the wrong artifact name.
Add a bit more logging.

* TM-51  Fix up possible problem with allocating by class distribution.

If we encounter a class we haven't seen before, there won't be any tests.
This means we should give it some weight.  '1' is far too small.

* TM-51  Test that we are definitely increment the run count.

Tracking down whether the zipped csv file should have incremented.

* TM-51  Better default value for missing test/class names.

Begin by using mean unit test duration, but we have the option to bump
that to the mean class unit tests duration.

* TM-51  More debug information around csv writing.

We should be incrementing the tests.

* TM-51  Reload the csv before updating it.

* TM-51  Reduce verbosity of logging.

* TM-51  Reinstate unit tests.  Remove logging verbosity.

* TM-51  Load tests from artifactory in memory and avoid interim file.

* TM-51  Better handling of zero duration tests.

Ensure we return zero times from junit artifacts which may either be zero or have no recorded time.  Before writing the tests duration csv file, store those with a known time, and then store those with zero using the average time.

* TM-51  Log whether we have recorded a test.

Tracking down the curious case where we seem to not be rerunning the
same set of tests on the second run.

* TM-51  Capture junit files as well.

Trying to track down whether some tests are intermittently run.

* TM-51  Change task dependencies to ensure ziptask is triggered.

* TM-51  Remove test assertion, and trigger build

* TM-51  Add corda/enterprise to artifactory tag name.

Moved properties to own file.

* TM-51  Remove unnecessary mean class-based duration.

* TM-51  Add more BucketingAllocator tests.

We need these to nail down its behaviour some more.

* TM-51  Further log information.

We don't seem to be finding the tests in the 'production' runs which is odd.

* TM-51  corda type double set?

* TM-51  do not set the project type in the properties.

SRP and all that.

* TM-51  better plan reporting

* TM-51  duration may be zero

Another runtime problem that doesn't show in tests.

* TM-51  better plan reporting

* fix missing space after image id

* fix merge issue in DistributedTesting

* TM-51  remove unused code when GET/PUT-ting to Artifactory.

* TM-51  put tasks in gradle group and tidy up zip task creation

* TM-51 Fix the junit XML path.

* TM-51 Fix the task graph

* TM-51 Less logging
2019-11-02 09:07:53 +00:00
LankyDan
3aaddb47ea Attempt to fix detekt issues in DriverDSLImpl again 2019-11-01 14:36:47 +00:00
LankyDan
dfb86f5d9c Attempt to fix detekt issues in DriverDSLImpl 2019-11-01 14:21:17 +00:00
LankyDan
4aa9add8c8 CORDA-2050 Upgrade Corda to Java 11 (compatibility mode) (#5356) Upgrade Corda to run with Java 11 (compatibility mode) - see https://github.com/corda/corda/pull/5356 - 3fafbe55
Reapply change that was lost during merge - Adjust resolution of byteman jar to use java 11 compatible mechanism. - a1077092

Manual cherry pick of these changes (a1077092 + 3fafbe55)
2019-11-01 11:50:16 +00:00
Viktor Kolomeyko
c53ea9dde5 Eliminate extensive printout when Byteman not found on the classpath. (#1277)
Byteman is absent for most of the integration tests and long stacktrace is seen in the log
presently that un-necessarily attracts attention and consumes logging space.

(cherry picked from commit 2b6e59e7bd)
2019-11-01 11:50:16 +00:00
LankyDan
bedfba8c3d ENT-1967: Illustration for Byteman library can be used in Node integration test. (#1204) - c396b80a
(Only took `DriverDSLImpl` changes)

Simplifying internal startNode with bytemanPort parameter - d88b02f7

Manual cherry pick of these changes (c396b80a + d88b02f7)
2019-11-01 11:50:16 +00:00
Christian Sailer
9f15457045 add else branch to avoid warning that fails the warning check 2019-11-01 11:48:07 +00:00
Christian Sailer
119f939ee1 Fix and or suppress detekt warnings 2019-11-01 11:48:07 +00:00
Dan Newton
268d129838 CORDA-3194 Replay start flow events when responding flow fails initial checkpoint commit (#2601)
* CORDA-3194 Replay start flow events when responding flow fails initial checkpoint commit

Logic has already been added to recover from initial checkpoint commit
failures on the initiating flow side but this did not suffice for
the same failure occurring on the responding flow's side.

The same idea has been added to resolve the responding flow's issue.

`ExternalMessageEvent` now has a `flowId` that is maintained on the
event. Messages can then be replayed to start/restart the flow, while
the event provides the flow id to each flow start.

Each `ExternalMessageEvent` implementation generates a random `flowId`
when constructed.

Events are stored in Artemis. This allows the solution to recover across
node restarts as the events will be pulled from artemis again when
restarting.

In the future `flowId`s will probably moved off of the events and
generated purely on the responding flow's node.

* CORDA-3194 Add test to verify that errors removing a responding flow are recoverable
2019-11-01 11:48:07 +00:00
Christian Sailer
1f71b071aa CORDA-3217 and CORDA-3195 Various bits arond SQL exceptions and flow hospital (#2605)
* Unwrap rx.OnErrorNotImplementedException so the hospital can handle the cause appropriately

* Add db failure cordapp

* Renamed folders to avoid ambiguity in gradle

* Add integration test for exception hospitalisation when thrown from an RX observable.

* Make the test slightly cleaner

* Fix the schema to actually match the requirements for my custom state. Thanks a bunch, H2.

* Switch test to use SqlException base class.

* Schedule error event if we detect that a commit or db flush has thrown (forcing the flow to error even if customer code then goes ahead to swallow the exception)

* Revert change to schedule extra error

* Add more tests for edge case with DB exceptions, changed CorDapp to suppor this an hook in the flow hospital

* Warning about unsubscribe
Check state transitioned from clean to error for hospital admission.

* Match the test to our actual expectations

* Revert "Revert change to schedule extra error"

This reverts commit 43d47937

* Prevent suppression of errors arising in `transaction()` and `jdbcConnection()`

* Test for SqlException caught trying to escape from recordTransaction and suppressed outside being intercepted.

* More tests for various error/catch combinations

* Clean up and comments

* Code reformat

* Fix test compilation
2019-11-01 11:48:07 +00:00
Stefan Iliev
03ab258fc2 Revert "CORDA-3307 - add support for environment variables in linux (#5523)" (#5643)
This reverts commit c882b221a5.
2019-10-29 17:55:58 +00:00
Stefano Franz
d693a9c1ce
TM-65 re-add local port availability check (#5618)
* re-add local port availability check

* attempt to fix issue with port allocator

* ensure the serversocket is closed
2019-10-21 16:17:17 +01:00