295 Commits

Author SHA1 Message Date
Viktor Kolomeyko
ad981c7f13
ENT-2569: Integration test to simulate FlowWorker restart. (#1463)
* ENT-2569: Integration test to simulate Rpc/FlowWorker restart.

Initial analysis shows that 15-25 MB cannot be accounted for after RpcWorker restart.
Please see below for stats sample.
I will need to run memory profiler to understand where the memory is going.

Sample 1:
```
Memory stats @Very beginning - Used memory: 66,167 KB, Total memory: 1,158,144 KB, Max memory: 7,414,784 KB
Memory stats @Between restarts - Used memory: 214,952 KB, Total memory: 3,228,160 KB, Max memory: 7,414,784 KB
Memory stats @Very end - Used memory: 231,285 KB, Total memory: 3,226,112 KB, Max memory: 7,414,784 KB
```

Sample 2:
```
Memory stats @Very beginning - Used memory: 65,649 KB, Total memory: 1,125,376 KB, Max memory: 7,414,784 KB
Memory stats @Between restarts - Used memory: 202,542 KB, Total memory: 3,390,976 KB, Max memory: 7,414,784 KB
Memory stats @Very end - Used memory: 217,969 KB, Total memory: 3,387,392 KB, Max memory: 7,414,784 KB
```

* ENT-2569: Repair FlowWorkerTest and make it use signed network parameters.

* ENT-2569: Take flow worker up and down multiple times.

* ENT-2569: Add memory consumption stats.

Sample output for 10 iterations (1 warm-up and 9 subsequent):
```
Memory stats @Very beginning - Used memory: 126,495 KB, Total memory: 1,599,488 KB, Max memory: 7,416,832 KB
Memory stats @After warm-up round - Used memory: 172,411 KB, Total memory: 2,096,128 KB, Max memory: 7,416,832 KB
Memory stats @Testing done - Used memory: 196,912 KB, Total memory: 2,213,376 KB, Max memory: 7,416,832 KB
```
So during 9 iterations/restart cycles we have lost ~25MB.

* ENT-2569: Re-structure the test in preparation for 2nd legal name.

* ENT-2569: Mote test re-structure.

* ENT-2569: Introduce BankB into the test scenario.
2018-10-12 13:51:11 +01:00
Andrius Dagys
504cefc6f6 CORDA-535: Move MySQL notary into a separate module 2018-10-10 16:07:16 +01:00
Andrius Dagys
b7d59785e7 Merge remote-tracking branch 'open/master' into andrius/merge-10-10 2018-10-10 10:45:21 +01:00
Andrius Dagys
9ebeac1ad8
CORDA-535: Extract notary implementations into CorDapps (#3978)
* Move Raft and BFT notaries into separate modules

* Move schemas

* Fix tests & demos

* Modified logic for creating notary services:

Added a new field 'className' to the notary configuration. The node now
loads the specified implementation via reflection. The default className
value points to the simple notary implementation for backwards compatibility.
Relevant schemas are loaded in a similar fashion.

For backwards compatibility purposes the default SimpleNotaryService will
remain built-in to node, but its cordapp will be generated on startup – so
the loading of notary services is streamlined.

* Move test namedcache factory to test utils
2018-10-10 10:04:22 +01:00
Anthony Keenan
862626f3a3 Fix another merge conflict 2018-10-09 21:22:24 +01:00
Chris Rankin
4cb31f4c8b Merge commit '39434dcbecdd2cd656e40622530e84d02443b8e2' into chrisr3-os-merge 2018-10-08 10:42:55 +01:00
Viktor Kolomeyko
8c23abbd7a
ENT-2500: Corda Firewall should log some packet statistics (#1426)
* ENT-2500: Refactoring to allow FirewallAuditService have inbound as well as outbound logging.

Change some visibility modifiers to `private`. Use in-place initialisation where possible.

* ENT-2500: Record accepted package coming into Float

* ENT-2500: Introduce optional BridgeAuditService in `nodeApi` module

* ENT-2500: Switch FirewallAuditService to use `ApplicationMessage` and bind outgoing message stats.

* ENT-2500: Introduce scheduled executor and audit service configuration.

* ENT-2500: Stats formatting.

* ENT-2500: Stats formatting unit test.

* ENT-2500: Minor changes to LoggingFirewallAuditService and its unit test.

* ENT-2500: Additional configuration parameter documentation update.

* ENT-2500: Supply optional parameter.

* ENT-2500: Address PR comments.

* ENT-2500: Make API more consistent by using `RoutingDirection`, re-jig `State` data structure, improve unit test.

* ENT-2500: Add breakdown by endpoint address.

* ENT-2500: Compilation fix after rebase in `master`.

* ENT-2500: Making `AuditServiceConfiguration` not optional and supplying default settings.
Also few minor changes.
2018-10-02 14:13:08 +01:00
Rick Parker
8bbc0d9f43
ENT-2431 Add caching metrics, consolidate tracing config and better default cache sizes (#1418) 2018-10-01 14:28:03 +01:00
rick.parker
d915c413c6 Enterprise only compilation fixes. 2018-09-24 11:16:15 +01:00
szymonsztuka
af0c45b287 Added required terminate and isWaitingForShutdown methods by CordaRPCOps. 2018-09-20 20:19:35 +01:00
szymonsztuka
8e9ce48404 Follow refactoring CHECKPOINT_CONTEXT from SerializationDefaults to CheckpointSerializationDefaults etc in Enterprise only code. 2018-09-20 20:17:37 +01:00
Viktor Kolomeyko
4c68b515f8
ENT-2489: Experiment with multi-identity RPC worker (#1377)
* ENT-2489: Trivial change to prep for multiple RpcWorkerServiceHubs (i.e. identities)

* ENT-2489: Allow passing `targetLegalIdentity` from Client RPC call.

* ENT-2489: Starting RpcWorker with multiple RpcWorkerServiceHubs (unfinished)

* ENT-2489: Starting RpcWorker with single (for now) RpcWorkerServiceHub

* ENT-2489: Tighten-up integration tests assertions

* ENT-2489: Introduce RPCOpsRouting

* ENT-2489: Output configs for reference

* ENT-2489: Extend test for RpcWorker to operate with multiple identities.

* ENT-2489: Remove un-necessary P2P address

* ENT-2489: New test for RpcWorker getting paid.

* ENT-2489: Make RpcWorkerMultiIdentityTest work

* ENT-2489: Use MAX_RPC_MESSAGE_SIZE when configuring RPC broker.

* ENT-2489: Add exception clause when client attempts to use the wrong identity.

* ENT-2489: Fixes post merge from `master`.

* ENT-2489: Fixes post merge from `master`.

* ENT-2489: Explicitly specify X500 name in test.

* ENT-2489: Use single flow worker and switch anonymity off when making payment.

* ENT-2489: Fix for RpcWorkerTest.

Add `NetworkMapUpdater` to `FlowWorkerServiceHub` or else no-one will ever send a signal on `networkMapCache.nodeReady` future.
Not having `networkMapCache.nodeReady` in completed state, will prevent SMM from operating properly.

* ENT-2489: Handle gracefully the fact that session might be already closed.

* ENT-2489: Fix incorrect merge from `master`.

* ENT-2489: Make `RPCOpsRouting` generic with regard to `RPCOps` following discussion with @mnesbit

* ENT-2489: Make `methodTable` uniform for all the legal names.

* ENT-2489: Make `ObservableContext` non-generic.

* ENT-2489: Tidy-up shutdown sequence post merge from `master`

* ENT-2489: Correct exception type thrown

* ENT-2489: Generics test compilation fix.
2018-09-19 11:04:37 +01:00
Viktor Kolomeyko
dbee84c01d
ENT-2489: Perform cleaner shutdown of Rpc/Flow workers (#1398)
* ENT-2489: Perform cleaner shutdown of P2P connections.

* ENT-2489: SessionFactory tidy-up

* ENT-2489: Tidy-up RpcFlowWorkerDriver shutdown sequence

* ENT-2489: Fix test compilation issue.

* ENT-2489: Introduce helper function following review from @mnesbit
2018-09-18 17:54:08 +01:00
Viktor Kolomeyko
18d4013d0c
ENT-2489: Introduce NetworkMapUpdater into FlowWorkerServiceHub (#1396)
Without this change RpcWorkerTest currently in `master` just hangs
as FlowWorker never deemed to be ready from NM point of view.
2018-09-18 15:16:20 +01:00
Chris Rankin
4602a21a35
Merge pull request #1391 from corda/chrisr3-os-merge
Merge from Open Source
2018-09-13 15:14:50 +01:00
Shams Asari
725441b39c
Minor changes to bring in-sync with OS (#1390) 2018-09-13 12:53:06 +01:00
Chris Rankin
96597ba084 Merge commit '1c7dfd4b7b310e86b8aa973b8350487ba63c08b1' into chrisr3-os-merge 2018-09-13 11:06:52 +01:00
Chris Rankin
1c7dfd4b7b
CORDA-1964: Unify versioning of the shadow plugin, and resolve more issues for Gradle 5.0. (#3918) 2018-09-13 10:55:52 +01:00
cburlinchon
90c5f82514
Use external bridge (#1384) 2018-09-11 10:22:07 +01:00
Shams Asari
31530b35c6 Merge fixes 2018-09-10 11:45:53 +01:00
Shams Asari
46cbfefc77 Merge remote-tracking branch 'open/master' into os-merge-4f8a564
# Conflicts:
#	experimental/behave/src/main/kotlin/net/corda/behave/process/Command.kt
#	node/src/main/kotlin/net/corda/node/internal/AbstractNode.kt
2018-09-07 14:49:52 +01:00
Anthony Keenan
458bedd936
Merge pull request #1374 from corda/anthony-os-merge-20180906
O/S Merge 20180906
2018-09-07 12:14:23 +01:00
Shams Asari
c6400cf344 Some clean up of behave code (#3907)
In particular, fixing the recursive call of the "use" method
2018-09-06 16:44:18 +01:00
Anthony Keenan
4a70d1a996 Fixing enterprise only uses of NetworkMapCacheImpl 2018-09-06 14:50:44 +01:00
Patrick Kuo
767580c298 Flow worker POC zookeeper flow partitioning (#1369)
* Flow worker zookeeper WIP

* remove copyright decaration

* remove shaded curator and depends on node-api shadow instead
2018-09-06 10:09:31 +01:00
Michele Sollecito
b36240a7e7 Merge remote-tracking branch 'remotes/open/master' into fixes/certificateStoreStubs 2018-09-04 17:33:29 +01:00
Michele Sollecito
31e58dd2e5 [CORDA-1937]: Fixes to enterprise float and bridge. 2018-09-04 11:20:34 +01:00
Clinton Alexander
a28fa69865 Updated repository lists to reduce dependency on Jitpack and removed unused repositories. 2018-09-04 11:19:03 +01:00
Mike Hearn
97aef9c8a1 Merge remote-tracking branch 'open/master' into mike-merge-f6ee263db10 2018-09-03 20:13:38 +02:00
Shams Asari
11e2bef240
ENT-2473: Removed copyright headers (#1364) 2018-09-03 16:38:52 +01:00
Matthew Nesbit
7ead9e5698 ENT-2293 & ENT-2353: Basic implementation of an RPC worker and Flow Worker (#1308) 2018-08-29 13:45:27 +01:00
szymonsztuka
c7f666102f Fixes after merge remote-tracking branch 'remotes/open/master' into szymonsztuka/os-merge-20180824 2018-08-24 12:27:47 +01:00
Tommy Lillehagen
acd3f44181 Merge remote-tracking branch 'open/master' 2018-08-23 13:55:11 +01:00
Tommy Lillehagen
d2ef16cbfd
Deterministic JVM (#3386)
* CID-251 - Deterministic JVM

* CID-251 - Add DJVM documentation

* CID-251 - Address review comments from @chrisr3

* CID-251 - Address further review comments from @chrisr3

* CID-251 - Use shadowJar to generate fat JAR

* CID-251 - Address review comments from @exFalso

* CID-251 - Improve naming in ReferenceMap

* CID-251 - Add test for Kotlin meta-class behaviour

* CID-251 - Address review comments from @shamsasari

* CID-251 - Add description of high-level flow

* CID-251 - Refactoring

* CID-251 - Rename package to net.corda.djvm

* CID-251 - Include deterministic-rt.jar as runtime dependency

* CID-251 - Add Gradle task for generating whitelist from deterministic rt.jar

* CID-251 - Error messages for StackOverflow/OutOfMemory, update whitelist

* CID-251 - Reduce set definition of pinned classes

* CID-251 - Tidy up logic around pinned classes

* CID-251 - Shade ASM dependency and split out CLI tool

* CID-251 - Address review comments from @mikehearn (part 1)

* CID-251 - Address review comments from @mikehearn (part 2)

* CID-251 - Address review comments from @mikehearn (part 3)

* CID-251 - Address review comments from @exFalso

* CID-251 - Address review comments from @mikehearn (part 4)

* CID-251 - Address review comments from @exFalso and @mikehearn

* CID-251 - Address review comments from @mikehearn (part 5)
2018-08-22 16:01:39 +01:00
rick.parker
046209ea0d Fix compilation error 2018-08-09 10:27:58 +01:00
szymonsztuka
2939e68bd5 Merge OS - fix method name invoked in Enterprise module. 2018-08-01 14:30:03 +01:00
Tudor Malene
954ab0ca13
Merge pull request #1310 from corda/tudor-os-merge-31-07
Tudor os merge 31 07
2018-08-01 09:14:20 +01:00
Tudor Malene
46e42421a4 fix os merge 2018-07-31 18:30:41 +01:00
josecoll
4db315f81a
ENT-1424 - Setup Compatibility testing environment (#1211)
* Updated interoperability scenarios to use GA Release Candidate

* Removed "with proxy" and updated incorrect reference to OS master.

* Update to resolve Artifactory tools if available (blob-inspector, network-bootstrapper, database-migration)

* Allow passing in of argument to specify one or more scenarios to run.

* Updated to RC03.

* Added Oracle support in Behave framework and included functional test scenarios.

* behave test using snapshot (#970)

* make behave use snapshots from artifactory

* rebase and fix

* address PR issues

* address PR issues

* gradle task for bdd (#989)

* gradle task for bdd (#989)

* fix typo

* Publishing behave-api jar for use by external applications (eg. cordapp-option).
Fixed SIMMM valuation behave scenario test and startup.

* Publishing behave-api jar for use by external applications (eg. cordapp-option).
Fixed SIMMM valuation behave scenario test and startup.

* Various fixes to make BDD working in Teamcity (#1063)

* fix typo

* print docker log

* print docker log

* attempt to fix docker

* attempt to fix docker

* attempt to fix docker

* update to RC04

* disable clean up

* try to find out why the test pick up the wrong jar

* test

* minor fixes

* fix wrong bootstrapping procedure

* remove DISABLE_CLEANUP

* attempt to fix error

* fixes to get postgres working

* enable cleanup

* generate cucumber report

* bump to RC06

* attempt to make cucumber generate http report

* cucumber report

* some minor cleanup

* removed DP3 nodes from scenarios, added oracle and bumped to RC07

* fix licence issue

* fixed bootstrapper placing corda jar

* Update interoperability.feature

* Added missing published artifact.

* Fixed compilation errors raised by stricter compilation flag checking.

* Minor changes following rebase from master and re-testing.

* Updates to reflect last minute naming and versioning changes applied to 3.0 release.
Updated scenarios to use the officially released version of CE 3.0

* Updates following lastest PR review feedback.

* Updates following lastest PR review feedback.

* Align with master (maxMessageSize = mazTransactionSize).

* Fix snapshot version regex matching.

* Set logging level to DEBUG to ensure pattern matches catch log message output which has been downgraded from INFO

* Add all supported interoperable versions.
2018-07-31 17:53:24 +01:00
cburlinchon
0e807a2c3c
ENT-2353 - Flow Worker (#1307)
* Basic flow worker and serviceHub
2018-07-30 17:23:00 +01:00
Viktor Kolomeyko
2a61baa5df
ENT-2293: POC work for RPC Worker (#1300)
* ENT-2293: Skeleton of the new "rpcWorker" module

* ENT-2293: Add first failing test
(RPC server has not been implemented or started yet)

* ENT-2293: Further progress to start RPC server (unfinished)

* ENT-2293: RPC server started to work.

* ENT-2293: Slim down RpcWorker.

* ENT-2293: Minor changes.

* ENT-2293: Minor refactoring

* ENT-2293: Expand test coverage.

* ENT-2293: Introduce PicoCli for command line parsing.

* ENT-2293: Changes to make RPC server actually start.

* ENT-2293: Create simple RPC client.

* ENT-2293: Initialize serialization outside of RpcWorker.
2018-07-27 11:02:07 +01:00
Michele Sollecito
81b9885f03 Fixed a compilation error. 2018-07-23 17:16:24 +01:00
Viktor Kolomeyko
8c103d4247
ENT-2261: Systematic integration test for FlowProcessing (#1278)
* ENT-2261: Skeleton of parameterized test.

* ENT-2261: Programmatically compose rules and install them.

* ENT-2261: Separate counter decrement from termination.

* ENT-2261: Extend coverage to "executePersistCheckpoint".

* ENT-2261: Extend coverage to "ClientMessageImpl.acknowledge()"

* ENT-2261: Extend coverage to "RPCServer.context()"

* ENT-2261: Extend coverage to "ActionExecutorImpl.executeReleaseSoftLocks()" (unfinished)

* ENT-2261: Re-structure test data slightly.
"executeReleaseSoftLocks" seems to be failing.

* ENT-2261: Correct expectations in terms of number of transactions that ought to be produced.

Also use "eventually" construct to cater for node restart which may take some time.

* ENT-2261: Incorporate feedback from @exFalso and use polling approach.

* ENT-2261: Additional debug output.

* ENT-2261: Further expand coverage to RPCServer.context()

Also tighten-up assertion checks.

* ENT-2261: Further expand coverage to ActionExecutorImpl.executeCreateTransaction()

* ENT-2261: Further expand coverage to ActionExecutorImpl.executeRemoveCheckpoint()

* ENT-2261: Further expand coverage to ActionExecutorImpl.executePersistDeduplicationIds() and ActionExecutorImpl.executeCommitTransaction()

* ENT-2261: Revert non-material change.
2018-07-18 13:18:01 +01:00
Viktor Kolomeyko
a4e325a494
ENT-1967: Extend integration test to cover Node process termination scenario. (#1254)
* ENT-1967: Enable byteman library

* ENT-1967: Add an integration test to experiment with Byteman.

This needs to be running with: `-Dexperimental.test.enable`
As in: `gradlew -Dexperimental.test.enable integrationTest`

* ENT-1967: Modify Node driver to allow for optional instrumentation and use it in the integration test

* ENT-1967: Rely on port allocation

* ENT-1967: Install the rule that works

* ENT-1967: Trying to introduce counter rule (doesn't work)

* ENT-1967: Install rules that make correct use of countdown and also improve debug logging for Byteman

* ENT-1967: Add assertion to validate that exception is indeed thrown as per rules installed.

* ENT-1967: Less logging and more assertions

* ENT-1967: Replace `fun` with `val`

* ENT-1967: Un-break DriverDSL public API.

* ENT-1967: Minor change

* ENT-1967: Remove Byteman settings from NodeParameters and hide them inside InternalDriverDSL.

* ENT-1967: Introduce node restart scenario

* ENT-1967: Change the place where JVM is terminated and ensure that StdOut and StdErr not lost during restarts.

* ENT-1967: Changes after rebase.

* ENT-1967: Do not use InMemory DB for test that involves Node restart.

* ENT-1967: Minor fixes post merge from `master`.

* ENT-1967: Tighten-up test checks.
2018-07-11 12:54:56 +01:00
Viktor Kolomeyko
c396b80afe
ENT-1967: Illustration for Byteman library can be used in Node integration test. (#1204)
* ENT-1967: Enable byteman library

* ENT-1967: Add an integration test to experiment with Byteman.

This needs to be running with: `-Dexperimental.test.enable`
As in: `gradlew -Dexperimental.test.enable integrationTest`

* ENT-1967: Modify Node driver to allow for optional instrumentation and use it in the integration test

* ENT-1967: Rely on port allocation

* ENT-1967: Install the rule that works

* ENT-1967: Trying to introduce counter rule (doesn't work)

* ENT-1967: Install rules that make correct use of countdown and also improve debug logging for Byteman

* ENT-1967: Add assertion to validate that exception is indeed thrown as per rules installed.

* ENT-1967: Less logging and more assertions

* ENT-1967: Replace `fun` with `val`

* ENT-1967: Un-break DriverDSL public API.

* ENT-1967: Minor change

* ENT-1967: Remove Byteman settings from NodeParameters and hide them inside InternalDriverDSL.

* ENT-1967: Change the way how Jars resolved and use `Try` construct.
2018-07-10 10:05:07 +01:00
Shams Asari
300e0573fe
Merge pull request #1205 from corda/os-merge-244167d
O/S merge from 244167d
2018-07-05 17:16:52 +01:00
Shams Asari
0c8c914882 Removing warnings 2018-07-05 15:45:28 +01:00
Matthew Nesbit
123a2af5e0 Bridge to Firewall renames
More renaming

Update diagrams

Update changelog to point out breaking change fo the rename

Address Richard's PR comments
2018-07-05 15:20:28 +01:00
Shams Asari
20a589d66d Merge remote-tracking branch 'open/master' into os-merge-244167d
# Conflicts:
#	docs/source/example-code/src/main/kotlin/net/corda/docs/FlowCookbook.kt
#	docs/source/example-code/src/main/kotlin/net/corda/docs/LaunchSpaceshipFlow.kt
#	docs/source/example-code/src/main/kotlin/net/corda/docs/tutorial/tearoffs/TutorialTearOffs.kt
#	docs/source/running-a-node.rst
#	experimental/behave/build.gradle
#	experimental/behave/src/scenario/kotlin/net/corda/behave/scenarios/helpers/Substeps.kt
#	experimental/kryo-hook/build.gradle
#	experimental/quasar-hook/build.gradle
#	node/src/integration-test/kotlin/net/corda/node/AuthDBTests.kt
#	node/src/main/kotlin/net/corda/node/services/config/NodeConfiguration.kt
#	node/src/test/java/net/corda/node/services/vault/VaultQueryJavaTests.java
#	node/src/test/kotlin/net/corda/node/internal/NodeTest.kt
#	node/src/test/kotlin/net/corda/node/services/messaging/ArtemisMessagingTest.kt
#	node/src/test/kotlin/net/corda/node/services/persistence/HibernateConfigurationTest.kt
#	samples/attachment-demo/src/integration-test/kotlin/net/corda/attachmentdemo/AttachmentDemoTest.kt
#	samples/irs-demo/src/integration-test/kotlin/net/corda/test/spring/SpringDriver.kt
#	testing/node-driver/src/main/kotlin/net/corda/testing/node/internal/NodeBasedTest.kt
2018-07-04 18:38:34 +01:00
Shams Asari
244167d3e9
Enabled warnings as errors (#3514) 2018-07-04 17:17:27 +01:00