Commit Graph

682 Commits

Author SHA1 Message Date
Jonathan Locke
612fd83893 Fixed node driver issue 2019-11-14 11:25:56 +00:00
Jonathan Locke
c31cddaffc Fixed merge conflicts 2019-11-14 10:40:12 +00:00
Dan Newton
b15db200e3 NOTICK: Quick and dirty change to stop "Unable to start notaries." error message (#5686)
* Quick and dirty change to stop "Unable to start notaries." error message (#5686)

"Unable to start notaries. A required port might be bound already" is
returned whenever a startup error occurs while starting the notary nodes
in driver tests. This hides the real error.

This change prints the actual error to std_err and read from file
at a later point. This means the real error is not lost and will be
shown in failed builds.

* Suppress detekt warnings
2019-11-07 09:43:29 +00:00
stefano
6a3a9eded3 Merge remote-tracking branch 'origin/release/os/4.3' into my_merge_branch
# Conflicts:
#	core/src/main/kotlin/net/corda/core/internal/TransactionVerifierServiceInternal.kt
#	node/src/integration-test/kotlin/net/corda/node/services/rpc/RpcExceptionHandlingTest.kt
#	testing/node-driver/src/main/kotlin/net/corda/testing/node/internal/DriverDSLImpl.kt
2019-11-05 17:19:18 +00:00
Chris Rankin
02ed8e91b3 CORDA-2876: Ensure NodeConfig does not have devModeOptions outside of devMode. (#5676) 2019-11-05 16:59:40 +00:00
Chris Rankin
f226ddc4f2 CORDA-2876: Integrate the DJVM with the Corda Node. (#5633)
* * CORDA-2876: Migrate DJVM serialization modules into Corda.
* Pre-generate Corda classes for DJVM sandbox when node boots in production mode.
* Ensure that all DJVM test contract CorDapps are signed.
* Test examining attachments within DJVM sandbox.
* Test Contract.verify() using cryptographic verify function.
* Add test cases for more non-determinism in Contract.verify().
* Update node-driver to support testing nodes with DJVM support.
* Modify Node to allow alternative DJVM configurations for testing.
* Refactor DeterministicVerifierFactoryService for default use-case.
* Small whitespace and code-style refactors.
* Create and activate a DJVM execution profile for the Node.
* Revert making Verifier implement AutoCloseable.
* Allow the node to cache sandboxed Corda byte-code for reuse.
* Use updated Quasar agent that knows not to touch DJVM classloaders.
* Fix Quasar's package exclusions globs for DJVM.
* Deserialise LedgerTransaction into the sandbox for Contract.verify().
* Add the DJVM's serialisation modules to the Corda node.
* Update the node for the latest DJVM API, and preserve the ConstructorForDeserialization annotation on user contract classes.
* Add corda-dev to repositories while DJVM is SNAPSHOT.
* Migrate DJVM specialisation into AbstractNode's ServiceHubInternalImpl.
* Exclude sandbox.** and shaded djvm.** classes from Quasar agent.
* Add the corda-dev repository to :node for the deterministic runtime.
* Turn Verifier into an abstract base class that is specialised by BasicVerifier and DeterministicVerifier.
* Add the Corda deterministic libraries to the Node, and split the DJVM sandbox across two SandboxClassLoader instances.
* Add DJVM to contract verification path inside Corda Node.
* Minor lambda simplifications and removing unused import.
* CORDA-2871: Remove @CordaSerializable from LedgerTransaction.
* CORDA-2871: Add a callback to ServicesForResolution to allow the Node to modify a LedgerTransaction object.
* CORDA-2871: Refactor the contract verification code into a separate class,
 and allow LedgerTransaction to choose different Verifier objects.
* Update DJVM to use Corda 4.4-SNAPSHOT. (#95)
* CORDA-3330: Allow DJVM to preload / pregenerate classes from selected jars. (#92)
* Add support for SourceClassLoader.getResources() to DJVM.
* Allow a SandboxConfiguration to preload sandbox byte-code for all classes inside jars containing META-INF/DJVM-preload.
* CORDA-3309: Remove explicit try-catch in favour of UncaughtExceptionHandler. (#91)
* CORDA-3309: Install UncaughtExceptionHandler for DJVM tasks. (#88)
* Fix tests broken by Windows line endings. (#82)
* CORDA-3292: Reimplement ExecutionProfile as a data class. (#80)
* CORDA-2877: Refactor how we create child SandboxConfiguration objects. (#76)
* CORDA-2877: Load bytecode from a persistent cache to prevent repeated rewriting. (#75)
* Refactor byte-code cache to SandboxConfiguration instead of AnalysisConfiguration. We cannot "mix and match" byte-code generated by different sets of rules.
* CORDA-3137: Enhance annotation handling so that we can allow some annotations to be mapped into the sandbox without also needing to be stitched. (#72)
* CORDA-2871: Minor cosmetic fixes. (#69)
* CORDA-3218: Align DJVM with internal Corda Serialisation API. (#68)
* Ensure we get the latest SNAPSHOT of the serialisation code.
* CORDA-2871: Refactor SourceClassLoader to define source classes. (#66)
* Rewrite SourceClassLoader to support parent/child relationships.
* Revert catching TypNotPresebtException - it was a symptom of a bigger problem.
* Remove AutoCloseable from AnalysisConfiguration and SourceClassLoader.
* SourceClassLoader.getResource() must delegate to its parent first.
* CORDA-2871: Ensure ClassLoader.loadClass() throws ClassNotFoundException for all cases where the class cannot be found. (#64)
* CORDA-2871: Modify sandbox tasks to implement both java.Function and sandbox.Function (#62)
* Make TaskExecutors implement BiFunction to make them composable.
* Create ImportTask to wrap a java.Function inside a sandbox.Function.
* Add createExecutor() and createRawExecutor() APIs to SandboxClassLoader.
* Update serialization to use SandboxClassLoader.toSandboxClass().
* Remove a layer of lambdas from the serialisation code.
* Update SandboxExecutor and SandboxRawExecutor.
* Rename Executor to TaskFactory.
* Rename dangling executor -> taskFactory.
* CORDA-2871: Sanity fixes! (#63)
* Improve message for SandboxClassLoadingException.
* Fix serialisation API for using sandboxed environment.
* CORDA-3174: Extend serialisation to include InputStream and OpaqueBytesSubSequence. (#60)
* Update DJVM Example project for serialisation.
* Add serializers for InputStream and OpaqueBytesSubSequence.
* Support ZIP Inflater and CRC32 inside the sandbox.
* Allow the DJVM to wrap java.io.InputStream as sandbox.java.io.InputStream.
* Configure tests also to preserve @DeprecatedConstructorForDeserialization.
* CORDA-3174: Implement Corda serialization modules. (#59)
* Create DJVM serialization modules.
* Create test cases for Array<T>, List<T> and List<Array<T>>.
* Refactor SandboxPrimiveSerializer for all primitive types.
* Implement SandboxCollectionSerializer to support Collection types.
* Implement SandboxMapSerializer to support Map types.
* Attempt to fix infinite loop when computing Collection and Map fingerprints.
* Apply special handling when deserialising sandbox.java.lang.Character.
* Remap Java primitive types to sandbox Java object types to deter evolution.
* Use Class.getPackage().getName() to determine sandbox package name.
* Implement SandboxEnumSerializer to support Enum types.
* Implement SandboxPublicKeySerializer to support Java security keys.
* Add serialization projects to the composite example project.
* Implement serializers for BigInteger, BigDecimal, Currency and StringBuffer.
* Test that deserialising does not instantiate the untrusted user classes.
* Implement serializers for java.time.* types.
* Add serialiser for BitSet - currently disabled until BitSet itself is supported.
* Add serialisers for EnumSet and Class.
* Include support for EnumMap in the SandboxMapSerializer.
* Ensure the DJVM Example project's tests preserve @CordaSerializable.
* Add support for UUID as a primitive type.
* Use common abortReadOnly() method for declaring serialization as unsupported.
* Streamline the API for deserialising into the sandbox.
* Add preliminary support for deserialising X.509 certificates.
* Implement serializer for java.util.Optional.
* Refactor configuration of the sandbox serialization scheme.
* Add tests for deserialising arrays of basic types.
* Include method annotations in annotation stitching. This ensures that `@ConstructorForDeserialization` is not dropped.
* Enable test for SandboxBitSetSerializer.
* Enable tests for X.509 serializers.
* Implement serializers for ProtonJ primitive types.
* Serialize java.util.Date as a primitive type.
* Add the bintray Gradle plugin to the serialisation modules.
* Do not publish serialisation modules - they will become part of Corda itself.

* CORDA-2876: Only apply DJVM sources to Node Driver when devMode=true.

* Resolve DeteKT warnings.

* Require Node's JVM to set -Dnet.corda.djvm=true in order to enable DJVM.

* Enable DJVM for DemoBench nodes.

* Disable Quasar instrumentation verification for DemoBench nodes.

* Upgrade to DJVM 1.0-RC01.

* Try to modify DriverParameters in a more "ABI friendly" way.

* Refactor and simplify sandbox deserialisation of primitive objects.

* Review fixes.

* Update EvolutionSerializerFactory to handle sandboxed primitive boxed types.
2019-11-05 13:44:18 +00:00
Dimos Raptis
485feb2d6c CORDA-3350: Increase size of constraints column (#5639)
* CORDA-3350: Increase size of constraints column (#5639)

* Detekt

* Update api file with new threshold

* Add check in transaction builder

* Revert "Add check in transaction builder"

This reverts commit ca3128f44c.

* Add check for max number of keys

* Update api file

* Address Tudor's comments

* Remove check for pre-5 and add test for EC keys

* fix typo and rename liquibase script

* updated docs with measurement numbers for composite keys

* Make detekt happy again
2019-11-05 11:00:26 +00:00
Jonathan Locke
c193aa46f0
CID-1154: reliable finality merge to OS (#5658)
CID-1154: reliable finality merge to OS (#5658)
2019-11-05 10:48:00 +00:00
Stefano Franz
e4e920eee9
multiprocess port allocator is no longer used, so we can remove the tests as they add a significant amount of time to run (2-3 min) (#5663) 2019-11-04 11:56:38 +00:00
LankyDan
3c0631a26a Fix VaultObserverExceptionTest due to differences between ENT and OS 2019-11-04 09:42:41 +00:00
Barry
91e6c9783f TM-51 Read and write test results to artifactory. (#5597)
* TM-51  Prep for reading and writing test results to artifactory.

* TM-51  Tests from target branch if no tests for current branch

* TM-51  Placeholder for test averaging over runs.

* TM-51  Replace slashes in branch names used as tags.

* TM-51  More placeholder work for the mean duration work.

* TM-51  Write out average tests results as as csv.

The csv file should grow and be updated on each run.  This includes whether or not we are running unit tests, integration tests and so on.

* TM-51  Comment out old junit test archiving, add more comments.

* TM-51  Zip task needs to depend on a csv creation task.

If there isn't a csv file present, then the zip task doesn't run due to 'NO-SOURCE'

* TM-51  Zip task should ignore empty dirs

* TM-51  Fix up loading of test results.

We were looking for the wrong artifact name.
Add a bit more logging.

* TM-51  Fix up possible problem with allocating by class distribution.

If we encounter a class we haven't seen before, there won't be any tests.
This means we should give it some weight.  '1' is far too small.

* TM-51  Test that we are definitely increment the run count.

Tracking down whether the zipped csv file should have incremented.

* TM-51  Better default value for missing test/class names.

Begin by using mean unit test duration, but we have the option to bump
that to the mean class unit tests duration.

* TM-51  More debug information around csv writing.

We should be incrementing the tests.

* TM-51  Reload the csv before updating it.

* TM-51  Reduce verbosity of logging.

* TM-51  Reinstate unit tests.  Remove logging verbosity.

* TM-51  Load tests from artifactory in memory and avoid interim file.

* TM-51  Better handling of zero duration tests.

Ensure we return zero times from junit artifacts which may either be zero or have no recorded time.  Before writing the tests duration csv file, store those with a known time, and then store those with zero using the average time.

* TM-51  Log whether we have recorded a test.

Tracking down the curious case where we seem to not be rerunning the
same set of tests on the second run.

* TM-51  Capture junit files as well.

Trying to track down whether some tests are intermittently run.

* TM-51  Change task dependencies to ensure ziptask is triggered.

* TM-51  Remove test assertion, and trigger build

* TM-51  Add corda/enterprise to artifactory tag name.

Moved properties to own file.

* TM-51  Remove unnecessary mean class-based duration.

* TM-51  Add more BucketingAllocator tests.

We need these to nail down its behaviour some more.

* TM-51  Further log information.

We don't seem to be finding the tests in the 'production' runs which is odd.

* TM-51  corda type double set?

* TM-51  do not set the project type in the properties.

SRP and all that.

* TM-51  better plan reporting

* TM-51  duration may be zero

Another runtime problem that doesn't show in tests.

* TM-51  better plan reporting

* fix missing space after image id

* fix merge issue in DistributedTesting

* TM-51  remove unused code when GET/PUT-ting to Artifactory.

* TM-51  put tasks in gradle group and tidy up zip task creation

* TM-51 Fix the junit XML path.

* TM-51 Fix the task graph

* TM-51 Less logging
2019-11-02 09:07:53 +00:00
LankyDan
3aaddb47ea Attempt to fix detekt issues in DriverDSLImpl again 2019-11-01 14:36:47 +00:00
LankyDan
dfb86f5d9c Attempt to fix detekt issues in DriverDSLImpl 2019-11-01 14:21:17 +00:00
LankyDan
4aa9add8c8 CORDA-2050 Upgrade Corda to Java 11 (compatibility mode) (#5356) Upgrade Corda to run with Java 11 (compatibility mode) - see https://github.com/corda/corda/pull/5356 - 3fafbe55
Reapply change that was lost during merge - Adjust resolution of byteman jar to use java 11 compatible mechanism. - a1077092

Manual cherry pick of these changes (a1077092 + 3fafbe55)
2019-11-01 11:50:16 +00:00
Viktor Kolomeyko
c53ea9dde5 Eliminate extensive printout when Byteman not found on the classpath. (#1277)
Byteman is absent for most of the integration tests and long stacktrace is seen in the log
presently that un-necessarily attracts attention and consumes logging space.

(cherry picked from commit 2b6e59e7bd)
2019-11-01 11:50:16 +00:00
LankyDan
bedfba8c3d ENT-1967: Illustration for Byteman library can be used in Node integration test. (#1204) - c396b80a
(Only took `DriverDSLImpl` changes)

Simplifying internal startNode with bytemanPort parameter - d88b02f7

Manual cherry pick of these changes (c396b80a + d88b02f7)
2019-11-01 11:50:16 +00:00
Christian Sailer
9f15457045 add else branch to avoid warning that fails the warning check 2019-11-01 11:48:07 +00:00
Christian Sailer
119f939ee1 Fix and or suppress detekt warnings 2019-11-01 11:48:07 +00:00
Dan Newton
268d129838 CORDA-3194 Replay start flow events when responding flow fails initial checkpoint commit (#2601)
* CORDA-3194 Replay start flow events when responding flow fails initial checkpoint commit

Logic has already been added to recover from initial checkpoint commit
failures on the initiating flow side but this did not suffice for
the same failure occurring on the responding flow's side.

The same idea has been added to resolve the responding flow's issue.

`ExternalMessageEvent` now has a `flowId` that is maintained on the
event. Messages can then be replayed to start/restart the flow, while
the event provides the flow id to each flow start.

Each `ExternalMessageEvent` implementation generates a random `flowId`
when constructed.

Events are stored in Artemis. This allows the solution to recover across
node restarts as the events will be pulled from artemis again when
restarting.

In the future `flowId`s will probably moved off of the events and
generated purely on the responding flow's node.

* CORDA-3194 Add test to verify that errors removing a responding flow are recoverable
2019-11-01 11:48:07 +00:00
Christian Sailer
1f71b071aa CORDA-3217 and CORDA-3195 Various bits arond SQL exceptions and flow hospital (#2605)
* Unwrap rx.OnErrorNotImplementedException so the hospital can handle the cause appropriately

* Add db failure cordapp

* Renamed folders to avoid ambiguity in gradle

* Add integration test for exception hospitalisation when thrown from an RX observable.

* Make the test slightly cleaner

* Fix the schema to actually match the requirements for my custom state. Thanks a bunch, H2.

* Switch test to use SqlException base class.

* Schedule error event if we detect that a commit or db flush has thrown (forcing the flow to error even if customer code then goes ahead to swallow the exception)

* Revert change to schedule extra error

* Add more tests for edge case with DB exceptions, changed CorDapp to suppor this an hook in the flow hospital

* Warning about unsubscribe
Check state transitioned from clean to error for hospital admission.

* Match the test to our actual expectations

* Revert "Revert change to schedule extra error"

This reverts commit 43d47937

* Prevent suppression of errors arising in `transaction()` and `jdbcConnection()`

* Test for SqlException caught trying to escape from recordTransaction and suppressed outside being intercepted.

* More tests for various error/catch combinations

* Clean up and comments

* Code reformat

* Fix test compilation
2019-11-01 11:48:07 +00:00
Stefan Iliev
03ab258fc2 Revert "CORDA-3307 - add support for environment variables in linux (#5523)" (#5643)
This reverts commit c882b221a5.
2019-10-29 17:55:58 +00:00
Stefano Franz
d693a9c1ce
TM-65 re-add local port availability check (#5618)
* re-add local port availability check

* attempt to fix issue with port allocator

* ensure the serversocket is closed
2019-10-21 16:17:17 +01:00
Stefan Iliev
c882b221a5 CORDA-3307 - add support for environment variables in linux (#5523)
* Added a new way for environment variables to be loaded, which allows for underscore based separation.

* Moved test to its own kotlin file.

* Added case insensitivity support.

* The corda. prefix is now case insensitive too.

* Removed unused variable.

* Added env variables support for driverDSL. Shadowing corda. properties raises an exception.

* Driver api stability fix.

* Changed type of cordapps param to reflect the real one, rather than what IntelliJ auto completed.

* Some detekt issue fixes. Spread operator removed, baselined api stability constructors and buggy line.

* Fixed misspelled variable.

* Reverted unintentional changes.

* Added suppress instead of changing baseline.

* Reworked logic to handle previously defined CORDA_ starting properties and handle accordingly. Fixed a bug where wrong class was used for reflection walking.

* Fix for detekt issues.

* Changed message to a more understandable one.

* Changelog + doc note, console error grammar.

* Changes according to PR review.
2019-10-21 12:01:14 +01:00
Stefano Franz
6e98adb085
use zulu for jdk in testing image (#5583)
* use zulu for jdk
add some parallel groups

* port kubesTest to Java
remove asterix from tests listed by ListTests, instead add after allocation
2019-10-14 13:35:22 +01:00
Dimos Raptis
7666ca0d80 [CORDA-3303] - Avoid flushing when inside a cascade (#5575)
* [CORDA-3303] - Avoid flushing when inside a cascade

* Remove listener infrastructure
2019-10-11 10:58:26 +01:00
Stefano Franz
298c91ce82
Webook test branch (#5528)
* add ability to index by class OR method

* disable unit tests again

* pending commit

* stream output of build for debugging

* remove git from the base image

* re-enable NodeRPCTests

* add ability to distribute tests based on TeamCity CSV output for test times

* try enabling unit tests again after test distribution has been fixed

* refactor BucketingAllocator to be a class, which bucketingAllocatorTask delegates to.

* finishing touches for improved test distribution

* create new pipelines for builds
2019-10-08 16:33:24 +02:00
josecoll
246ec8766e
CORDA-3224 JDK11 built and published artifacts to include classifier. (#5488)
* JDK11 built and published artifacts to include classifier.

* Added additional non-intuitive gradle dust sprinkled recommended by CR.

* Added additional non-intuitive gradle dust sprinkled recommended by CR.

* Bump corda gradle plugins version.

* Incorporating PR review feedback.

* jdk11Classifier
2019-10-08 10:45:10 +01:00
Ryan Fowler
6ca4b589e2 CORDA-3091: Move executor thread management into CordaRPCConnection (#5491) 2019-09-26 17:48:07 +01:00
Viktor Kolomeyko
51330c2e44 CORDA-3232: Support of multiple interfaces for RPC calls (#5495)
* CORDA-3232: Make backward compatible RPC client changes

Such that it will be able to talk to new and old server versions.

* CORDA-3232: Make backward compatible RPC server changes

Such that it will be able to talk to new and old client versions.

* CORDA-3232: Trick Detekt

* CORDA-3232: Integration test for multi-interface communication.

* CORDA-3232: Add legacy mode test.

* CORDA-3232: Making Detekt happier

* CORDA-3232: Fix Detekt baseline after merge with `4.3` branch

* CORDA-3232: Incrementing Platform version

As discussed with @lockathan

* CORDA-3232: Fix legacy test post platform version increment

* CORDA-3232: Use recursive logic to establish complete population of method names

* Revert "CORDA-3232: Incrementing Platform version"

This reverts commit d75f48aa

* CORDA-3232: Remove logic that conditions on PLATFORM_VERSION

* CORDA-3232: Making Detekt happier

* CORDA-3232: Few more changes after conversation with @mnesbit

* CORDA-3232: Make a strict match to `CordaRPCOps` on client side

Or else will fail:
net.corda.tools.shell.InteractiveShellIntegrationTest.dumpCheckpoints creates zip with json file for suspended flow

Flagging that `InternalCordaRPCOps.dumpCheckpoints` cannot be called.

* CORDA-3232: Address PR comments by @rick-r3

* CORDA-3232: Address further review input from @rick-r3

* Change the way how methods stored in the map;
* Extend test to make sure that `CordaRPCOps` can indeed be mixed with other RPC interfaces.
2019-09-26 16:01:14 +01:00
Tudor Malene
298d8ba69c CORDA-3024 Rename the webserver (#5489) 2019-09-26 10:20:49 +01:00
Ed Prosser
defa2565ea Merge branch 'release/os/4.1' of https://github.com/corda/corda into ENT-4104-4.3-EdP 2019-09-25 10:47:06 +01:00
josecoll
f4f46af706
CORDA-3209 Fix vault query for participants specified in common criteria (#5508)
* Generalise participant parsing code & additional test cases.

* Use a common predicate to expand the participants query (when specified more than once - eg. in fungible and linear query criteria).

* Introduce some re-usable functions.

* Additional code clean-up and improvements.

* Fix detekt MaxLineLength errors.
2019-09-24 10:32:06 +01:00
opticyclic
eaed69f26f Check If Quasar Is Active Using API (#5471)
* Do a proper check for whether the quasar agent is active instead of sniffing JVM args

* Use JavaAgent.isActive instead of SuspendableHelper.isJavaAgentActive for clarity
2019-09-18 17:15:56 +01:00
josecoll
3fafbe551c
CORDA-2050 Upgrade Corda to Java 11 (compatibility mode) (#5356)
Upgrade Corda to run with Java 11 (compatibility mode) - see https://github.com/corda/corda/pull/5356
2019-09-18 10:26:26 +01:00
Roger Willis
1c2c3d3fed
Identity service refactor for confidential-identities and accounts (#5434)
* Removed IdentityServiceInternal as it is no longer used.

* Removed externalIdForPublicKey API from KMS and added it to IdentityService.
Added a registerKeyToExternalId API on IdentityService.

* Fix remaining compile errors.

* Removed "registerKeyToParty" and in its place added a new registerKey method which takes a PublicKey, Party and optionally a UUID.
Added a cache to the "PersistentIdentityService" to store other node's public keys.
Added the cache and new hibernate entity to all teh places where one needs to add them.
New keys created by teh node now automatically get associated entries in the KEY -> PARTY map and optionally the KEy -> EXT ID map.
Added a test.

* Removed old comments and TODOs.

* Fixed broken test.
Added comments/explanations for what's going on in IdentityService.
Updated kdocs.

* First try at Implementing publicKeysForExternalId.

* Fixed broken test.

* Added migration.
Amended existing persistent identity service migration to handle new migration.
Addressed some review comments.

* Fixed broken test - whoops!

* Implemented mock identity service methods.

* Added back exception when remapping a key to a different party.

* Fixed compile errors.
Fixed broken tests.

* Use set instead of first entry in ourNames.
2019-09-16 13:14:12 +01:00
Zoltan Kiss
f171de7b69 TM-23 Fail build on compiler warnings (#5453)
* java compile respects compilation.allWarningsAsErrors

* suppress or cleanup warnings

* suppress warning

* use non-deprecated kotlin dependency

* rename property

* handle property existence check

* Deal with warnings
2019-09-11 16:34:51 +01:00
Dan Newton
4cbe22949d CORDA-3018 Whitelisting attachments by public key - phase two tooling (#5386)
Allow node operators to blacklist signing keys (using blacklistedAttachmentSigningKeys config option). These blacklisted keys prevent attachments that are received over the network from being trusted. The docs have been updated to detail how to generate the key hashes that the config requires.

A new shell command attachments trustRoots has been added to see what attachments exist on the node along with information about their trust and where it comes from.

run dumpCheckpoints has been replaced by checkpoints dump as InternalCordaRPCOps needed to change to prevent a function that is meant to be internal from being visible on the shell.
2019-09-10 12:16:34 +01:00
Stefano Franz
a842740c9e
WIP Kubenetes parallel build (#5396)
* Split integration tests

* add simple example of printing all methods annotated with @Test

* add docker plugin to root project
remove docker plugin from child projects
add Dockerfile for image to use when testing
add task to build testing image to root project

* add comment describing proposed testing workflow

* simple attempt at running tests in docker container

* add my first k8s interaction script

* add fabric8 as dependnency to buildSrc

* before adding classpath

* collect reports from containers and run through testReports

* re-enable kubes backed testing

* for each project
1. add a list tests task
2. use this list tests task to modify the included tests
3. add a parallel version of the test task

* tweak logic for downloading test report XML files

* use output of parallel testing tasks in report tasks to determine build resultCode

* prepare for jenkins test

* prepare for jenkins test

* make docker reg password system property

* add logging to print out docker reg creds

* enable docker build

* fix gradle build file

* gather xml files into root project

* change log level for gradle modification

* stop printing gradle docker push passwd

* tidy up report generation

* fix compilation errors

* split signature constraints test into two

* change Sig constraint tests type hierarchy

* tidy up build.gradle

* try method based test includes

* add unit test for test listing

* fix  bug with test slicing

* stop filtering ignored tests to make the numbers match existing runs

* change log level to ensure print out

* move all plugin logic to buildSrc files

* tidy up test modification
add comments to explain what DistributedTesting plugin does

* move new plugins into properly named packages

* tidy up runConfigs

* fix compile errors due to merge with slow-integration-test work

* add system parameter to enable / disable build modification

* add -Dkubenetise to build command

* address review comments

* type safe declaration of parameters in KubesTest
2019-09-03 15:40:08 +00:00
Stefano Franz
c2057e0893 CORDA-3180: Added ability to lookup the associated UUID for a public key to KeyManagementService (#5411)
* expose identity cache to KMSinternal

* apply shams comments

* Addressed review comments.
2019-09-01 07:57:46 +01:00
Stefano Franz
cd0d5c7724 Ensure that ServiceHub.WithEntityManager has a database transaction available (#5413)
* tidy up withEntityManager

* address rick review comment

* api break
2019-08-30 15:58:50 +01:00
Dimos Raptis
3023e3ab87 [NOTICK] Expose type in CryptoService (#5416) 2019-08-30 09:22:15 +01:00
Stefano Franz
e35c0c1df7
CORDA-3009 - Migrate identity service to use to string short (#5217)
* migrate PersistentIdentityService to use key.toShortString()
update definition of PublicKeyToExternalId mapping to allow fast lookup by externalId/publicKey

* fix misspelled table name

* add test of migration script

* add design document for proposal to move IdentityService to using the correct PK.toStringShort() method for hashing a publickey

* add enterprise testing considerations to design

* address review comments

* fix compilation errors

* modify PublicKeyToOwningIdentityCache to use toStringShort() as it's lookup key

* address syzmon's code review comments
2019-08-28 14:16:18 +00:00
James Higgs
101d978050 [CORDA-3149] Update cache to check node identity keys in identity table (#5371) 2019-08-19 20:03:51 +01:00
Shams Asari
69d1b4e4e5
CORDA-3138: Removed InMemoryTransactionsResolver as it's not needed and other resolution cleanup (#5370) 2019-08-16 12:34:25 +01:00
James Higgs
72ac722451 [CORDA-3130] Add a cache for looking up external UUIDs from public keys (#5357) 2019-08-14 13:24:56 +01:00
Viktor Kolomeyko
6db3ded032 CORDA-3139: Cater for port already bound scenario during port allocation (#5361)
* CORDA-3139: Cater for port already bound scenario during port allocation

Also moved `SharedMemoryIncremental` into a separate file as it getting bigger
and improved readability of logic and added some logging.

* CORDA-3139: Fix the unit test

* CORDA-3139: Improve logging when failing

* CORDA-3139: Improve stability of the test
2019-08-12 10:38:15 +01:00
Dan Newton
fc265ee472 CORDA-3018 Whitelisting attachments by public key - relax signer restrictions (#5358)
* CORDA-3018 Allow loading an untrusted contract jar if another attachment exists that was signed with the same keys and uploaded by a trusted uploader

`TransactionUtils.isAttachmentTrusted` requirements have been relaxed
to allow an untrusted attachment to be loaded as long as another
attachment exists that is signed by the same keys and was uploaded
by a trusted uploader.

The requirement of containing the same contract classes has been
removed. Therefore the contents of the existing trusted attachment
no longer matters.

* CORDA-3018 Allow a subset/intersection of signers in `isAttachmentTrusted`

Allow a subset/intersection of signers to satisfy the signer
requirements of `isAttachmentTrusted`. This allows an "untrusted"
attachment that is signed by one or more keys to be "trusted" as long
as another trusted attachment already exists that is signed by at least
one of the "untrusted" attachments signers.

A cache of trusted and untrusted public keys is now held (replacing the
previous cache of `List<PublicKey>`.

Tests have been added to `NodeAttachmentServiceTest` to confirm that
an attachment query using an `EQUAL` statement will actually return
attachments that are signed by any of the keys passed into the query.

Confirming this allowed an `EQUAL` query to satisfy the search that
had to be done as part of this change.

`MockAttachmentStorage`'s query criteria was updated to better match
the real `NodeAttachmentService` implementation.

* CORDA-3018 Update cache name and kdoc on `isAttachmentTrusted`

* CORDA-3018 Verify that chains of trust do not occur

* CORDA-3018 Switch keys around to improve chain of trust tests
2019-08-08 09:33:45 +01:00
James Higgs
44428b6048 ENT-3801 Backport to OS (#5355)
* [ENT-3801] Store transactions in the database during transaction resolution (#2305)

* ENT-3801: Store downloaded txns as part of the backchain resolution into the db rather than the checkpoint

It's very inefficient to store the downloaded backchain in the checkpoint as more of it downloaded. Instead, if a threshold is reached (which currently defaults at 0) then the backchain is stored in the transactions table as unverified. A new is_verified column has been added to track this. Initially testing on the OS codebase has been very promising but unfortunately this current code is not quite ready. I had to quickly port it to ENT as this is meant to be an ENT-only optimisation.

To that effect, there is a TransactionResolver abstraction with two implementations: an in-memory one which has the old behaviour, and which will be the behaviour for OS, and a db one.

DBTransactionStorage hasn't been fully updated and I had to comment out the optimistic path for now.

Most of these changes will need to be ported to OS to keep the merge conflicts in check, but obviously not DbTransactionsResolver and the "is_verified" changes in DBTransactionStorage. DBTransactionStorage does have other refactoring which will make sense to port though.

* [ENT-3801] Start work on allowing modifications in AppendOnlyPersistentMap

* [ENT-3801] Add transaction resolver tests

* [ENT-3801] Adjust suspendable annotations

* [ENT-3801] Fix the ResolveTransactionFlow tests

* [ENT-3801] Update ResolveTransactionsFlow tests

* [ENT-3801] Add a liquibase migration script for isVerified

* [ENT-3801] Ensure the migration runs in the correct place

* [ENT-3801] Handle resolution of already present transactions

* [ENT-3801] Fix compile error in performance test app

* [ENT-3801] Logging and comment updates, plus a test case

* [ENT-3801] Add a notary change resolution test

* [ENT-3801] Add a contract upgrade transaction test

* [ENT-3801] Change new column to be a character based status

* [ENT-3801] Migration script type change

* [ENT-3801] Address first round of review comments

* [ENT-3801] Update variable names in AppendOnlyPersistentMap

* [ENT-3801] Another variable name clarification

* [ENT-3801] Fix missing name changes

* [ENT-3801] Make the signature list immutable when constructing cache value

* [ENT-3801] Add a locking strategy for unverified transactions

* [ENT-3801] Address tidying up review comments

* [ENT-3801] First attempt at ensuring locks are released after commit

* [ENT-3801] Remove references to old cache name

* [ENT-3801] Update locking logic

* [ENT-3801] Fix potential deadlock with read/write transaction locks

* [ENT-3801] Remove read locks, and ensure minimal extra suspends

* [ENT-3801] Fix build issues in tests

* [ENT-3801] Use the correct clock when calculating sleep durations

* [ENT-3801] Add a pessimism flag for writing verified transactions

* [ENT-3801] Change logging statement to debug

(cherry picked from commit 8ab6a55e17)

* [NOTICK] Fix up imports for some changed files

* [NOTICK] Fix transaction resolution tests

* [NOTICK] Reinstate the DBTransactionsResolver

* [NOTICK] Add the topological sort back to recordTransactions

* [NOTICK] Adjust test case to remove dependency on query ordering

* [NOTICK] Make test code match that in ENT
2019-08-07 16:49:49 +01:00
josecoll
b4749eb8f9
CORDA-2991 (Cont): set node info polling interval to 1 second in DriverDSL Node Startup (#5285)
* Fix test execution flakiness on fast hardware: set node info polling interval to 1 second in DriverDSL Node Startup (was only being set to 1 sec in Node Registration). Follow-up to https://github.com/corda/corda/pull/5240

* Diagnostics: add additional logging to NodeInfoFilersCopier.

* Diagnostics: add additional logging to NodeInfoWatcher

* Downgrade logging level severity to debug.

* Make atomic counter unique across instances.
2019-07-17 14:48:46 +01:00
Matthijs van den Bos
af44dfa309 Fix API stability issue 2019-07-17 13:24:04 +02:00