Commit Graph

143 Commits

Author SHA1 Message Date
Adel El-Beik
1547efb093
CORDA-3755: Switched attachments map to a WeakHashMap (#6214)
* Bump OS release version 4.6

* CORDA-3755: Switched attachments map to a WeakHashMap

* CORDA-3755: Added explicit strong references to map key.

* CORDA-3755: Keeping detekt happy.

* CORDA-3755: Test a gc in verify.

* CORDA-3755: Making detekt happy.

* CORDA-3755: Suppress warnings for weak reference test.

* CORDA-3755: Fixing build failure with attachments.

* CORDA-3755: Rewrite based on Ricks input - now handles attachment already existing in map!

* CORDA-3755: Refactor WeakReference behaviour into AttachmentsHolderImpl and provide alternate version of this class for core-deterministic.

* CORDA-3755: Added more tests for WeakHashMap.

* CORDA-3755: Ignore the tests using System.gc keep for local testing only

* CORDA-3755: Adding comment to explain the ignored tests.

* Make AttachmentsHolderImpl package-private inside core-deterministic, just like it is inside core.

* CORDA-3755: Update assertions following review comments.

* CORDA-3755: Removing import

* CORDA-3755: Removed unused var.

* CORDA-3755: Reverting files that somehow got changed in rebase.

Co-authored-by: nargas-ritu <ritu.gupta@r3.com>
Co-authored-by: Chris Rankin <chris.rankin@r3.com>
2020-05-12 09:51:12 +01:00
LankyDan
565afc5fdb NOTICK Fix kill flow tests due to storing failed flows
Failed flows are stored after the checkpoint table rework. This meant
that some of the asserts in `FlowIsKilledTest` and `KillFlowTest` were
wrong.
2020-05-06 10:02:40 +01:00
nargas-ritu
5c16e361fd
Merge pull request #6186 from xavierlepretre/release/os/4.5
CORDA-3754 Make TransactionBuilder's input references deep copied on copy.
2020-05-01 09:54:18 +01:00
Xavier Leprêtre
c4fd63ced6 Reverted comment change about immutable accessors. 2020-04-30 18:46:14 +04:00
Dan Newton
cb84fd86ee
CORDA-3291 Fix broken tests (#6197) 2020-04-30 10:27:36 +01:00
Adel El-Beik
3259b595d7
CORDA-3715: Check contract classes hav… (#6155)
* CORDA-3715: When loading cordapps now check that contract classes have class version between 49 and 52

* CORDA-3715: Now check class version when contract verification takes place.

* CORDA-3715: Making detekt happy with number of levels in func

* CORDA-3715: Make use of new ClassGraph release which provides class file major version number.

* CORDA-3715: Changed package name in test jars

* CORDA-3715: Use ClassGraph when loading attachments.

* CORDA-3715: Reverted file to 4.5 version

* CORDA-3715: Updating method to match non deterministic version.

* CORDA-3715: Added in default param.

* CORDA-3715: Adjusted min JDK version to 1.1

* CORDA-3715: Switching check to JDK 1.2

* CORDA-3715: Now version check SerializationWhitelist classes.

* CORDA-3715: Switched default to null for range.
2020-04-30 08:57:37 +01:00
Dan Newton
297e504740
CORDA-3291 isKilled flag and session errors for killed flows (#6170)
* CORDA-3291 `isKilled` flag and session errors for killed flows

## Summary

Two major improvements have been worked on:

- A new flag named `isKilled` has been added to `FlowLogic` to allow
developers to break out of loops without suspension points.
- Killed flows now send session errors to their counter parties allowing
their flows to also terminate without further coordination.

Achieving these changes required a __fundamental__ change to how flows are
killed as well as how they sleep.

## `isKilled` flag

The addition of `FlowLogic.isKilled` allows flows to check if the
current flow has been killed. They can then throw an exception to lead
to the flow's termination (following the standard error pathway). They
can also perform some extra logic or not throw an exception if they
really wanted to.

No matter what, once the flag is set, the flow will terminate. Due to
timing, a killed flow might successfully process its next suspension
event, but it will then process a killed transition and terminate.

## Send session errors when killing a flow

A flow will now send session errors to all of its counter parties. They
are transferred as `UnexpectedFlowEndException`s. This allows initiated
flows to handle these errors as they see fit, although they should
probably just terminate.

## How flows are killed

### Before

Originally we were relying on Quasar to interrupt a flow's fiber, we
could then handle the resulting `InterruptedException`. The problem with
this solution is that it only worked when a flow was already suspended
or when a flow moved into suspension. Flows stuck in loops did not work.

### After

We now *do not* use Quasar to interrupt a flow's fiber. Instead, we
switch `FlowStateMachine.isKilled` to true and schedule a new event.
Any event that is processed after switching this flag will now cause a
`KilledFlowTransition`. This transition follows similar logic to how
error propagation works. Note, the extra event allows a suspended flow
to be killed without waiting for the event that it was _really_ waiting
for.

This allows a lot of the tidy up code in `StateMachineManager.killFlow`
to be removed as tidy up is executed as part of removing a flow.
Deleting a flow's checkpoint and releasing related soft locks is still
handled manually in case of infinite loops but also triggered as part
of the actions executed in a transition.

This required flow sleeping to be changed as we no longer rely on
quasar.

## How flows now sleep

The reliance on Quasar to make a flow sleep has been removed.

Instead, when a flow sleeps we create a `ScheduledFuture` that is
delayed for the requested sleep duration. When the future executes it
schedules a `WakeUpFromSleep` event that wakes up the flow... Duh.

`FlowSleepScheduler` handles the future logic. It also uses the same
scheduled thread pool that timed flows uses.

A future field was added to `StateMachineState`. This removes the 
need for concurrency control around flow sleeps as the code path does
not need to touch any concurrent data structures.

To achieve this:

- `StateMachineState.future` added as a `var`
- When the `ScheduledFuture` is created to wake up the flow the passed
in `StateMachineState` has its `future` value changed
- When resumed `future` and `isWaitingForFuture` are set to `null` and
`false` respectively
- When cancelling a sleeping flow, the `future` is cancelled and nulled
out. `isWaitingForFuture` is not changed since the flow is ending anyway
so really the value of the field is not important.
2020-04-28 15:53:44 +01:00
Xavier Leprêtre
9396ee9551 Make TransactionBuilder's input references deep copied on copy. 2020-04-25 22:50:09 +04:00
Dimos Raptis
963de40902
[NOTICK] - Enable check in detekt for unused imports (#6106)
* [NOTICK] - Enable check in detekt for unused imports

* Put back accidental removal of used import

* Some more accidental removals
2020-03-26 15:46:33 +00:00
Kyriakos Tharrouniatis
f6952963a8 CORDA-3677 FlowExternalOperation serialising reference to FlowLogic (#6094)
* Stop capturing 'FlowLogic' references in flowAsyncOperation;

Creating concrete classes removes the implicit reference to FlowLogic (as this) being included in the anonymous object

* Modify test code so that lambdas no longer get implicit references to their enclosing 'FlowLogic'
2020-03-26 10:00:57 +00:00
Kyriakos Tharrouniatis
caf152f175
CORDA-3677 FlowExternalOperation serialising reference to FlowLogic (#6094)
* Stop capturing 'FlowLogic' references in flowAsyncOperation;

Creating concrete classes removes the implicit reference to FlowLogic (as this) being included in the anonymous object

* Modify test code so that lambdas no longer get implicit references to their enclosing 'FlowLogic'
2020-03-26 09:23:38 +00:00
LankyDan
82e068be94 Merge branch 'release/os/4.4' into dan/os-4.4-to-4.5-merge-2020-03-23 2020-03-23 11:46:24 +00:00
Dan Newton
668748b054
CORDA-3669 Do not execute ExecuteAsyncOperation multiple times (#6087)
* CORDA-3669 Do not execute `ExecuteAsyncOperation` multiple times

When a `FlowExternalOperation` or `FlowExternalAsyncOperation` executes
and completes a flag (`isFlowResumed`) is switched to true.

This flag was used inside of `DoRemainingWorkTransition` to decide
whether to skip over the execution of an event.

Since this flag was being switched to true when the external operation's
 future completed, it was possible for _unexpected_ events to be placed
in the fiber's queue that would retrigger the
`FlowIORequest.ExecuteAsyncOperation`, that is held as the checkpoint's
next `FlowIORequest`to process.

By using the existing `StateMachineState.isTransactionTracked` (and
renaming it to `isWaitingForFuture`) we can decide to not process the
`FlowIORequest.ExecuteAsyncOperation` if it has already been called
before. This moves this code path in line with
`FlowIORequest.WaitForLedgerCommit`.

Random `DoRemainingWork` events can now be pushed to the fiber's queue
without causing the `FlowIORequest.ExecuteAsyncOperation` to execute
again.
2020-03-20 19:02:34 +00:00
Chris Rankin
dd7852f2b8
CORDA-3668: Prevent AttachmentURLConnection from assigning ALL_PERMISSIONS. (#6080)
* CORDA-3668: Prevent AttachmentURLConnection from assigning ALL_PERMISSIONS to classes inside an attachment.

* Strengthen the comment warning about AttachmentURLConnection.getPermission.
2020-03-19 11:20:53 +00:00
Dan Newton
fb64e47326
NOTICK Increase timeouts in external operation tests (#6076)
* NOTICK Increase timeouts in external operation tests

It seems these tests are timing out when they shouldn't be when running
on the CI servers.

To resolve this, the timeouts of `getOrThrow` calls have been increased
to a minute.

Reliance on timeouts has also been reduced through using locks. This
should speed up tests that don't need to wait for the timeout anymore to
 progress.

* NOTICK Remove hospital checking code

* NOTICK Change locking code
2020-03-18 13:56:51 +00:00
Chris Rankin
ffa2caed32 Merge branch 'release/os/4.4' into chrisr3-44-merge 2020-03-04 16:27:44 +00:00
Christian Sailer
e38cd9ec63
ENT-5043 separate out test utils (#5998)
* Remove unused dependencies from test-common

* Explicit imports and formatting

* Add core-test-utils project

* Add dependency

* Move Kryo serialization context to node-api (not serialization as we do not want to pull kryo into the serialization lib)

* Move AMQP server serialization scheme to node api

* Move serialization tests to node-api

* Move internal test helpers without further dependencies.

* Move out some types from RPCClientProxyHandler to node-api in preparation for moving the AMQP scheme

* Move client AMQP context to node-api so we can move the test serialization rule out.

* Move InternalSerializationTestHelpers to core-test-utils

* Moved testing.core to core-test-utils

* Make detekt happy

* Add api-scanner to core-test-utils

* Remove inlined package names introduced by IntelliJ refactoring

* Update api-current.txt to account for reordering.

* Add core-test-utils to list of published artifacts.

* Add missing import

* Location of things in api text has moved again (publish name of artefact?)

* Revert all additions to the API, leaving just the reordering

* Code review: fix up core-test-utils build.gradle and introduce kryo version constant.

* Remove OpenSsl flag from ssl config stub (can't be used from node-api)

* Suppress detekt warning

* Move core test util tests to the right module

* Expose kotlin test as a transient dependency - projects have come to rely on that.

* Fix typo in package name
2020-03-03 15:55:17 +00:00
Adel El-Beik
fe625d0f37
CORDA-3584: Now cope with 2 contract jars with same hash but different name (#5952)
* CORDA-3484: Now cope with 2 contract jars with same hash but different name, we just select one and use that.

* ENT-3584: Contract jars are now generated on the fly.

* CORDA-3584: Reverted changes to CordappProviderImpl. Exception is raised if node started with multiple jars with same hash.

* ENT-3584: Fixing test failure.

* CORDA-3584: Switch to test extension method instead of reflection to access internal member.

* ENT-3584: Address review comment. Dont fully qualify exception.

* CORDA-3584: Address review comment and converted lazy to a resettable one.

* CORDA-3584: Removed unused logger.

* CORDA-3584: Fixed visibility.

* CORDA-3584: Removed synchronized

* CORDA-3584: Removed CordappResolver

* CORDA-3584: Reverted change in gradle file and fixed test.

* CORDA-3584: Removed V3 from test description as it wasn't actually V3 specific.

* CORDA-3584: Address review comment. Let classes be garbage collected.
2020-02-25 11:18:01 +00:00
James Higgs
3547b629c3
[NOTICK] Add a detekt rule to catch tests with no timeout (#5959)
* [NOTICK] Add a custom detekt rule for tests with no timeout, and fix remaining missing timeouts

* [NOTICK] Add a test for custom detekt rules and tidying

* add timeout annotation to new test

Co-authored-by: Stefano Franz <roastario@gmail.com>
2020-02-14 17:31:02 +00:00
Razvan Codreanu
c6ab07d92c
TM-197 Fix failing test caused by regression and ignoring flaky DistributedServiceTests (#5934)
* TM-197 Setting bouncy castle provider in order for the test to pass

* TM-197 setting timeout for all builds at 3 hours

* TM-197 ignoring unstable tests
2020-02-12 14:26:06 +00:00
Razvan Codreanu
42eca48a02
TM-197 Build stability changes (#5947)
* TM-197 switching 4.3 to use local k8s instances and also make the maximum duration of builds 3 hours, fix 1 test and ignore 2 flaky ones

* update to use local-k8s version of the plugin

Co-authored-by: Stefano Franz <roastario@gmail.com>
2020-02-12 13:58:30 +00:00
Stefano Franz
b23af5f0d2
add timeouts to all tests (#5875) 2020-02-11 10:14:05 +00:00
Dan Newton
4bae045a58 ENT-4601 Public API to run external operations from a flow (#5833)
Deprecate FlowAsyncOperation and reimplement public versions FlowExternalOperation and FlowExternalAsyncOperation.

await added to FlowLogic to allow easy calling from both Java and Kotlin. There are two overrides of await (one for FlowExternalOperation and FlowExternalAsyncOperation).

Implementations of FlowExternalOperation return a result (written as blocking code) from their execute function. This operation will then be executed using a thread provided by the externalOperationExecutor.

Implementations of FlowExternalAsyncOperation return a future from their execute function. This operation must be executed on a newly spawned thread or one provided by a thread pool. It is up to developers to handle threading in this scenario.

The default thread pool (externalOperationExecutor) can be configured through the flowExternalOperationThreadPoolSize node config.

The current implementation leaves FlowAsyncOperation alone, meaning that any developers that have used it (even though it is internal) won't need to change their apps. If this was not concern I would delete it completely and replumb the state machine code. Instead, it has been marked with @DoNotImplement and executeAsync is annotated with @Deprecated
2020-01-22 09:27:17 +00:00
Stefano Franz
af30e40397 CORDA-3485 Restore CollectSignaturesFlow.kt to allow multiple collections from Well Known sessions (#5800)
* modify CollectSignaturesFlow.kt to allow multiple collections from wellKnown party initiated sessions

* detekt fixes

* review comments

* move require lambdas back outside of the function definition of requires

* address review comments

* fix detekt

* fix api scanner
2019-12-17 16:26:09 +00:00
Christian Sailer
14050826e9 ENT-4595 harmonize core and serialization (#5792)
* Harmonize serialization/core and deterministic counterparts

* Fix test for changed private alias key behaviour

* Detekt errors

* roll back project.xml
2019-12-09 14:17:48 +00:00
stefano
6a3a9eded3 Merge remote-tracking branch 'origin/release/os/4.3' into my_merge_branch
# Conflicts:
#	core/src/main/kotlin/net/corda/core/internal/TransactionVerifierServiceInternal.kt
#	node/src/integration-test/kotlin/net/corda/node/services/rpc/RpcExceptionHandlingTest.kt
#	testing/node-driver/src/main/kotlin/net/corda/testing/node/internal/DriverDSLImpl.kt
2019-11-05 17:19:18 +00:00
Chris Rankin
f226ddc4f2 CORDA-2876: Integrate the DJVM with the Corda Node. (#5633)
* * CORDA-2876: Migrate DJVM serialization modules into Corda.
* Pre-generate Corda classes for DJVM sandbox when node boots in production mode.
* Ensure that all DJVM test contract CorDapps are signed.
* Test examining attachments within DJVM sandbox.
* Test Contract.verify() using cryptographic verify function.
* Add test cases for more non-determinism in Contract.verify().
* Update node-driver to support testing nodes with DJVM support.
* Modify Node to allow alternative DJVM configurations for testing.
* Refactor DeterministicVerifierFactoryService for default use-case.
* Small whitespace and code-style refactors.
* Create and activate a DJVM execution profile for the Node.
* Revert making Verifier implement AutoCloseable.
* Allow the node to cache sandboxed Corda byte-code for reuse.
* Use updated Quasar agent that knows not to touch DJVM classloaders.
* Fix Quasar's package exclusions globs for DJVM.
* Deserialise LedgerTransaction into the sandbox for Contract.verify().
* Add the DJVM's serialisation modules to the Corda node.
* Update the node for the latest DJVM API, and preserve the ConstructorForDeserialization annotation on user contract classes.
* Add corda-dev to repositories while DJVM is SNAPSHOT.
* Migrate DJVM specialisation into AbstractNode's ServiceHubInternalImpl.
* Exclude sandbox.** and shaded djvm.** classes from Quasar agent.
* Add the corda-dev repository to :node for the deterministic runtime.
* Turn Verifier into an abstract base class that is specialised by BasicVerifier and DeterministicVerifier.
* Add the Corda deterministic libraries to the Node, and split the DJVM sandbox across two SandboxClassLoader instances.
* Add DJVM to contract verification path inside Corda Node.
* Minor lambda simplifications and removing unused import.
* CORDA-2871: Remove @CordaSerializable from LedgerTransaction.
* CORDA-2871: Add a callback to ServicesForResolution to allow the Node to modify a LedgerTransaction object.
* CORDA-2871: Refactor the contract verification code into a separate class,
 and allow LedgerTransaction to choose different Verifier objects.
* Update DJVM to use Corda 4.4-SNAPSHOT. (#95)
* CORDA-3330: Allow DJVM to preload / pregenerate classes from selected jars. (#92)
* Add support for SourceClassLoader.getResources() to DJVM.
* Allow a SandboxConfiguration to preload sandbox byte-code for all classes inside jars containing META-INF/DJVM-preload.
* CORDA-3309: Remove explicit try-catch in favour of UncaughtExceptionHandler. (#91)
* CORDA-3309: Install UncaughtExceptionHandler for DJVM tasks. (#88)
* Fix tests broken by Windows line endings. (#82)
* CORDA-3292: Reimplement ExecutionProfile as a data class. (#80)
* CORDA-2877: Refactor how we create child SandboxConfiguration objects. (#76)
* CORDA-2877: Load bytecode from a persistent cache to prevent repeated rewriting. (#75)
* Refactor byte-code cache to SandboxConfiguration instead of AnalysisConfiguration. We cannot "mix and match" byte-code generated by different sets of rules.
* CORDA-3137: Enhance annotation handling so that we can allow some annotations to be mapped into the sandbox without also needing to be stitched. (#72)
* CORDA-2871: Minor cosmetic fixes. (#69)
* CORDA-3218: Align DJVM with internal Corda Serialisation API. (#68)
* Ensure we get the latest SNAPSHOT of the serialisation code.
* CORDA-2871: Refactor SourceClassLoader to define source classes. (#66)
* Rewrite SourceClassLoader to support parent/child relationships.
* Revert catching TypNotPresebtException - it was a symptom of a bigger problem.
* Remove AutoCloseable from AnalysisConfiguration and SourceClassLoader.
* SourceClassLoader.getResource() must delegate to its parent first.
* CORDA-2871: Ensure ClassLoader.loadClass() throws ClassNotFoundException for all cases where the class cannot be found. (#64)
* CORDA-2871: Modify sandbox tasks to implement both java.Function and sandbox.Function (#62)
* Make TaskExecutors implement BiFunction to make them composable.
* Create ImportTask to wrap a java.Function inside a sandbox.Function.
* Add createExecutor() and createRawExecutor() APIs to SandboxClassLoader.
* Update serialization to use SandboxClassLoader.toSandboxClass().
* Remove a layer of lambdas from the serialisation code.
* Update SandboxExecutor and SandboxRawExecutor.
* Rename Executor to TaskFactory.
* Rename dangling executor -> taskFactory.
* CORDA-2871: Sanity fixes! (#63)
* Improve message for SandboxClassLoadingException.
* Fix serialisation API for using sandboxed environment.
* CORDA-3174: Extend serialisation to include InputStream and OpaqueBytesSubSequence. (#60)
* Update DJVM Example project for serialisation.
* Add serializers for InputStream and OpaqueBytesSubSequence.
* Support ZIP Inflater and CRC32 inside the sandbox.
* Allow the DJVM to wrap java.io.InputStream as sandbox.java.io.InputStream.
* Configure tests also to preserve @DeprecatedConstructorForDeserialization.
* CORDA-3174: Implement Corda serialization modules. (#59)
* Create DJVM serialization modules.
* Create test cases for Array<T>, List<T> and List<Array<T>>.
* Refactor SandboxPrimiveSerializer for all primitive types.
* Implement SandboxCollectionSerializer to support Collection types.
* Implement SandboxMapSerializer to support Map types.
* Attempt to fix infinite loop when computing Collection and Map fingerprints.
* Apply special handling when deserialising sandbox.java.lang.Character.
* Remap Java primitive types to sandbox Java object types to deter evolution.
* Use Class.getPackage().getName() to determine sandbox package name.
* Implement SandboxEnumSerializer to support Enum types.
* Implement SandboxPublicKeySerializer to support Java security keys.
* Add serialization projects to the composite example project.
* Implement serializers for BigInteger, BigDecimal, Currency and StringBuffer.
* Test that deserialising does not instantiate the untrusted user classes.
* Implement serializers for java.time.* types.
* Add serialiser for BitSet - currently disabled until BitSet itself is supported.
* Add serialisers for EnumSet and Class.
* Include support for EnumMap in the SandboxMapSerializer.
* Ensure the DJVM Example project's tests preserve @CordaSerializable.
* Add support for UUID as a primitive type.
* Use common abortReadOnly() method for declaring serialization as unsupported.
* Streamline the API for deserialising into the sandbox.
* Add preliminary support for deserialising X.509 certificates.
* Implement serializer for java.util.Optional.
* Refactor configuration of the sandbox serialization scheme.
* Add tests for deserialising arrays of basic types.
* Include method annotations in annotation stitching. This ensures that `@ConstructorForDeserialization` is not dropped.
* Enable test for SandboxBitSetSerializer.
* Enable tests for X.509 serializers.
* Implement serializers for ProtonJ primitive types.
* Serialize java.util.Date as a primitive type.
* Add the bintray Gradle plugin to the serialisation modules.
* Do not publish serialisation modules - they will become part of Corda itself.

* CORDA-2876: Only apply DJVM sources to Node Driver when devMode=true.

* Resolve DeteKT warnings.

* Require Node's JVM to set -Dnet.corda.djvm=true in order to enable DJVM.

* Enable DJVM for DemoBench nodes.

* Disable Quasar instrumentation verification for DemoBench nodes.

* Upgrade to DJVM 1.0-RC01.

* Try to modify DriverParameters in a more "ABI friendly" way.

* Refactor and simplify sandbox deserialisation of primitive objects.

* Review fixes.

* Update EvolutionSerializerFactory to handle sandboxed primitive boxed types.
2019-11-05 13:44:18 +00:00
Dimos Raptis
485feb2d6c CORDA-3350: Increase size of constraints column (#5639)
* CORDA-3350: Increase size of constraints column (#5639)

* Detekt

* Update api file with new threshold

* Add check in transaction builder

* Revert "Add check in transaction builder"

This reverts commit ca3128f44c.

* Add check for max number of keys

* Update api file

* Address Tudor's comments

* Remove check for pre-5 and add test for EC keys

* fix typo and rename liquibase script

* updated docs with measurement numbers for composite keys

* Make detekt happy again
2019-11-05 11:00:26 +00:00
Chris Rankin
346410a2e1 CORDA-3327: Fix API scanning for Java 11. (#5645) 2019-10-30 17:24:34 +00:00
Tudor Malene
712b8b7719 CORDA-2965 Make Tx verification exceptions serializable. (#5560) 2019-10-09 11:43:06 +01:00
josecoll
3fafbe551c
CORDA-2050 Upgrade Corda to Java 11 (compatibility mode) (#5356)
Upgrade Corda to run with Java 11 (compatibility mode) - see https://github.com/corda/corda/pull/5356
2019-09-18 10:26:26 +01:00
Roger Willis
1c2c3d3fed
Identity service refactor for confidential-identities and accounts (#5434)
* Removed IdentityServiceInternal as it is no longer used.

* Removed externalIdForPublicKey API from KMS and added it to IdentityService.
Added a registerKeyToExternalId API on IdentityService.

* Fix remaining compile errors.

* Removed "registerKeyToParty" and in its place added a new registerKey method which takes a PublicKey, Party and optionally a UUID.
Added a cache to the "PersistentIdentityService" to store other node's public keys.
Added the cache and new hibernate entity to all teh places where one needs to add them.
New keys created by teh node now automatically get associated entries in the KEY -> PARTY map and optionally the KEy -> EXT ID map.
Added a test.

* Removed old comments and TODOs.

* Fixed broken test.
Added comments/explanations for what's going on in IdentityService.
Updated kdocs.

* First try at Implementing publicKeysForExternalId.

* Fixed broken test.

* Added migration.
Amended existing persistent identity service migration to handle new migration.
Addressed some review comments.

* Fixed broken test - whoops!

* Implemented mock identity service methods.

* Added back exception when remapping a key to a different party.

* Fixed compile errors.
Fixed broken tests.

* Use set instead of first entry in ourNames.
2019-09-16 13:14:12 +01:00
Zoltan Kiss
f171de7b69 TM-23 Fail build on compiler warnings (#5453)
* java compile respects compilation.allWarningsAsErrors

* suppress or cleanup warnings

* suppress warning

* use non-deprecated kotlin dependency

* rename property

* handle property existence check

* Deal with warnings
2019-09-11 16:34:51 +01:00
Dan Newton
4cbe22949d CORDA-3018 Whitelisting attachments by public key - phase two tooling (#5386)
Allow node operators to blacklist signing keys (using blacklistedAttachmentSigningKeys config option). These blacklisted keys prevent attachments that are received over the network from being trusted. The docs have been updated to detail how to generate the key hashes that the config requires.

A new shell command attachments trustRoots has been added to see what attachments exist on the node along with information about their trust and where it comes from.

run dumpCheckpoints has been replaced by checkpoints dump as InternalCordaRPCOps needed to change to prevent a function that is meant to be internal from being visible on the shell.
2019-09-10 12:16:34 +01:00
Nick Dunstone
9d6922d11e CORDA-3177 Additional Back Chain Resolution performance enhancements (#5415)
* Applied backchain fetch optimisations

* Removed test for bulk fetch

* Removed import

* Removed extraneous import

* Review comment
2019-09-05 14:27:42 +01:00
willhr3
07b96aea18 CORDA-2925 Rebase identity service changes onto 4.3 (#5407)
* CORDA-2925 Rebase identity service changes onto 4.3

* CORDA-2925 Move migration to after v13

* CORDA-2925 Update schema list

* Change corda-version
2019-08-29 11:57:07 +01:00
Stefano Franz
e35c0c1df7
CORDA-3009 - Migrate identity service to use to string short (#5217)
* migrate PersistentIdentityService to use key.toShortString()
update definition of PublicKeyToExternalId mapping to allow fast lookup by externalId/publicKey

* fix misspelled table name

* add test of migration script

* add design document for proposal to move IdentityService to using the correct PK.toStringShort() method for hashing a publickey

* add enterprise testing considerations to design

* address review comments

* fix compilation errors

* modify PublicKeyToOwningIdentityCache to use toStringShort() as it's lookup key

* address syzmon's code review comments
2019-08-28 14:16:18 +00:00
Shams Asari
69d1b4e4e5
CORDA-3138: Removed InMemoryTransactionsResolver as it's not needed and other resolution cleanup (#5370) 2019-08-16 12:34:25 +01:00
Dan Newton
fc265ee472 CORDA-3018 Whitelisting attachments by public key - relax signer restrictions (#5358)
* CORDA-3018 Allow loading an untrusted contract jar if another attachment exists that was signed with the same keys and uploaded by a trusted uploader

`TransactionUtils.isAttachmentTrusted` requirements have been relaxed
to allow an untrusted attachment to be loaded as long as another
attachment exists that is signed by the same keys and was uploaded
by a trusted uploader.

The requirement of containing the same contract classes has been
removed. Therefore the contents of the existing trusted attachment
no longer matters.

* CORDA-3018 Allow a subset/intersection of signers in `isAttachmentTrusted`

Allow a subset/intersection of signers to satisfy the signer
requirements of `isAttachmentTrusted`. This allows an "untrusted"
attachment that is signed by one or more keys to be "trusted" as long
as another trusted attachment already exists that is signed by at least
one of the "untrusted" attachments signers.

A cache of trusted and untrusted public keys is now held (replacing the
previous cache of `List<PublicKey>`.

Tests have been added to `NodeAttachmentServiceTest` to confirm that
an attachment query using an `EQUAL` statement will actually return
attachments that are signed by any of the keys passed into the query.

Confirming this allowed an `EQUAL` query to satisfy the search that
had to be done as part of this change.

`MockAttachmentStorage`'s query criteria was updated to better match
the real `NodeAttachmentService` implementation.

* CORDA-3018 Update cache name and kdoc on `isAttachmentTrusted`

* CORDA-3018 Verify that chains of trust do not occur

* CORDA-3018 Switch keys around to improve chain of trust tests
2019-08-08 09:33:45 +01:00
James Higgs
44428b6048 ENT-3801 Backport to OS (#5355)
* [ENT-3801] Store transactions in the database during transaction resolution (#2305)

* ENT-3801: Store downloaded txns as part of the backchain resolution into the db rather than the checkpoint

It's very inefficient to store the downloaded backchain in the checkpoint as more of it downloaded. Instead, if a threshold is reached (which currently defaults at 0) then the backchain is stored in the transactions table as unverified. A new is_verified column has been added to track this. Initially testing on the OS codebase has been very promising but unfortunately this current code is not quite ready. I had to quickly port it to ENT as this is meant to be an ENT-only optimisation.

To that effect, there is a TransactionResolver abstraction with two implementations: an in-memory one which has the old behaviour, and which will be the behaviour for OS, and a db one.

DBTransactionStorage hasn't been fully updated and I had to comment out the optimistic path for now.

Most of these changes will need to be ported to OS to keep the merge conflicts in check, but obviously not DbTransactionsResolver and the "is_verified" changes in DBTransactionStorage. DBTransactionStorage does have other refactoring which will make sense to port though.

* [ENT-3801] Start work on allowing modifications in AppendOnlyPersistentMap

* [ENT-3801] Add transaction resolver tests

* [ENT-3801] Adjust suspendable annotations

* [ENT-3801] Fix the ResolveTransactionFlow tests

* [ENT-3801] Update ResolveTransactionsFlow tests

* [ENT-3801] Add a liquibase migration script for isVerified

* [ENT-3801] Ensure the migration runs in the correct place

* [ENT-3801] Handle resolution of already present transactions

* [ENT-3801] Fix compile error in performance test app

* [ENT-3801] Logging and comment updates, plus a test case

* [ENT-3801] Add a notary change resolution test

* [ENT-3801] Add a contract upgrade transaction test

* [ENT-3801] Change new column to be a character based status

* [ENT-3801] Migration script type change

* [ENT-3801] Address first round of review comments

* [ENT-3801] Update variable names in AppendOnlyPersistentMap

* [ENT-3801] Another variable name clarification

* [ENT-3801] Fix missing name changes

* [ENT-3801] Make the signature list immutable when constructing cache value

* [ENT-3801] Add a locking strategy for unverified transactions

* [ENT-3801] Address tidying up review comments

* [ENT-3801] First attempt at ensuring locks are released after commit

* [ENT-3801] Remove references to old cache name

* [ENT-3801] Update locking logic

* [ENT-3801] Fix potential deadlock with read/write transaction locks

* [ENT-3801] Remove read locks, and ensure minimal extra suspends

* [ENT-3801] Fix build issues in tests

* [ENT-3801] Use the correct clock when calculating sleep durations

* [ENT-3801] Add a pessimism flag for writing verified transactions

* [ENT-3801] Change logging statement to debug

(cherry picked from commit 8ab6a55e17)

* [NOTICK] Fix up imports for some changed files

* [NOTICK] Fix transaction resolution tests

* [NOTICK] Reinstate the DBTransactionsResolver

* [NOTICK] Add the topological sort back to recordTransactions

* [NOTICK] Adjust test case to remove dependency on query ordering

* [NOTICK] Make test code match that in ENT
2019-08-07 16:49:49 +01:00
Chris Rankin
813ff39eaa CORDA-3127: Move evaluationDependsOn() from core to core-tests. (#5349)
* CORDA-3127: Move evaluationDependsOn() from core to core-tests.

* Increase JVM memory for node unit tests to 3G.
2019-08-02 09:05:19 +01:00
LankyDan
9bf26c20e0 CORDA-2920 Hash to Signature Constraint automatic propagation
Allow Hash Constraints to propagate to Signature Constraints. When
propagating, the new jar is added to the transaction instead of the
original contract attachment (jar).

The following requirements must be met to do so:
- System property "net.corda.node.disableHashConstraints" must be set
  to `true`
- The min platform version must be 4 or greater
- There must be an input state with a hash constraint
- There must be no output states with a hash constraint
- The new jar must be signed

If these requirements are not met, the original Hash
Constraint will be kept and the contract attachment related to it is
also used.

This transition is done at the start of `handleContract` as it is not
the normal path a transition would follow. It is considered a backdoor
and should be treated separately from the rest of the attachment and
constraint logic. Furthermore, it will only work in private network
since all nodes must set the special `disableHashConstraints` flag.
2019-07-29 12:58:58 +02:00
Christian Sailer
687fa81894 ETO-39 Move core tests (#5255)
* Moved core.flows and core.contract tests

* move internal tests

* move node, schemas and serialization tests

* Move transactions tests

* Move test access helper into test package

* Move remaining tests, and create access helpers to test internals.

* Fix test dependencies.

* Delete some cruft that flagged up on the API checker

* Add readme explaing where tests need to go.

* fix typo
2019-07-09 10:09:21 +01:00