Commit Graph

3100 Commits

Author SHA1 Message Date
Chris Rankin
758a69f904
Validate LedgerTransaction deserialised from AttachmentsClassLoader. (#7049) (#7052) 2022-01-31 09:40:01 +00:00
Adel El-Beik
7752fc8c9d
CORDA-4189: Removed existing vendor test (#7048) 2022-01-27 13:49:08 +00:00
Adel El-Beik
fb624a2b06
CORDA-4189: Now compare against Corda Community Edition as well. (#7046)
* CORDA-4189: Now compare against Corda Community Edition as well.

* CORDA-4189: Removed unused imports.
2022-01-26 11:33:23 +00:00
Ramzi El-Yafi
831f6d60cc Merge remote-tracking branch 'origin/release/os/4.8' into ramzi/os-48-49-merge 2022-01-21 10:55:29 +00:00
Viktor Kolomeyko
835321bb70
ENT-6286: OS: Upgrade version of Artemis to 2.19.1 (#6975)
Fixes DDoS attack mentioned on the Jira ticket.

PR upgrades Artemis library to version 2.19.1.
This is our own release of Apache Artemis library which has vulnerability fix for v2.20 applied.

**_Breaking changes discovered during Artemis upgrade:_**
1. When the queue is created as temporary, it needs to explicitly be specified as non-durable.
2. By default, Artemis Client performs Host DNS name check against the certificate presented by the server. Our TLS certificates fail this check and this verification has to be explicitly disabled, see use of: `TransportConstants.VERIFY_HOST_PROP_NAME`.
3. Artemis Server now caches login attempts, even unsuccessful ones. When we add RPC users dynamically via DB insert this may have an unexpected outcome if the user with the same `userName` and `password` was not available previously.
To workaround permissions changing dynamically, authorization and authentication caches had to be disabled.
4. When computing `maxMessageSize`, the size of the headers content is now taken into account as well.
5. Artemis handling of start-up errors has changed. E.g. when the port is already bound.
6. A number of deprecated APIs like: `createTemporaryQueue`, `failoverOnInitialAttempt`, `NullOutputStream`, `CoreQueueConfiguration`.
7. Log warning message is produced like: `AMQ212080: Using legacy SSL store provider value: JKS. Please use either 'keyStoreType' or 'trustStoreType' instead as appropriate.`
8. As reported by QA, Artemis now produces more audit logging more details [here](https://r3-cev.atlassian.net/browse/ENT-6540). Log configuration been adjusted to reduce such output.
2022-01-21 09:18:14 +00:00
Adel El-Beik
b17e4571bf
ENT-6386: Revert change of behaviour in rpcOps.wellKnownPartyFromX500Name for revoked identity (#7032)
* ENT-6386: Reverting wellKnownPartyFromX500Name functionality to work … (#4347)
2022-01-20 11:08:13 +00:00
Dan Newton
c05c1934cf
ENT-6547 Remove url classloader in shell loading (#7034)
The `corda-shell` jar will already be installed if it exists in the the
node's `/drivers` directory. There is no need to include a
`URLClassLoader` to load its classes.

Rely on the process's main classloader.
2022-01-19 18:26:26 +00:00
Ramzi El-Yafi
caa4de47d3
ENT-6533 Update validating notary flow to be non-idempotent (#7030) 2022-01-19 13:15:51 +00:00
Lorcan Wogan
c2140f727c
ENT-6025 remote artemis channel does not exist resulting in infinite retry loop (#7020)
* ENT-6025 remote artemis channel does not exist resulting in infinite retry loop

* ENT-6025 rename test

* ENT-6025 fix detekt and add description

* ENT-6025 add check on count of connected stack
2022-01-18 10:13:29 +00:00
Chris Rankin
552317f2b3 Merge branch 'release/os/4.8' into chrisr3-os48-merge 2022-01-17 11:02:11 +00:00
Chris Rankin
4f1a07cbcc
ENT-6548: Ensure LazyMappedList is realised with correct SerializationContext. (#7028) 2022-01-17 09:57:33 +00:00
Dan Newton
bd9b50bf7c ENT-6376 Don't hospitalize session end in ReceiveFinalityFlow (#4325)
Do not keep a flow in for observation if it receives an unexpected
session end message while in `ReceiveFinalityFlow` and
`ReceiveTransactionFlow` (due to being called by the former).

This is done by checking the message of the `UnexpectedFlowEndException`
that is thrown when a session end message instead of a data message and
if the stacktrace has `ReceiveTransactionFlow` at the top, after
removing statemachine stack frames.

Checking the stacktrace for `ReceiveTransactionFlow` is important
because the unexpected session end session message is only ok if a
transaction has not already been received. For example, if
`ResolveTransactionsFlow` is in the stack, then this indicates failure
when receiving transaction dependencies on a transaction that should be
recorded.

Also added a test that highlights that the `UnexpectedFlowEndException`
caused by the session end message can be caught, therefore users can
determine their own behaviour if desired.
2022-01-13 11:02:46 +00:00
Dan Newton
56c9d6404f
ENT-6378 Migrate corda-shell to external repo (#7005)
Remove the shell code from the OS code base, this includes the modules:

- `:tools:shell`
- `:tools:shell-cli`

The shell will be run within a node if it exists within the node's `drivers` directory. 

This is done by using a `URLClassloader` to load the `InteractiveShell` class into Corda's JVM process and running `startShell` and `runLocalShell`.

Running the shell within the `:samples` will require adding:

```
cordaDriver "net.corda:corda-shell:<corda_shell_version>"
```

To the module's `build.gradle` containing `deployNodes`. The script will then include the shell in the created nodes.
2022-01-12 11:54:18 +00:00
Dan Newton
f2b3db9c7b
ENT-6315 Allow dumping of paused flows (#7008)
Checkpoint dumping of paused flows was not working because the dumper
expects a flow to have a `FlowState` of `Unstarted` or `Started`,
however due to a memory optimisation paused flows have their `FlowState`
set to `Paused`. This was causing causing an exception as well as a loss
of potentially useful information.

A flag `alwaysDeserializeCheckpoint` has been added to
`Checkpoint.Serialized.deserialize` which skips the memory optimisation
and forces the deserialization of the flow's `FlowState`.

Paused flows are now included in the dumped output along with their real
`FlowState` which is useful to users even if the flow is paused rather
than waiting for something to complete.

The status of the flow has also been added to the JSON output to assist
users in debugging their flows.
2022-01-11 10:22:49 +00:00
Dan Newton
044202550d
ENT-6421 Public version of FlowManagerRPCOps (#7006)
A public version of `FlowManagerRPCOps` which does not live in an
internal package has been added. This new interface shares the same name
as the internal one.

Because of the name sharing, the internal version has been
`@Deprecated`.

`FlowManagerRPCOpsImpl` implements both the new and old interfaces. This
 allows for backwards compatibility, allowing old shells or clients to
 call the old interface on newer nodes without breaking.
2022-01-07 16:40:29 +00:00
Viktor Kolomeyko
505ade9c86
ENT-6388: Eliminate usage of Mina's ConcurrentHashSet (#6977)
* ENT-6388: Eliminate usage of Mina's `ConcurrentHashSet`

* ENT-6388: Restrict to `MutableSet<ScheduledStateRef>`
2021-12-02 13:24:11 +00:00
Kyriakos Tharrouniatis
3cbfb0e024
ENT-6355 Empty flow arguments on error transition (#4307)
A user passed in a `FlowLogic` as an argument into another `FlowLogic`
called `subFlow` on it and had it throw an exception.

This all occurred before the first checkpoint, causing the state machine
to try and persist a FAILED checkpoint containing the flow's arguments.
Because the arguments contained a `FlowLogic` that had been started via
`subFlow` it held a reference to `FlowLogic._stateMachine` which cannot
be serialized.

This caused the flow to fail when trying to persist the fact that it
failed.

The flow arguments are now emptied during `ErrorFlowTransition` to
resolve this issue which mimics the behaviour of the first suspend.
Note, this only takes the arguments out of the serialized checkpoint, it
does not affect the flow metadata and therefore a flow's arguments can
still be viewed.

Co-authored-by: Dan Newton <dan.newton@r3.com>
2021-12-01 14:57:30 +00:00
Chris Rankin
e93e7c2846
ENT-6412: Upgrade ClassGraph 4.8.90 -> 4.8.135. (#6973) 2021-11-23 09:43:41 +00:00
Chris Rankin
caeafb8201
ENT-6357: Deserialize LedgerTransaction elements for each Contract.verify(). (#6962)
* ENT-6357: Deserialize LedgerTransaction elements for each Contract.verify().

* Lock the LedgerTransaction and NetworkParameters objects down for contract verification.

* Refactor BasicVerifier to be package private instead of public.

* Simplify verifyConstraints() operation.

* Review fixes: replace HashSet with LinkedHashSet, and add signing parties to commands via mapIndexed.

* Ensure tests also run notary nodes "out of process".

* Streamline SerializationContext switching.

* Cache deserialised cryptographic instances during contract verification.

* Invoke Class.forName() instead of ClassLoader.loadClass() to reduce contention on the system classloader's lock.

* Deserialization cache key now pre-computes its hash code.

* Allow AttachmentsClassLoader to be used concurrently.

* Cache all Envelope objects for reuse during contract verification.

* Generate CertPathProxy hash code using conventional algorithm.

* Adjust CustomSerializer.Proxy to allow better access to SerializationContext.
2021-11-10 16:38:40 +00:00
Walter Oggioni
883e794853
ENT-6330 Fixed reading jar entries in memory (#6960)
* ENT-6330 Fixed reading jar entries in memory

This is a trivial fix that is however enough to allow to send zip bombs as attachments without the node crashing, a size limit could be added for increased reliability

* added attachment cumulative size check

* added compression ratio check

* added unit test and moved the code to a standalone verifier object

* removed attachment check from AttachmentClassLoader to minimize performance impact
2021-10-13 09:46:20 +01:00
Agnieszka Szczepanska
0b9513ffa3 Merge branch 'release/os/4.7' into aga/make-recordDependencies-suspendable-4.8 2021-04-27 15:08:13 +01:00
Agnieszka Szczepanska
9aa4389902 resolve merge conflicts 2021-04-26 11:35:17 +01:00
Agnieszka Szczepanska
00c43e501c delete duplicated imports 2021-04-21 12:31:03 +02:00
Agnieszka Szczepanska
88f584cac5 fast forward merge 2021-04-21 12:16:22 +02:00
Agnieszka Szczepanska
c48f32215c Merge branch 'release/os/4.4' into aga/make-recordDependencies-suspendable-4.5 2021-04-20 16:25:47 +02:00
Agnieszka Szczepanska
8dd9dee9c4
NOTICK - make recordDependencies suspendable (#6896)
* add suspendable annotation to recordDependencies

* add suspendable annotation to recordDependencies implementation
2021-04-14 08:30:53 +01:00
Stefan Iliev
7b9b2569a9
NAAS-107 StartupQueueTime metric fix (#6886)
* Fixed StartupQueueTime metric producing negative numbers.
2021-03-09 18:00:04 +00:00
Ramzi El-Yafi
fe8baf4b17
NAAS-49 Flag notary/extraConfig block as sensitive (#6880) 2021-02-19 17:05:23 +00:00
Adel El-Beik
9cd02dc62d Merge remote-tracking branch 'origin/release/os/4.4' into adel/merge-from-4.4-19-Feb-21 2021-02-19 13:55:04 +00:00
Denis Rekalov
23fab7ae0c
ENT-5891 Disable flaky test (#6879) 2021-02-19 13:54:34 +00:00
Adel El-Beik
360b3f8d0c
CORDA-4125: Backport AttachmentClassLoader updates to 4.4 to enable ENT-6152 backport (#6878)
* CORDA-3755: Switched attachments map to a WeakHashMap (#6214)

* Bump OS release version 4.6

* CORDA-3755: Switched attachments map to a WeakHashMap

* CORDA-3755: Added explicit strong references to map key.

* CORDA-3755: Keeping detekt happy.

* CORDA-3755: Test a gc in verify.

* CORDA-3755: Making detekt happy.

* CORDA-3755: Suppress warnings for weak reference test.

* CORDA-3755: Fixing build failure with attachments.

* CORDA-3755: Rewrite based on Ricks input - now handles attachment already existing in map!

* CORDA-3755: Refactor WeakReference behaviour into AttachmentsHolderImpl and provide alternate version of this class for core-deterministic.

* CORDA-3755: Added more tests for WeakHashMap.

* CORDA-3755: Ignore the tests using System.gc keep for local testing only

* CORDA-3755: Adding comment to explain the ignored tests.

* Make AttachmentsHolderImpl package-private inside core-deterministic, just like it is inside core.

* CORDA-3755: Update assertions following review comments.

* CORDA-3755: Removing import

* CORDA-3755: Removed unused var.

* CORDA-3755: Reverting files that somehow got changed in rebase.

Co-authored-by: nargas-ritu <ritu.gupta@r3.com>
Co-authored-by: Chris Rankin <chris.rankin@r3.com>

* CORDA-3769: Switched attachments class loader cache to use caffeine (#6326)

* CORDA-3769: Switched attachments class loader cache to use caffeine with original implementation used by determinstic core.

* CORDA-3769: Removed default ctor arguments.

* CORDA-3769: Switched mapping function to Function type to avoid synthetic method being generated.

* CORDA-3769: Now using a cache created from NamedCacheFactory for the attachments class loader cache.

* CORDA-3769: Making detekt happy.

* CORDA-3769: The finality tests now check for UntrustedAttachmentsException which will actually happen in reality.

* CORDA-3769: Refactored after review comments.

* CORDA-3769: Removed the AttachmentsClassLoaderSimpleCacheImpl as DJVM does not need it. Also updated due to review comments.

* CORDA-3769: Removed the generic parameters from AttachmentsClassLoader.

* CORDA-3769: Removed unused imports.

* CORDA-3769: Updates from review comments.

* CORDA-3769: Updated following review comments. MigrationServicesForResolution now uses cache factory. Ctor updated for AttachmentsClassLoaderSimpleCacheImpl.

* CORDA-3769: Reduced max class loader cache size

* CORDA-3769: Fixed the attachments class loader cache size to a fixed default

* CORDA-3769: Switched attachments class loader size to be reduced by fixed value.

* CORDA-4125: Parameter has been added to a private ctor.

Co-authored-by: nargas-ritu <ritu.gupta@r3.com>
Co-authored-by: Chris Rankin <chris.rankin@r3.com>
2021-02-19 13:38:22 +00:00
Dan Newton
41b4f9b3aa
CORDA-4110 startFlowDynamicWithClientId restrict permissions (#6875)
Restrict permissions for `startFlowDynamicWithClientId` to not work with
 `InvokeRpc.startFlow` and instead require `InvokeRpc
 .startFlowWithClientId`.
2021-02-18 14:46:09 +00:00
William Vigor
7261442c98
CORDA-4123 Change MockNetworkParameterStorage to not rehash NetworkParameters (#6872)
* CORDA-4123 Change Mock Network Parameter Storage to not rehash the Network Parameters

* Fix Detekt
2021-02-16 16:25:38 +00:00
Adel El-Beik
1a7401472f
CORDA-4120: Disable the URLConnection cache to prevent file handle leaks in ServiceLoader. (#6869) 2021-02-12 17:24:00 +00:00
William Vigor
20dbbf008d
CORDA-4103 Feature Branch: Serialization injection for transaction building (#6867)
* CORDA-4105 Add public API to allow custom serialization schemes (#6848)

* CORDA-4105 Add public API to allow custom serialization schemes

* Fix Detekt

* Suppress warning

* Fix usused import

* Improve API to use generics

This does not break Java support (only Intelij gets confused).

* Add more detailed documentation to public interfaces

* Change internal variable name after rename public API

* Update Public API to use ByteSquence instead of SerializedBytes

* Remove unused import

* Fix whitespace.

* Add added public API to .ci/api-current.txt

* Improve public interfaces

Rename CustomSchemeContext to SerializationSchemeContext to improve
clarity and move to it's own file. Improve kdoc to make things less
confusing.

* Update API current with changed API

* CORDA-4104 Implement custom serialization scheme discovery mechanism (#6854)

* CORDA-4104 Implement CustomSerializationScheme Discovery Mechanism

Discovers a single CustomSerializationScheme implementation inside
the drivers dir using a system property.

* Started MockNetwork test

* Add driver test of Custom Serialization Scheme

* Fix detekt and minor style error

* Respond to review comments

Allow non-single arg constructors (there must be one no args
constructor), move code from SerializationEnviroment into its
own file, improve exceptions to be more user friendly.

* Fix minor bug in Scheme finding code  + improve error messages

* CORDA-4104 Improve test coverage of custom serialization scheme discovery (#6855)

* CORDA-4104 Add test of classloader scanning for CustomSerializationSchemes

* Fix Detekt

* NOTICK Clarify KDOC on SerializationSchemeContext (#6865)

* CORDA-4111 Change Component Group Serialization to use contex when the lazy map is constructed (#6856)

Currently the component group will recheck the thread local (global)
serialization context when component groups are serialized lazily.
Instead store the serialization context when the lazy map is constructed
and use that latter when doing serialization lazily.

* CORDA-4106 Test wire transaction can still be written to the ledger (#6860)

* Add test that writes transaction to the Database

* Improve test check serialization scheme in test body

* CORDA-4119 Minor changes to serialisation injection for transaction building (#6868)

* CORDA-4119 Minor changes to serialisation injection for transaction building

Scan the CorDapp classloader instead of the drivers classloader.
Add properties map to CustomSerialiaztionContext (copied from SerializationContext).
Change API to let a user pass in the serialization context in TransactionBuilder.toLedgerTransaction

* Improve KDOC + fix shawdowing issue in CordaUtils

* Pass only the properties map into theTransactionBuilder.toWireTransaction

Not the entire serializationContext

* Revert change to CordaUtils

* Improve KDOC explain pitfalls of setting properties
2021-02-11 15:27:03 +00:00
Dan Newton
cef5ea40ca Merge branch 'release/os/4.7' into dan/merge-os-4.7-to-4.8-2021-02-08 2021-02-08 14:38:17 +00:00
Dan Newton
26d6ccdc25 CORDA-4110 startFlowDynamicWithClientId permissions 2021-02-08 11:34:25 +00:00
Dan Newton
d807082cee Merge branch 'release/os/4.6' into dan/os-4.6-to-4.7-merge-2021-02-05
# Conflicts:
#	node/src/main/kotlin/net/corda/node/internal/rpc/proxies/AuthenticatedRpcOpsProxy.kt
2021-02-05 16:13:28 +00:00
Dan Newton
4e437ace2a
CORDA-4110 startFlowDynamicWithClientId permissions (#6857) 2021-02-05 16:05:55 +00:00
Dan Newton
88172b630d ENT-6142 Use ArrayList for SessionState structures (#4169)
Prevent some serialization errors that occur due to serialization
and deserialization of `ArrayList$SubList` found inside the
`SessionState` data structures.

To prevent this, an explicit `ArrayList` is used rather than a `List`.

Overload the `List` operator functions so that `+` returns an
`ArrayList` instead of a `List`.

Create `toArrayList` for a few conversions.
2021-01-29 11:15:28 +00:00
Dan Newton
c79ad972d0 ENT-6142 Flows become dead due to uncaught exceptions (#4158)
If a flow fails outside of its normal error processing code path it will
end up in `FlowDefaultUncaughtExceptionHandler`.

This handler will put the flow into overnight observation if possible.
This is done in-memory and the database.

Even with this being done, the fiber itself has blown up and therefore
does not manage to get to `SMM.removeFlow` which is where
`SMM.decrementLiveFibers` is called. For example, a flow that errored
will hit this code eventually. This code is also hit when a flow is
suspended and a shutdown event is sent to it.

The `liveFibers` latch blocks the SMM from shutting down until all flows
have finished or processed shutdown events.

The changes described below resolve this problem.

Any flow that goes to the `FlowDefaultUncaughtExceptionHandler` will be
put marked as dead (`StateMachineState.isDead`). Highlighting that the
flow cannot continue to process events normally as it has broken out
of its event loop

Retrying and shutdown are done manually rather than injecting events
into the flow fiber's queue, because it can't execute its event loop.

Killing a dead flow executes an altered version of
`retryFlowFromSafePoint`. It does this so it can delete the checkpoint
and then continue using the checkpoint it just deleted to run the
kill flow transition on a new fiber.

If a killed flow reaches the `FlowDefaultUncaughtExceptionHandler` it
will be forcibly killed via `killFlowForcibly` which deletes the
checkpoint/or updates it to KILLED and then calls `removeFlow` to bypass
any event processing. This means that a flow that was dead and was killed
will be terminated manually if it reaches the handler again. The same is
true for flows that were not dead before but reached the handler after
being killed.

Also, `FlowCreator.createFlowFromCheckpoint` now retains the `isKilled`
state of the previous fiber's state.
2021-01-29 11:15:28 +00:00
Alexey Koren
56df286410
Add ability to set custom initiating flows for NotaryService (#6832) 2021-01-26 13:56:42 +00:00
Adel El-Beik
284fd48918
CORDA-4101: Remove attachmentPresence cache (#6845)
* CORDA-4101: Remove the attachmentCache and just rely on attachmentContentCache

* CORDA-4101: Remove the attachmentCacheBound variable.
2021-01-13 16:28:56 +00:00
nkovacsx
be6587a602 Merge branch 'release/os/4.7' into nandor/4.7-4.8-merge-os 2021-01-08 14:25:33 +00:00
nkovacsx
7a079a166b
CORDA-4097 Return stream instead of List in getAllCertificates (#6840) 2021-01-08 09:41:31 +00:00
Denis Rekalov
f5d9d97573 Merge branch 'release/os/4.7' into denis/merge-4.7-to-4.8 2020-12-14 10:56:23 +00:00
Stefan Iliev
2c25a5c768
AG-323 Flow startup queue metric (#6811) 2020-11-30 16:49:08 +00:00
Denis Rekalov
503644ca29
CORDA-4083: Fix Postresql migration, synch migration scripts from ENT (#6817) 2020-11-24 20:03:36 +00:00
Edoardo Ierina
02018b75e9
ENT-6018: renamed hash agility system properties (#6808) 2020-11-17 12:50:31 +00:00
Waldemar Żurowski
700cb51091
INFRA-965: forward merge from 4.6 2020 11 15 (#6807)
* CORDA-4045 Disable flaky test (#6792)

Disable `Restart does not set senderUUID` as it is unstable.

* INFRA-965: forward merge from 4.5 (2020-11-15) (#6806)

* EG-4168 Updating contributors.md list for OS 4.5 release branch (#6784)

* INFRA-965: forward merge from 4.4 (2020-11-14) (#6805)

* EG-4168 Updating contributors.md list for OS 4.4 release branch (#6785)

* INFRA-965: forward merge from 4.3 (2020-11-13) (#6803)

* INFRA-965: Jenkins/NexusIQ integration should target patches (#6802)

as well as major/minor releases

* Updated nightly builds to match changes for release branch builds

* Updated JDK11 builds to match changes for release branch builds

Co-authored-by: ivanterziev <61829352+ivanterziev-r3@users.noreply.github.com>

* Backporting updates for check-pr-title workflow

Co-authored-by: ivanterziev <61829352+ivanterziev-r3@users.noreply.github.com>

Co-authored-by: Ross Nicoll <ross.nicoll@r3.com>
Co-authored-by: ivanterziev <61829352+ivanterziev-r3@users.noreply.github.com>
2020-11-16 07:18:22 +00:00