* ENT-12366: External verifier now sets appclassloader to legacy contracts directory instead of the cordapps directory.
* ENT-12366: Now check legacy-contracts exists before start external verifier.
* Serialization performance test of creating wire transaction
* Initial serialization refactoring to enable future caching of schema
* Add caching of schema
* Move encoder pool to the companion object so it actually gets re-used!
* Slightly better cache concurrency for LocalSerializerFactory
* Upgrade grgit to 4.1.1 as 4.0.0 seems to have vanished
The `TransactionBuilder` has been updated to look for any missing dependencies to legacy contract attachments, in the same way it does for missing dependencies for CorDapps in the "cordapps" directory,
Since `TransactionBuilder` does verification on the `WireTransaction` and not a `SignedTransaction`, much of the verification logic in `SignedTransaction` had to moved to `WireTransaction` to allow the external verifier to be involved. The external verifier receives a `CoreTransaction` to verify instead of a `SignedTransaction`. `SignedTransaction.verify` does the signature checks first in-process, before then delegating the reset of the verification to the `CoreTransaction`.
A legacy contract dependency is defined as an attachment containing the missing class which isn't also a non-legacy Cordapp (i.e. a CorDapp which isn't in the "cordapp" directory).
This requires Kotlin 1.2 versions of core and serialization (core-1.2 and serialization-1.2 respectively), which are just "shell" modules and which compile the existing source code with Kotlin 1.2. The 1.2 plugin does not work with the current version of Gradle and so the 1.2 compiler has to be called directly.
Now with two versions of Kotlin in the code base, each module needs to have its version manually specified to ensure a clean separation. Otherwise, the default Kotlin version can override 1.2 when needed.
Some of the code was tidied-up or improved to enable it to be cross-compiled. For post-1.2 APIs being used, they have been copied into core-1.2 with the same method signatures. OpenTelemetryComponent was moved to node-api, along with the dependency, to avoid also having a 1.2 version for the opentelemetry module.
* ENT-11055: Basic external verification
Introduction of the external transaction verifier, a separate JVM process for verifying `SignedTransaction`s. The end goal is for this verifier to be built with Kotlin 1.2 so that it creates a compatible verification environment for transactions with 4.11 contracts. For now however the verifier is built against Kotlin 1.8, same as the node.
External verification is enabled when the the system property `net.corda.node.verification.external` is set to `true`. When enabled, all verification requests made via `SignedTransaction.verify` are sent to the external verifier, regardless of the transaction content. It will do the vast bulk of the verification and then send the result back, namely if an exception occurred. If it did, then it's re-thrown in the node.
The external verifier is a stateless process, with no connection to the node's database. All transaction resolution information needed to create the relevant ledger transaction object are made to the node, which waits in a loop servicing these requests until it receives the result. The verifier Jar is embedded in the Corda node Jar, and is extracted and run when needed for the first time. The node opens up a local port for the verifier to communicate with, which is specified to the verifier in the process command line. This all means there is no extra configuration or deployment required to support external verification.
The existing code had some initial attempts and abstractions to support a future external verification feature. However,
they were either incorrect or didn't quite fit. One such example was `TransactionVerifierService`. It incorrectly operated on the `LedgerTransaction` level, which doesn't work since the transaction needs to be first serialised. Instead a new abstraction, `VerificationSupport` has been introduced, which represents all the operations needed to resolve and verify a `SignedTransaction`, essentially replacing `ServicesForResolution` (a lot of the changes are due to this). The external verifier implements this with a simple RPC mechanism, whilst the node needed a new (internal) `ServiceHub` abstraction, `VerifyingServiceHub`. `ServicesForResolution` hasn't been deleted since it's public API, however all classes implementing it must also implement `VerifyingServiceHub`. This is possible to do without breaking compatibility since `ServicesForResolution` is annotated with `@DoNotImplement`.
Changes to `api-current.txt` were made due to the removal of `TransactionVerifierService`, which was clearly indicated as an internal class, and returning `TransactionBuilder.toLedgerTransactionWithContext` back to an internal method.
* Address review comments
* One bulk load states method
* Merge fix
Replaced usage of `@Test.expected` annotation parameter with more specific exception assertions. This is also needed to migrate away from the explicit timeouts in every tests.
Major changes due to JDK 17:
1. JDK17 JCE Provider now has built-in support for eddsas, corda uses
the bouncycastle (i2p) implementation. This PR removes the conflicting
algorithms from the built-in JCE provider.
2. JavaScript scripting has been removed from the JDK, the corda log4j config was using
scripting to conditionally output additional diagnostic info if the MDC
was populated. This PR has removed the scripting.
3. The artifactory plug-ins used are now deprecated, this PR has removed them
and uses the same code as Corda 5 for publishing to artifactory.
4. Javadoc generation has been modified to use the latest dokka plug-ins.
5. Gradle 7.6 has implemented an incredibly annoying change where transitive
dependencies are not put on the compile classpath, so that they have to be
explicitly added as dependencies to projects.
6. Mockito has been updated, which sadly meant that quite a few source files
have to changes to use the new (org.mockito.kotlin) package name. This makes
this PR appear much larger than it is.
7. A number of tests have been marked as ignored to get a green, broadly they fall
into 3 classes.
The first is related to crypto keypair tests, it appears some logic
in the JDK prefers to use the SunJCE implementation and we prefer to use
bouncycastle. I believe this issue can be fixed with better test setup.
The second group is related to our use of a method called "uncheckedCast(..)",
the purpose of this method was to get rid of the annoying unchecked cast compiler
warning that would otherwise exist. It looks like the Kotlin 1.9 compiler type
inference differs and at runtime sometimes the type it infers is "Void" which causes
an exception at runtime. The simplest solution is to use an explicit cast instead of
unchecked cast, Corda 5 have removed unchecked cast from their codebase.
The third class are a number of ActiveMQ tests which appear to have a memory leak somewhere.
There was a mistake made when we first introduced notary request signature checking, in that we didn't wrap it in SerializedBytes so it always got deserialized as part of the flow message payload. So to check the signature, it has to be re-serialized. This means for cross-version compatibility we can never change the serialized format of NotarisationRequest. In this case we need make sure that every SecureHash mentioned in that data structure is a distinct instance, even if the values are repeated / identical, as that is how it was in Corda 1.
With the introduction of interning of SecureHash, this ceased to be true once again, including undoing the attempts to force it on the sending side that had been introduced in previous versions of Corda. So here we introduce a way to force it, and consolidate the forcing to distinct SecureHash instances in the NotarisationRequest itself, rather than leaving to the caller of the constructor to remember to do it, so that serialized form will always be as per Corda 1.
ENT-6947: Implement interning for SecureHash, CordaX500Name, PublicKey, AsbtractParty and SignatureAttachmentConstraint, including automatic detection of internable types off companion objects in AMQP & Kyro deserialization. In some cases, add new factory methods to companion objects, and make main code base use them.
Performance tested in performance cluster with no negative impact visible (so default concurrency setting seems okay).
Testing suggests 5-6x memory saving for tokens in TokensSDK in memory selector. Should see approx. 1 million tokens per GB or better (1.5 million for the tokens we tested with).
* ENT-6357: Deserialize LedgerTransaction elements for each Contract.verify().
* Lock the LedgerTransaction and NetworkParameters objects down for contract verification.
* Refactor BasicVerifier to be package private instead of public.
* Simplify verifyConstraints() operation.
* Review fixes: replace HashSet with LinkedHashSet, and add signing parties to commands via mapIndexed.
* Ensure tests also run notary nodes "out of process".
* Streamline SerializationContext switching.
* Cache deserialised cryptographic instances during contract verification.
* Invoke Class.forName() instead of ClassLoader.loadClass() to reduce contention on the system classloader's lock.
* Deserialization cache key now pre-computes its hash code.
* Allow AttachmentsClassLoader to be used concurrently.
* Cache all Envelope objects for reuse during contract verification.
* Generate CertPathProxy hash code using conventional algorithm.
* Adjust CustomSerializer.Proxy to allow better access to SerializationContext.
* CORDA-4105 Add public API to allow custom serialization schemes (#6848)
* CORDA-4105 Add public API to allow custom serialization schemes
* Fix Detekt
* Suppress warning
* Fix usused import
* Improve API to use generics
This does not break Java support (only Intelij gets confused).
* Add more detailed documentation to public interfaces
* Change internal variable name after rename public API
* Update Public API to use ByteSquence instead of SerializedBytes
* Remove unused import
* Fix whitespace.
* Add added public API to .ci/api-current.txt
* Improve public interfaces
Rename CustomSchemeContext to SerializationSchemeContext to improve
clarity and move to it's own file. Improve kdoc to make things less
confusing.
* Update API current with changed API
* CORDA-4104 Implement custom serialization scheme discovery mechanism (#6854)
* CORDA-4104 Implement CustomSerializationScheme Discovery Mechanism
Discovers a single CustomSerializationScheme implementation inside
the drivers dir using a system property.
* Started MockNetwork test
* Add driver test of Custom Serialization Scheme
* Fix detekt and minor style error
* Respond to review comments
Allow non-single arg constructors (there must be one no args
constructor), move code from SerializationEnviroment into its
own file, improve exceptions to be more user friendly.
* Fix minor bug in Scheme finding code + improve error messages
* CORDA-4104 Improve test coverage of custom serialization scheme discovery (#6855)
* CORDA-4104 Add test of classloader scanning for CustomSerializationSchemes
* Fix Detekt
* NOTICK Clarify KDOC on SerializationSchemeContext (#6865)
* CORDA-4111 Change Component Group Serialization to use contex when the lazy map is constructed (#6856)
Currently the component group will recheck the thread local (global)
serialization context when component groups are serialized lazily.
Instead store the serialization context when the lazy map is constructed
and use that latter when doing serialization lazily.
* CORDA-4106 Test wire transaction can still be written to the ledger (#6860)
* Add test that writes transaction to the Database
* Improve test check serialization scheme in test body
* CORDA-4119 Minor changes to serialisation injection for transaction building (#6868)
* CORDA-4119 Minor changes to serialisation injection for transaction building
Scan the CorDapp classloader instead of the drivers classloader.
Add properties map to CustomSerialiaztionContext (copied from SerializationContext).
Change API to let a user pass in the serialization context in TransactionBuilder.toLedgerTransaction
* Improve KDOC + fix shawdowing issue in CordaUtils
* Pass only the properties map into theTransactionBuilder.toWireTransaction
Not the entire serializationContext
* Revert change to CordaUtils
* Improve KDOC explain pitfalls of setting properties
* CORDA-3824: Add unit tests
* CORDA-3824: Fix property rename in AMQP object evolution
* Rename deserializedException to deserializedObject
* Rename test class to EvolutionObjectBuilderRenamedPropertyTests
* Added descriptions of the different object evolution stages in this test
* Rename file containing the serialized object
* Regenerate serialized data
* Add a comment explaining the commented out code.
* Restrict new behaviour to EvolutionObjectBuilder and simplify the loop that builds constructor slots.
* CORDA-3936: Add a fallback mechanism for Enums incorrectly serialised using their toString() method.
* Backport missing piece of Enum serializer from Corda 4.6.
* CORDA-3717: Apply custom serializers to checkpoints
* Remove try/catch to fix TooGenericExceptionCaught detekt rule
* Rename exception
* Extract method
* Put calls to the userSerializer on their own lines to improve readability
* Remove unused constructors from exception
* Remove unused proxyType field
* Give field a descriptive name
* Explain why we are looking for two type parameters when we only use one
* Tidy up the fetching of types
* Use 0 seconds when forcing a flow checkpoint inside test
* Add test to check references are restored correctly
* Add CheckpointCustomSerializer interface
* Wire up the new CheckpointCustomSerializer interface
* Use kryo default for abstract classes
* Remove unused imports
* Remove need for external library in tests
* Make file match original to remove from diff
* Remove maySkipCheckpoint from calls to sleep
* Add newline to end of file
* Test custom serializers mapped to interfaces
* Test serializer configured with abstract class
* Move test into its own package
* Rename test
* Move flows and serializers into their own source file
* Move broken map into its own source file
* Delete comment now source file is simpler
* Rename class to have a shorter name
* Add tests that run the checkpoint serializer directly
* Check serialization of final classes
* Register as default unless the target class is final
* Test PublicKey serializer has not been overridden
* Add a broken serializer for EdDSAPublicKey to make test more robust
* Split serializer registration into default and non-default registrations. Run registrations at the right time to preserve Cordas own custom serializers.
* Check for duplicate custom checkpoint serializers
* Add doc comments
* Add doc comments to CustomSerializerCheckpointAdaptor
* Add test to check duplicate serializers are logged
* Do not log the duplicate serializer warning when the duplicate is the same class
* Update doc comment for CheckpointCustomSerializer
* Sort serializers by classname so we are not registering in an unknown or random order
* Add test to serialize a class that references itself
* Store custom serializer type in the Kryo stream so we can spot when a different serializer is being used to deserialize
* Testing has shown that registering custom serializers as default is more robust when adding new cordapps
* Remove new line character
* Remove unused imports
* Add interface net.corda.core.serialization.CheckpointCustomSerializer to api-current.txt
* Remove comment
* Update comment on exception
* Make CustomSerializerCheckpointAdaptor internal
* Revert "Add interface net.corda.core.serialization.CheckpointCustomSerializer to api-current.txt"
This reverts commit b835de79bd.
* Restore "Add interface net.corda.core.serialization.CheckpointCustomSerializer to api-current.txt""
This reverts commit 718873a4e9.
* Pass the class loader instead of the context
* Do less work in test setup
* Make the serialization context unique for CustomCheckpointSerializerTest so we get a new Kryo pool for the test
* Rebuild the Kryo pool for the given context when we change custom serializers
* Rebuild all Kryo pools on serializer change to keep serializer list consistent
* Move the custom serializer list into CheckpointSerializationContext to reduce scope from global to a serialization context
* Remove unused imports
* Make the new checkpointCustomSerializers property default to the empty list
* Delegate implementation using kotlin language feature
* CORDA-3716: Fix SandboxEnumSerializer to handle enums that override toString().
* Remove more uses of Enum.toString() from the Corda serializer.
* Add test coverage for this case to standard enum serializer.
* Increase maxWaitTimeout in IRSDemoTest to 150 seconds.
* CORDA-3716: Fix SandboxEnumSerializer to handle enums that override toString().
* Remove more uses of Enum.toString() from the Corda serializer.
* Add test coverage for this case to standard enum serializer.
* Increase maxWaitTimeout in IRSDemoTest to 150 seconds.
* ENT-4967: Require no classifier for corda-node-djvm, corda-deserializers-djvm.
* Also remove classifiers from core, serialization and finance-contracts.
* Compile corda-serialization-djvm for Java 8 and remove its classifier.
* [NOTICK] Add a custom detekt rule for tests with no timeout, and fix remaining missing timeouts
* [NOTICK] Add a test for custom detekt rules and tidying
* add timeout annotation to new test
Co-authored-by: Stefano Franz <roastario@gmail.com>
