* ENT-11055: Basic external verification
Introduction of the external transaction verifier, a separate JVM process for verifying `SignedTransaction`s. The end goal is for this verifier to be built with Kotlin 1.2 so that it creates a compatible verification environment for transactions with 4.11 contracts. For now however the verifier is built against Kotlin 1.8, same as the node.
External verification is enabled when the the system property `net.corda.node.verification.external` is set to `true`. When enabled, all verification requests made via `SignedTransaction.verify` are sent to the external verifier, regardless of the transaction content. It will do the vast bulk of the verification and then send the result back, namely if an exception occurred. If it did, then it's re-thrown in the node.
The external verifier is a stateless process, with no connection to the node's database. All transaction resolution information needed to create the relevant ledger transaction object are made to the node, which waits in a loop servicing these requests until it receives the result. The verifier Jar is embedded in the Corda node Jar, and is extracted and run when needed for the first time. The node opens up a local port for the verifier to communicate with, which is specified to the verifier in the process command line. This all means there is no extra configuration or deployment required to support external verification.
The existing code had some initial attempts and abstractions to support a future external verification feature. However,
they were either incorrect or didn't quite fit. One such example was `TransactionVerifierService`. It incorrectly operated on the `LedgerTransaction` level, which doesn't work since the transaction needs to be first serialised. Instead a new abstraction, `VerificationSupport` has been introduced, which represents all the operations needed to resolve and verify a `SignedTransaction`, essentially replacing `ServicesForResolution` (a lot of the changes are due to this). The external verifier implements this with a simple RPC mechanism, whilst the node needed a new (internal) `ServiceHub` abstraction, `VerifyingServiceHub`. `ServicesForResolution` hasn't been deleted since it's public API, however all classes implementing it must also implement `VerifyingServiceHub`. This is possible to do without breaking compatibility since `ServicesForResolution` is annotated with `@DoNotImplement`.
Changes to `api-current.txt` were made due to the removal of `TransactionVerifierService`, which was clearly indicated as an internal class, and returning `TransactionBuilder.toLedgerTransactionWithContext` back to an internal method.
* Address review comments
* One bulk load states method
* Merge fix
Replaced usage of `@Test.expected` annotation parameter with more specific exception assertions. This is also needed to migrate away from the explicit timeouts in every tests.
Major changes due to JDK 17:
1. JDK17 JCE Provider now has built-in support for eddsas, corda uses
the bouncycastle (i2p) implementation. This PR removes the conflicting
algorithms from the built-in JCE provider.
2. JavaScript scripting has been removed from the JDK, the corda log4j config was using
scripting to conditionally output additional diagnostic info if the MDC
was populated. This PR has removed the scripting.
3. The artifactory plug-ins used are now deprecated, this PR has removed them
and uses the same code as Corda 5 for publishing to artifactory.
4. Javadoc generation has been modified to use the latest dokka plug-ins.
5. Gradle 7.6 has implemented an incredibly annoying change where transitive
dependencies are not put on the compile classpath, so that they have to be
explicitly added as dependencies to projects.
6. Mockito has been updated, which sadly meant that quite a few source files
have to changes to use the new (org.mockito.kotlin) package name. This makes
this PR appear much larger than it is.
7. A number of tests have been marked as ignored to get a green, broadly they fall
into 3 classes.
The first is related to crypto keypair tests, it appears some logic
in the JDK prefers to use the SunJCE implementation and we prefer to use
bouncycastle. I believe this issue can be fixed with better test setup.
The second group is related to our use of a method called "uncheckedCast(..)",
the purpose of this method was to get rid of the annoying unchecked cast compiler
warning that would otherwise exist. It looks like the Kotlin 1.9 compiler type
inference differs and at runtime sometimes the type it infers is "Void" which causes
an exception at runtime. The simplest solution is to use an explicit cast instead of
unchecked cast, Corda 5 have removed unchecked cast from their codebase.
The third class are a number of ActiveMQ tests which appear to have a memory leak somewhere.
* Swap logic from receive finality to receive transaction flows
* Remote TPV check
* Make finality check more robust
* Make emulation of finality in tests compliant with changes
* Improve deferring of ack when issue transaction
* Remove API checking of SignedTransactionWithDistributionList as added it 4.11 so cannot be incompatible, yet.
* Regenerated API file from 4.10 to check only compatibility with 4.10
* Move function to private
* Revert "Regenerated API file from 4.10 to check only compatibility with 4.10"
This reverts commit 6428f957e1.
* Reset ReceiveTransactionFlow and ReceiveFinalityFlow APIs
This is so that the node archiving service, which scans for tables containing "transaction_id" column, can automatically archive the sender and receiver distribution record information with the transaction.
* ENT-9875: Added new network parameters
- Added `transactionRecoveryPeriod`
- Added `confidentialIdentityPreGenerationPeriod`
These new parameters are currently set to be nullable meaning they can be ignored and the duration if not specified will be null rather than, e.g., 0. This currently allows for nothing changing/breaking in the node-api
_Note: if these params can stay as nullable then the deprecated constructor might not even be needed (since the existing one will still work), needs to be discussed._
* Delay closing of attachment class loaders until all SerializationContext that refer to them (from BasicVerifier) have gone out of scope.
More comments
* Avoid any concurrency issues with queue processing
* Better concurrency behaviour
* Stop re-using attachment URLs as it turns out we can't close the URLClassLoaders when URLs are shared
* Refactor to use a ReferenceQueue.
Co-authored-by: Chris Rankin <chris.rankin@r3.com>
* ENT-6357: Deserialize LedgerTransaction elements for each Contract.verify().
* Lock the LedgerTransaction and NetworkParameters objects down for contract verification.
* Refactor BasicVerifier to be package private instead of public.
* Simplify verifyConstraints() operation.
* Review fixes: replace HashSet with LinkedHashSet, and add signing parties to commands via mapIndexed.
* Ensure tests also run notary nodes "out of process".
* Streamline SerializationContext switching.
* Cache deserialised cryptographic instances during contract verification.
* Invoke Class.forName() instead of ClassLoader.loadClass() to reduce contention on the system classloader's lock.
* Deserialization cache key now pre-computes its hash code.
* Allow AttachmentsClassLoader to be used concurrently.
* Cache all Envelope objects for reuse during contract verification.
* Generate CertPathProxy hash code using conventional algorithm.
* Adjust CustomSerializer.Proxy to allow better access to SerializationContext.
* CORDA-4105 Add public API to allow custom serialization schemes (#6848)
* CORDA-4105 Add public API to allow custom serialization schemes
* Fix Detekt
* Suppress warning
* Fix usused import
* Improve API to use generics
This does not break Java support (only Intelij gets confused).
* Add more detailed documentation to public interfaces
* Change internal variable name after rename public API
* Update Public API to use ByteSquence instead of SerializedBytes
* Remove unused import
* Fix whitespace.
* Add added public API to .ci/api-current.txt
* Improve public interfaces
Rename CustomSchemeContext to SerializationSchemeContext to improve
clarity and move to it's own file. Improve kdoc to make things less
confusing.
* Update API current with changed API
* CORDA-4104 Implement custom serialization scheme discovery mechanism (#6854)
* CORDA-4104 Implement CustomSerializationScheme Discovery Mechanism
Discovers a single CustomSerializationScheme implementation inside
the drivers dir using a system property.
* Started MockNetwork test
* Add driver test of Custom Serialization Scheme
* Fix detekt and minor style error
* Respond to review comments
Allow non-single arg constructors (there must be one no args
constructor), move code from SerializationEnviroment into its
own file, improve exceptions to be more user friendly.
* Fix minor bug in Scheme finding code + improve error messages
* CORDA-4104 Improve test coverage of custom serialization scheme discovery (#6855)
* CORDA-4104 Add test of classloader scanning for CustomSerializationSchemes
* Fix Detekt
* NOTICK Clarify KDOC on SerializationSchemeContext (#6865)
* CORDA-4111 Change Component Group Serialization to use contex when the lazy map is constructed (#6856)
Currently the component group will recheck the thread local (global)
serialization context when component groups are serialized lazily.
Instead store the serialization context when the lazy map is constructed
and use that latter when doing serialization lazily.
* CORDA-4106 Test wire transaction can still be written to the ledger (#6860)
* Add test that writes transaction to the Database
* Improve test check serialization scheme in test body
* CORDA-4119 Minor changes to serialisation injection for transaction building (#6868)
* CORDA-4119 Minor changes to serialisation injection for transaction building
Scan the CorDapp classloader instead of the drivers classloader.
Add properties map to CustomSerialiaztionContext (copied from SerializationContext).
Change API to let a user pass in the serialization context in TransactionBuilder.toLedgerTransaction
* Improve KDOC + fix shawdowing issue in CordaUtils
* Pass only the properties map into theTransactionBuilder.toWireTransaction
Not the entire serializationContext
* Revert change to CordaUtils
* Improve KDOC explain pitfalls of setting properties
* wip
* wip
* wip (need to review IEE comments)
* wip
* wip
* Small refactoring, fixed network-verifier's TestNotaryFlow
* Added command line option to explicitly enable hash agility support
* wip-do-not-push
* wip
* wip
* wip
* aligned merkletree/transaction hash algorithms
* wip
* Added mixed algorithm support for nodes vs leaves and corrected mixed algorithm tests
* moved global computeNonce and componentHash to DigestService
* added comment for failing test to fix
* wip
* Minor cleanups, added deprecated componentHash/computeNonce
* restored exploratory changes to failing SignedTransaction test
* cleaned up and minor rafactoring
* Fixed some tests with hardcoded hash algorithm
* some changes and cleanups following code review
* WIP commit before large change
* WIP Fixed 3 tests
* WIP removed direct references to randomSHA256() and sha256()
* Updated/added liquibase migrations to support larger hash algorithms
* Reviewed, cleanups, comments, fixes
* removing direct references to sha256()
* WIP verifying obligations test errors
* reviewing obligation/attachment issues with sha3_256
* Full review before PR - intermediate commits
* Reviewed and cleaned up
* Futher cleanup
* Fixed partial tree backward compatible json and cleanups
* all tests passing
* Removed couple of unused imports
* Reworked global componentHash function to avoid deprecated warnings
* replaced SHA3s with some alternate SHA2s
* Removed SHA3-256 and SHA3-512 references
* fixed some tests using non ubiquitous hash algorithms
* Fixed ABI compatibility (not for TransactionBuilder)
* Fixed ABI compatibility to TransactionBuilder
* couple of fixes
* fixed DigestService's randomHash
* Removed constructor with loosely typed args for private constructor of LedgerTransaction class (API removal)
* re-introduced LedgerTransaction deprecated ctor for deserialization
* Add possibility to load CustomMessageDigest bypassing JCA (#6798)
* Change api-current for DigestAlgorithm
* disable flaky tests
* addressed liquibase migration script versions
* Removed TODOs and cleanups
* relaxed privacy salt validation
* Fixed privacy salt test to comply with relaxed validation
* detekt and privacySalt validation
* diff cleanup
* diff cleanup
* removed unused import
* removed PrivacySalt's validateFor method and references
* removed invalid character
Co-authored-by: Denis Rekalov <denis.rekalov@r3.com>
