* CORDA-1208: Notary service should persist the notarisation request signature along with the committed input states.
This required modifying the uniqueness provider interface to accept the signature in addition to input states.
Until now the committed state log used to be stored as a map of (state reference -> (tranasction id, consuming party)).
Adding the serialized signature would mean inflating each state entry by around 700 bytes, which would be grossly inefficient.
Instead, two tables are now used: one for storing (state referece -> transaction id) map, and another for storing the notarisation
request details (transaction id, consuming party, date, signature).
* Update api - all of these changes are only related to custom notaries
Old code failed with:
```
11:34:01.536 [main] ERROR net.corda.node.internal.Node - Exception during node startup
java.nio.file.InvalidPathException: Illegal char <:> at index 2: /C:/Program%20Files/JetBrains/IntelliJ%20IDEA%20Community%20Edition%202017.3.1/lib/idea_rt.jar
at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182) ~[?:1.8.0_144]
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153) ~[?:1.8.0_144]
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77) ~[?:1.8.0_144]
at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94) ~[?:1.8.0_144]
at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255) ~[?:1.8.0_144]
at java.nio.file.Paths.get(Paths.java:84) ~[?:1.8.0_144]
at net.corda.node.utilities.JVMAgentRegistry.resolveAgentJar(JVMAgentRegistry.kt:46) ~[classes/:?]
at net.corda.node.internal.AbstractNode.initialiseJVMAgents(AbstractNode.kt:785) ~[classes/:?]
at net.corda.node.internal.AbstractNode.start(AbstractNode.kt:199) ~[classes/:?]
at net.corda.node.internal.Node.start(Node.kt:335) ~[classes/:?]
at net.corda.node.internal.NodeStartup.startNode(NodeStartup.kt:146) ~[classes/:?]
at net.corda.node.internal.NodeStartup.run(NodeStartup.kt:120) [classes/:?]
at net.corda.node.Corda.main(Corda.kt:16) [classes/:?]
```
The problem with the previous implementation is that the transaction would be deserialized with the schema specified
in the serialized form, but the calculation of the id would involve re-serializing properties using a local serialization context
which might produce a different result.
- Existing embedded Shell connects via RPC including checking RPC user credentials (before was a direct use of CordaRPCOps): in dev mode when console terminal is enabled, node created `shell` user.
- New Standalone Shell app with the same functionalities as Shell: connects to a node via RPC Client, can use SSL and run SSH server.
* Fix another non-serializable exception, add docs, fix a possible security issue.
* Update API definition to reflect methods added to make more exceptions serializable
This was achieved by having the hash in the node-info file to be just of the node's X.500 name. This also solves existing duplicate node-info file issues that we've been having.
Also updated the docsite.
* CORDA-1122 Switch AMQP bridge to use auto-commit sends and acks
* CORDA-1122 Codereview: Add comment for extra commit to rollback and add better message sequence logging to tricky test
* CORDA-986 and CORDA-985 CompositeKey and Signature verification performance fixes (#2467)
* CORDA-696: Create separate transaction types for contract upgrade transactions.
Add rationale around upgrade transactions
Move contract upgrade transaction resolution logic into internal until it's stabilised.
Throw a better exception when contract attachment not found
Default legacy contract constraint to always accepting - needs to be changed to whitelist constraint before merging
Introduce a new upgraded contract interface that allows specifying the legacy constraint.
Remove StateLoader, make all tx resolution functions take in ServicesForResolution
Contract upgrade transactions can handle whitelist by zone constraints
When creating a contract upgrade transaction, make sure the attachment of the old cordapp gets attached when using hash constraints.
Attachment lookup for a given contract class name only scans currently loaded cordapps, and we don't load old versions of cordapps.
CORDA-696: Update upgrade docs
* CORDA-1096 - Performance when loading multiple states from the vault (#2609)
* Provide efficient `loadStates()` implementation
* Replace loops using `loadState` with calls to `loadStates`
* Replace `map`/`flatMap` with just a single `flatMap`
* ENT-1443 Add cert role to CSR and doorman issue cert according to the cert role (#431)
* Doorman and HSM create certificate base on requested cert role specified in the certificate signing request.
(cherry picked from commit 94f7392)
* remove R3 corda code
CORDA-1010: Notary flow - clients now send a signature over a notarisation
request in addition to the transaction. This will be logged by the notary
to be able to prove that a particular party has requested the consumption
of a particular state.
* Jolokia agents are loaded dynamically if configured
* Renamed exportJmxTo (never used) to jmxMonitoringHttpPort and take it from config
* Updated documentation and tests
Further, the look up of the node's own node-info from its database has been tightened to ensure there isn't more than one.
Also fixed some brittle code which was assuming exactly one address rather than at least one.
* Update check api changes to look for internals
* Update several more uses of internal
* Make check-api-changes script filter out internal class usages
* Make CordaClock part of API
* Update api-current.txt
* Remove exclusion of nodeapi.internal
* Remove access to CordaPersistence from public api
* Don't expose DB Connection from StartedMockNode and remove unnecessary transaction from CustomVaultQueryTest
* Make internal tests that use need db access use InternalMockNetwork
* Make test certificates internal
* Address further review comments
* Revert some accidental changes to api-current.txt
* Address Shams' review comments
* Update Api Scanner to filter out CordaInternal attribute
* Update api-current.txt
* Remove superfluous brackets
* Add transaction to StartedMockNode
* More leaky transaction fixes
* Added per-cordapp configuration
* Added new API for Cordformation cordapp declarations to support per-cordapp configuration
* Added a cordapp configuration sample
* Remove sslConfiguration from public constructor of CordaRPCClient
* Address review comments
* Update api-current.txt
* sslConfiguration doesn't need to be a property
* Don't expose StartedNode via Node Driver
* Dont expose StartedNode/Abstract Node via MockNetwork
* Remove internal var from constructor as it doesn't hide from public api and change to internal initialisation method
* Update api
* Rename MockNode to StartedMockNode to avoid confusion
Update documentation
Update api-current.txt
* Fix typo
* Fix test failure
* Modify flow tests to use internal mock network and remove additional internal exposures from StartedMockNode
* Fix api-current
* Change InProcess and OutOfProcess to interfaces
* Explicitly declare MockNetwork parameters
Dont expose StateMachineManager
Move affected tests to use internal mock network
* Fix api-current
* Changes requested via review
* Fix IRS Demo address
* Fix api
* Remove internal attribute from classes in internal package
* Remove accidentally added code
* Move useHttps into NodeHandleInternal
* Remove duplicated code
* Update api-current
* Make webAddress internal on NodeHandle
* Make sure parameters in public api are explicitly specified
* Use correct address in IRS Demo
* Get webaddress from webserver handle
* Update api-current
* wire up and enforce max transaction size
* fixup after rebase
moved network parameter from AbstractNode to NodeProperties
* removed TODO
* fix broken import
* address PR issues
* remove API breaking change
address PR issue
* added max transaction size to driver and mock network.
address PR issues
* fix failing test
* added TODO
* fix verifier test
* fix spring driver build error
* Network parameters updates
Add two RPC methods networkParametersFeed and
acceptNewNetworkParameters. Implementation of client handling of network
parameters update event. Partial implementation of accepting new
parameters and installing them on the node as well as node startup with
updated parameters.
Move reading of network parameters on startup to separate
NetworkParametersReader class. Add tests.
Move NetworkParameters and NotaryInfo classes to core.
* Ignore evolvability test - to be fixed later
* Add documentation on update process
* Create CordaInternal attribute for properties on public classes that are not part of the api and apply to FlowLogic.stateMachine
* Remove startFlow from public test api and replace with startFlowAndReturnFuture
* Update api-current with changed signature
* Change test used in documentation to use public test methods
* Remove the rest of the unneccessary usages of the startFlow test utility
* Remove extra whitespace
* Rename startFlowAndReturnFuture back to startFlow
* Update api
* The annotation doesn't appear unless its marked as on the actual getter and setter
* Updated docs and removed pointless attribute
* Deleted whitespace
Minor tweaks to the invocation context code.
1) Un-deprecate FlowInitiator, move the deprecation to the field. This
eliminates large numbers of warnings and means developers are warned
only once in the place where they obtain one.
2) Add documentation for StateMachineInfo and create a type alias to give
it a better name in an ABI compatible way.
3) Improve markup on InvocationContext
4) Rename field from just "context" to "invocationContext" (Context is vague)
* set network registration poll interval via http cache control header from the server side
* default poll interval to 10 seconds if cache header not found
* address PR issues
* address PR issues
(cherry picked from commit dca8699)
* add cmdline option for network root truststore and password, instead of using node's truststore configuration to avoid confusion.
* revert line auto format
* fix failing integration test
* address PR issue
* Make notary service return the current time and the transaction time window
along with the TimeWindowInvalid error.
Deprecate TimeWindowChecker.
Add a static method for validating transaction time window to reduce code duplication.
* ENT-1403 Cache node attachments (and attachment content)
* ENT-1403 Make cache sizes configurable
* Update documentation with new config parameters
* Test that non-existence of attachments is not cached
* Remove unneeded defaults in interface
* It turned out we need the defaults on the interface in quite a few tests
* Codereview: typos, size in MB rather than bytes, charset in tests, move concurrencyLevel to a constant
* Codereview: Make the internal config value bytes again, but config file in MB
* Fix example config unit test
Changes compatible with R3.Corda (ENT-794):
1) Added Hibernate corda-wrapper-binary two to to columns.
2) Shorten names of tables in dummy schemas used in tests.
3) Undo removal of compound index of VaultTxnNote (b423fea).
4) Assertions for 2 vault tests don't rely on order of rows.
* Test for node restart
* Executor gets shutdown on stop, make sure we have one on start
* Reset shutdown otherwise AbstractNode.stop never gets called
* CORDA-928 cache query results via `getPeerByLegalName` and `getNodesByIndentityKey` to avoid hitting the DB hard in RPC handling.
* Skip cache invalidation during init() - caches are still null.
* Remove registeredNodes/partyNodes caching of data feed.
Rewrite data feed to be initialised off the DB.
Add start method to trigger readyness/artemis listeners if there are nodes in the DB.
* Invalidate cache last rather than first when updating
* ENT-1383 Make the transaction cache in DBTransactionStorage memory-weight based (rather than count based) so large transactions can no longer use an undue amount of memory.
* Code review: formatting and legibility
* Fix stupid type cast error
* More formatting
Introduced DigitalSignatureWithCert and SignedDataWithCert as internal APIs, with the expectation that they will become public; renamed the network parameters end-point to network-parameters; updated the network-map.rst doc; and did some refactoring.
* add foreign key names and move the participants mapping to the subclass so that the table name can be configured
* update api-current file
* fix compilation errors
* PR changes
* PR changes
* Raft notaries can share a single key pair for the service identity (in contrast to a shared composite public key, and individual signing key pairs). This allows adjusting the cluster size on the fly.
* Added test cases covering encrypted password usage
* Renamed UserAuthServiceTests as AuthDBTests: the integration tests checking user credentials loaded from external database (still limited to H2 in-memory for now).
* Some internal renamings
Add functions for constructing `FlowLogicRef` from class name, rather than requiring the class itself. This avoids requiring that schedulable states have access to the scheduled flow to instantiate, but instead can require it only actually scheduling the flow. This reduces the size of the JAR required to validate transactions containing these states.
Using the --just-generate-node-info flag for the notary nodes so that their identities can be submitted to the network map server, which does the network parameters generation.
With network parameters the CN is no longer needed to identify notaries. This frees it up to be used in the node's name alongside the other attributes.
Also, the identity generation logic has been simplified, removing the need to have magic string values for storing distributed identities in the keystore. Now there are just two alias prefixes: "identity" as it was previously, and "distributed-notary".
* Add roles to X509 certificates so that the identity service can always determine which certificate in a hierarchy is the well known identity
* Rename CLIENT_CA certificate type to NODE_CA
* Rename DOORMAN role to INTERMEDIATE_CA
* Correct issue in CashTests where instead of providing a well known identity to generateSpend(), a confidential identity was passed in and a confidential identity generated from it.
* Enforce role hierarchy in PKI
* Enforce that party certificates must be well known or confidential identities
* Add network map certificate role
Copying of the node-info files moved out of Cordform and into NetworkParametersGenerator (which is now called NetworkBootstrapper). This class becomes an external tool to enable deployment of nodes in a test setup on a single filesystem.
* * Document TestIdentity entropy and enforce that it actually works
* Ledger/transaction DSL default notary with fresh key
* MockServices default identity with fresh key
* makeTestIdentityService now takes vararg
* Require cordappPackages for MockServices
* DSL automatic serialization init
* Improve error when two MockNetworks used
* * Make cordappPackages required by MockNetwork
* Default identity service in MockServices
* Make notarySpecs Java-friendly
* Able to send hand coded messages to an Artemis node inbox
Get startup race condition fixed. Start cleanup work.
Fixup after rebase
Remove SASL hack for now
Minor tweaks. Enable AMQP mode manually.
Add configuration control
Slight clean up
Stop timeouts that don't work with AMQP
Rename class
Get TLS constants from :node-api
Primitive integration test
Put back commented line
Session per bridge to alow rollback on remote rejects.
Add more tests and handle multiple IP adddresses
Reduce logging
Fixup after rebase
Add a test to verify the remote end AMQP rejection logic works and does cause message replay.
Allow Artemis to duplicate after session rollback
Reduce number of threads
Move legacy bridge related code over to CoreBridgeManager
Shared threadpool for bridges
Add a test to confirm that no side effects when using a shared thread pool.
Address PR comments and remove dead lines
Rebase and add some comments
Remove a couple of blank lines
Ensure AMQP bridges are used in tests
Fixup after removal of testNodeConfiguration
Add a couple of doc comments
Add a couple of doc comments
Make things internal and use CordaFuture
Address some PR comments
Change comment type
* Use Artemis 2.2 to fix AMQP problems. Add explicit test of legacy core bridges, as marking the factory class private had silently broken them.
* Fix change due to using Artemis 2.2
* Take maximum message size from network parameters
* Add epoch handling
* Add handling of network parameters mismatch
Change NetworkMapClient and updater, add handle in
AbstractNode that results in node shutdown on parameters mismatch. Later
on we should implement proper handling of parameters updates.
Add tests of NetworkParameters wiring.
When node starts with compatibilityZone url configured it takes
networkParameters from the networkMap.
* Permit only one network parameters file
On node startup network parameters are read from node's base directory,
we permit only zero or one files to be there. If network map server is
configured the parameters can be downloaded at startup (if not present
in the directory already).
* Update docs on network map endpoints
* SignedNetworkMap verification fix
SignedNetworkMap verification should also include cert path validation,
which was probably moved away by accident, because docs say about the
exception CertPathValidatorException.
* Extend unit test on RPCSecurityManager
* Fix corner cases in permission parsing and bug in tryAuthenticate
* Rework docsite page
* Add missing ChangeLog entry
* Add support for external data source of access control data (RPC/Shell users credential and permissions), with optional in-memory caching.
* Support password encoded with Apache Shiro fully reversible Modular Crypt Format.
* Introduce 'security' field in Node configuration and related docsite page.
* JMX Jolokia instrumentation WIP (driverDSL, webserver, cordformation, hibernate statistics, access policy config file hardening)
* Cordformation changes to support jolokia agent instrumentation at JVM startup.
* Minor updates to reflect usage of Jolokia 1.3.7 (which uses slightly different .war naming)
* Use relative path reference in -javaagent to prevent problem with long path names with spaces.
* Fixed incorrect regex pattern and added assertion to test.
* Enable JMX monitoring.
* Reporting of Hibernate JMX statistics is configurable (by default, only switched on in devMode)
* Make Artemis JMX enablement configurable.
* Re-instate banning of java serialization.
* Improve JUnit.
* Fixes following rebase from master.
* Re-instated correct regex for picking up Jolokia agent jar.
* Fixed broken integration test.
* Updated documentation
* Updated following PR review feedback.
* Fixed compilation error caused by change in DriverDSL argument type.
* Fixed compilation error caused by change in DriverDSL argument type.
* Fail fast if jolokia-agent-jvm.jar is not located.
* Applied changes in cordformation following review feedback from CA.
* Rename certificate types
* Create separate certificate type for confidential identities
* Add name constraints to dev node CA
* Move dev node CA into getTestPartyAndCertificate()
This removes any need for the user implement and override types from the
super class
* CORDA-786 - Docs update
* CORDA-786 - Remove unneeded second annotation on the proxy objects
* Fix merge conflicts
* Remove MockServices.stateMachineRecordedTransactionMapping which does nothing
* Inline StateLoaderImpl
* Remove unused MockServices
* MockServices well-known identities not needed in a place
* A few things don't need a full-blown ServiceHub
* Generate networkParameteres for Cordformation.
Fix deployNodes task in Cordformation to generate NetworkParameters before running the nodes.
Add TestNetworkParametersGenerator utility loaded after node infos generation step.
* Get rid of bouncy castle provider dependency
For cordform-common. It caused problems with loading our custom
X509EdDSAEngine for generation of network parameters in deployNodes
task.
* new network map object for network map, and verify signature and root in Signed network map and node info
* fixup after rebase
* * added certificate and key to network map server
* move DigitalSignature.WithCert back to NetworkMap.kt, as its breaking API test, will raise another PR to move it back.
* Make DigitalSignature.WithCert not extend WithKey, as per PR discussion.
* various fixes after rebase.
* move Network map back to core/node, as its breaking API test
* revert unintended changes
* move network map objects to node-api
1. The runRPCCashIssue and runWebCashIssue gradle tasks didn't work because they were using the wrong ports
2. Notary lookup was failing because the lookup name didn't include the correct CN for the notary name (this slipped through when reverting the network parameters)
The ports change occurred in #1922 which was attempting the fix the runIssuer gradle task. This is actually a misleading and redundant task as all it does is start up the nodes, which is what the documented deployNodes already does. The ports runIssuer allocated to the nodes were different to the ones specified in deployNodes.
To make sure we have integration tests which closely match deployNodes, the BoC demo has been updated to make use of CordformDefinition. This keeps the node definitions in one place, removing the need to have disparate files in sync. runIssuer has been removed.
* make node info file copying optional by setting "compatabilityZoneURL" in driver
integration test for node using http network map using driver
some bug fixes
* rebase to feature branch and fixup
* add initialRegistration flag to driver
* remove useFileBaseNetworkMap flag, add network map server to DriverTest
* remove useFileBaseNetworkMap flag, add network map server to DriverTest
* use PortAllocation.Incremental instead of random
* * use PortAllocation.Incremental instead of random
* fix NodeInfoWatcher thread leak issue
* reset scheduler before create notary
* move port allocation out of companion object
* move port allocation out of companion object
* make node info file copier lateinit to avoid observable thread pool get created on init
* Revert "CORDA-296: added rpc that returns an observable for node state (#2004)"
This reverts commit 7d1f7ab
* Revert "CORDA-296: added rpc that returns an observable for node state (#2004)"
This reverts commit 7d1f7ab
Previously when de-anonymising a Party instance, the name of the Party was used rather than
the key, meaning a Party could be constructed with a random nonsense key and any name, and be treated as corresponding to the well known identity. This is not a security hole in itself as
in any real scenario a party shouldn't be trusted without having been registered, it creates
a significant risk of a security hole depending on how trusted the anonymous identity is, and
the returned identity is considered.
Pass notary identity into flow in `NodeStatePersistenceTests` rather than resolving it from the network map cache, which avoids a race condition between the flow starting and the notary registration being sent to the cache.
* consistent storage of Issuer Reference using `ByteArray` Kotlin type in Schema definition and a custom Hibernate Type to map this to a VARBINARY database type.
Creation of a new Issued type now also validates maximum size permissible (512).
* Clean up identities in CashTests so that the mini/mega corp keys and identities are correctly
paired together throughout. Previously `miniCorpServices` presented the same key as the
MegaCorp identity, but with the name Mini Corp attached.
* Correct key/name matches in VaultWithCashTest
* Split services in CashTests to not have multiple identities per service hub
* Removal of transaction contract state as BLOB in VaultStates table.
Transaction contract state now resolved using StateLoader (from DBTransactionStorage).
Fixed broken JUnits.
* Changes to address review comments by RP
Address logic error.
* Fixed failing JUnit (CashExitFlowTests.exit zero cash).
* Fix VaultQueryTests to respect transaction visibility boundaries.
* Adopt consistent use of "session" using DatabaseTransactionManager.
* Removed redundant transaction demarcation boundaries in Vault Query tests.
* Added explicit table names (mostly for join tables).
* Shorten or alter name of 2 tables backing notaries.
* Change a compound index declaration to one column index.
Detangle issuers in NodeVaultServiceTest, which has been using two keys on a single node, for different
issuing identities, which doesn't work right now because we don't actually have multiple identity support.
Also remove duplicate data in parameters to fillWithSomeTestCash().
Full requires the command to be `closed` before it can be removed from
the log during compaction. We don't close commands, thus and retaining
the log forever, allowing users to increase the cluster size. The
downsize is the unbounded growth of the size of the log.
Cluster membership changes need testing.
* Network map cache using Network map client instead of artemis. -- WIP
* fix up after rebase
* address PR issues, split network map update test, added todos to remove sleeps
* move jimfs and baseDir to field variable
Ensure when registering identities, we store the certificate closest to the trust root, with the same name. This ensures that looking up an identity by name produces the best match, not earliest registered identity (often but not necessarily the same).
* CORDA-296: added rpc that returns an observable for node state; used to let rpc clients know that the know is about to shut down
* replaced node shut down observation String with enum
Most uses where with MockNetwork which recently got a defaultNotaryIdentity property for dealing with the default single notary case. The remaining uses where in flows.
* Extracted out ShutdownManager into its own file
* Moved RPCDriver and ProcessUtilities into internal package
* Made n.c.testing.performance package internal
network-parameters file read in by the node at startup, of which only the list of notaries is used. For now, the driver and MockNetwork have been updated to require notaries to be started first. This is so that the same set of network parameters can be defined for all the nodes.
CN in the legal name is not longer disallowed since it's no longer reserved for distributed notary names.
Single-node notaries now only have one identity, their main identity. Nodes part of a cluster continue to have two.
(Based off Kasia's work)
* [CORDA-446] Clean up other mentions of network map node and logic
* Rename AbstractNetworkMapService to NetworkMapService and remove the empty NetworkMapService
* fix build
* fix artemismessaging tests
* pr comments
approach which assumes a dedicated node for observers: states that are
reported to the node will appear in the database and update feeds as
normal. Apps that expect all updates to be relevant to themselves may
need adjusting if they run on an observer node too, but this is likely
to be rare.
* CORDA-351: force update dependencies and suppress vulnerabilities not affecting corda
* CORDA-351: force update dependencies and suppress vulnerabilities not affecting corda
* Standardisation of Public Keys in Schema entities. (#68)
* Standardisation in usage of Public Keys in Schema entities.
Use PK Hash where optimal, otherwise use ByteArray/LOB representation of PK.
* Redundant after rebase.
* Use .encoded and Crypto.decode<Public|Private>Key(bytes) instead of Corda serialization.
* Optimize DBPartyAndCertificate entity to store and query on ownerKeyHash.
* Updated API stability check for schema attribute change.
* [CORDA-442] make MockNetwork not start a networkmap node
Now MockNetwork will put the appropriate NodeInfos inside each running node networkMapCache.
Tests relating to networkmap node starting and interaction have been removed since they where relaying on MockNetwork
They won't be reported as properties by the introspector and thus we
will fail to find a constructor for them. This makes sense as we will be
unable to serialise an object whose members we cannot read
