From 08decac922b29c893d588e616918e4edd5512c35 Mon Sep 17 00:00:00 2001 From: Rick Parker Date: Fri, 2 Mar 2018 15:36:54 +0000 Subject: [PATCH 1/5] CORDA-1144 Fix documentation that refers to IRS demo code (#2711) --- docs/source/event-scheduling.rst | 2 +- docs/source/oracles.rst | 12 ++++++------ docs/source/tutorial-tear-offs.rst | 2 +- 3 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/source/event-scheduling.rst b/docs/source/event-scheduling.rst index 3ba100a3ef..ce77035b8c 100644 --- a/docs/source/event-scheduling.rst +++ b/docs/source/event-scheduling.rst @@ -70,7 +70,7 @@ Let's take an example of the interest rate swap fixings for our scheduled events .. container:: codeset - .. literalinclude:: ../../samples/irs-demo/src/main/kotlin/net/corda/irs/contract/IRS.kt + .. literalinclude:: ../../samples/irs-demo/cordapp/src/main/kotlin/net/corda/irs/contract/IRS.kt :language: kotlin :start-after: DOCSTART 1 :end-before: DOCEND 1 diff --git a/docs/source/oracles.rst b/docs/source/oracles.rst index 802e9f33c0..7c1fa36f53 100644 --- a/docs/source/oracles.rst +++ b/docs/source/oracles.rst @@ -166,7 +166,7 @@ parameter and ``CommandData`` classes. Let's see how the ``sign`` method for ``NodeInterestRates.Oracle`` is written: -.. literalinclude:: ../../samples/irs-demo/src/main/kotlin/net/corda/irs/api/NodeInterestRates.kt +.. literalinclude:: ../../samples/irs-demo/cordapp/src/main/kotlin/net/corda/irs/api/NodeInterestRates.kt :language: kotlin :start-after: DOCSTART 1 :end-before: DOCEND 1 @@ -192,7 +192,7 @@ Binding to the network The first step is to create the oracle as a service by annotating its class with ``@CordaService``. Let's see how that's done: -.. literalinclude:: ../../samples/irs-demo/src/main/kotlin/net/corda/irs/api/NodeInterestRates.kt +.. literalinclude:: ../../samples/irs-demo/cordapp/src/main/kotlin/net/corda/irs/api/NodeInterestRates.kt :language: kotlin :start-after: DOCSTART 3 :end-before: DOCEND 3 @@ -201,7 +201,7 @@ done: The Corda node scans for any class with this annotation and initialises them. The only requirement is that the class provide a constructor with a single parameter of type ``ServiceHub``. -.. literalinclude:: ../../samples/irs-demo/src/main/kotlin/net/corda/irs/api/NodeInterestRates.kt +.. literalinclude:: ../../samples/irs-demo/cordapp/src/main/kotlin/net/corda/irs/api/NodeInterestRates.kt :language: kotlin :start-after: DOCSTART 2 :end-before: DOCEND 2 @@ -219,7 +219,7 @@ We mentioned the client sub-flow briefly above. They are the mechanism that cli use to interact with your oracle. Typically there will be one for querying and one for signing. Let's take a look at those for ``NodeInterestRates.Oracle``. -.. literalinclude:: ../../samples/irs-demo/src/main/kotlin/net/corda/irs/flows/RatesFixFlow.kt +.. literalinclude:: ../../samples/irs-demo/cordapp/src/main/kotlin/net/corda/irs/flows/RatesFixFlow.kt :language: kotlin :start-after: DOCSTART 1 :end-before: DOCEND 1 @@ -238,7 +238,7 @@ The oracle is invoked through sub-flows to query for values, add them to the tra the transaction signed by the oracle. Following on from the above examples, this is all encapsulated in a sub-flow called ``RatesFixFlow``. Here's the ``call`` method of that flow. -.. literalinclude:: ../../samples/irs-demo/src/main/kotlin/net/corda/irs/flows/RatesFixFlow.kt +.. literalinclude:: ../../samples/irs-demo/cordapp/src/main/kotlin/net/corda/irs/flows/RatesFixFlow.kt :language: kotlin :start-after: DOCSTART 2 :end-before: DOCEND 2 @@ -255,7 +255,7 @@ As you can see, this: Here's an example of it in action from ``FixingFlow.Fixer``. -.. literalinclude:: ../../samples/irs-demo/src/main/kotlin/net/corda/irs/flows/FixingFlow.kt +.. literalinclude:: ../../samples/irs-demo/cordapp/src/main/kotlin/net/corda/irs/flows/FixingFlow.kt :language: kotlin :start-after: DOCSTART 1 :end-before: DOCEND 1 diff --git a/docs/source/tutorial-tear-offs.rst b/docs/source/tutorial-tear-offs.rst index fe665bf3bf..0e5ae6e970 100644 --- a/docs/source/tutorial-tear-offs.rst +++ b/docs/source/tutorial-tear-offs.rst @@ -43,7 +43,7 @@ transaction components is exactly the same. Note that unlike ``WireTransaction`` The following code snippet is taken from ``NodeInterestRates.kt`` and implements a signing part of an Oracle. -.. literalinclude:: ../../samples/irs-demo/src/main/kotlin/net/corda/irs/api/NodeInterestRates.kt +.. literalinclude:: ../../samples/irs-demo/cordapp/src/main/kotlin/net/corda/irs/api/NodeInterestRates.kt :language: kotlin :start-after: DOCSTART 1 :end-before: DOCEND 1 From 91fd46f2c0d5d87e6b409149b040bcaea383a049 Mon Sep 17 00:00:00 2001 From: Joel Dudley Date: Fri, 2 Mar 2018 17:48:45 +0000 Subject: [PATCH 2/5] Updates text of running node shel. --- docs/source/tutorial-cordapp.rst | 35 ++++++++++++++++---------------- 1 file changed, 17 insertions(+), 18 deletions(-) diff --git a/docs/source/tutorial-cordapp.rst b/docs/source/tutorial-cordapp.rst index 503d799045..fbe694d863 100644 --- a/docs/source/tutorial-cordapp.rst +++ b/docs/source/tutorial-cordapp.rst @@ -222,28 +222,27 @@ For each node, the ``runnodes`` script creates a node tab/window: .. sourcecode:: none - ______ __ - / ____/ _________/ /___ _ - / / __ / ___/ __ / __ `/ It's kind of like a block chain but - / /___ /_/ / / / /_/ / /_/ / cords sounded healthier than chains. - \____/ /_/ \__,_/\__,_/ + ______ __ + / ____/ _________/ /___ _ + / / __ / ___/ __ / __ `/ Top tip: never say "oops", instead + / /___ /_/ / / / /_/ / /_/ / always say "Ah, Interesting!" + \____/ /_/ \__,_/\__,_/ - --- Corda Open Source 0.12.1 (da47f1c) ----------------------------------------------- - - 📚 New! Training now available worldwide, see https://corda.net/corda-training/ - - Logs can be found in : /Users/username/Desktop/cordapp-example/kotlin-source/build/nodes/PartyA/logs - Database connection url is : jdbc:h2:tcp://10.163.199.132:54763/node - Listening on address : 127.0.0.1:10005 - RPC service listening on address : localhost:10006 - Loaded plugins : com.example.plugin.ExamplePlugin - Node for "PartyA" started up and registered in 35.0 sec + --- Corda Open Source corda-3.0 (4157c25) ----------------------------------------------- - Welcome to the Corda interactive shell. - Useful commands include 'help' to see what is available, and 'bye' to shut down the node. + Logs can be found in : /Users/joeldudley/Desktop/cordapp-example/kotlin-source/build/nodes/PartyA/logs + Database connection url is : jdbc:h2:tcp://localhost:59472/node + Incoming connection address : localhost:10005 + Listening on port : 10005 + Loaded CorDapps : corda-finance-corda-3.0, cordapp-example-0.1, corda-core-corda-3.0 + Node for "PartyA" started up and registered in 38.59 sec - Fri Jul 07 10:33:47 BST 2017>>> + + Welcome to the Corda interactive shell. + Useful commands include 'help' to see what is available, and 'bye' to shut down the node. + + Fri Mar 02 17:34:02 GMT 2018>>> For every node except the notary, the script also creates a webserver terminal tab/window: From 26fe90c8e9a9b4534ae34804ace2d542a3badead Mon Sep 17 00:00:00 2001 From: Andrzej Cichocki Date: Mon, 5 Mar 2018 10:11:26 +0000 Subject: [PATCH 3/5] CORDA-973 Allow deserialization of any checkpoint/storage encoding. (#2693) --- .../nodeapi/internal/serialization/ServerContexts.kt | 6 ++++-- .../nodeapi/internal/serialization/SharedContexts.kt | 9 ++++++++- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/node-api/src/main/kotlin/net/corda/nodeapi/internal/serialization/ServerContexts.kt b/node-api/src/main/kotlin/net/corda/nodeapi/internal/serialization/ServerContexts.kt index cc8dcfa305..aa939c5002 100644 --- a/node-api/src/main/kotlin/net/corda/nodeapi/internal/serialization/ServerContexts.kt +++ b/node-api/src/main/kotlin/net/corda/nodeapi/internal/serialization/ServerContexts.kt @@ -35,14 +35,16 @@ val KRYO_STORAGE_CONTEXT = SerializationContextImpl(kryoMagic, emptyMap(), true, SerializationContext.UseCase.Storage, - null) + null, + AlwaysAcceptEncodingWhitelist) val AMQP_STORAGE_CONTEXT = SerializationContextImpl(amqpMagic, SerializationDefaults.javaClass.classLoader, AllButBlacklisted, emptyMap(), true, SerializationContext.UseCase.Storage, - null) + null, + AlwaysAcceptEncodingWhitelist) val AMQP_RPC_SERVER_CONTEXT = SerializationContextImpl(amqpMagic, SerializationDefaults.javaClass.classLoader, GlobalTransientClassWhiteList(BuiltInExceptionsWhitelist()), diff --git a/node-api/src/main/kotlin/net/corda/nodeapi/internal/serialization/SharedContexts.kt b/node-api/src/main/kotlin/net/corda/nodeapi/internal/serialization/SharedContexts.kt index 9620b3c999..25977e4c1e 100644 --- a/node-api/src/main/kotlin/net/corda/nodeapi/internal/serialization/SharedContexts.kt +++ b/node-api/src/main/kotlin/net/corda/nodeapi/internal/serialization/SharedContexts.kt @@ -2,8 +2,10 @@ package net.corda.nodeapi.internal.serialization +import net.corda.core.serialization.EncodingWhitelist import net.corda.core.serialization.SerializationContext import net.corda.core.serialization.SerializationDefaults +import net.corda.core.serialization.SerializationEncoding import net.corda.nodeapi.internal.serialization.amqp.amqpMagic import net.corda.nodeapi.internal.serialization.kryo.kryoMagic @@ -28,7 +30,8 @@ val KRYO_CHECKPOINT_CONTEXT = SerializationContextImpl(kryoMagic, emptyMap(), true, SerializationContext.UseCase.Checkpoint, - null) + null, + AlwaysAcceptEncodingWhitelist) val AMQP_P2P_CONTEXT = SerializationContextImpl(amqpMagic, SerializationDefaults.javaClass.classLoader, GlobalTransientClassWhiteList(BuiltInExceptionsWhitelist()), @@ -36,3 +39,7 @@ val AMQP_P2P_CONTEXT = SerializationContextImpl(amqpMagic, true, SerializationContext.UseCase.P2P, null) + +internal object AlwaysAcceptEncodingWhitelist : EncodingWhitelist { + override fun acceptEncoding(encoding: SerializationEncoding) = true +} From 4a73a80b3958705e6336e65ee0bff955f90c4e84 Mon Sep 17 00:00:00 2001 From: Shams Asari Date: Mon, 5 Mar 2018 11:10:38 +0000 Subject: [PATCH 4/5] CORDA-1160: Only read node-info files if their last modified time has changed. (#2717) --- .../net/corda/core/internal/InternalUtils.kt | 10 ++- .../node/services/network/NodeInfoWatcher.kt | 62 +++++++------------ 2 files changed, 32 insertions(+), 40 deletions(-) diff --git a/core/src/main/kotlin/net/corda/core/internal/InternalUtils.kt b/core/src/main/kotlin/net/corda/core/internal/InternalUtils.kt index ac4dabae8d..d3a924dfef 100644 --- a/core/src/main/kotlin/net/corda/core/internal/InternalUtils.kt +++ b/core/src/main/kotlin/net/corda/core/internal/InternalUtils.kt @@ -5,7 +5,6 @@ package net.corda.core.internal import net.corda.core.cordapp.Cordapp import net.corda.core.cordapp.CordappConfig import net.corda.core.cordapp.CordappContext -import net.corda.core.cordapp.CordappProvider import net.corda.core.crypto.* import net.corda.core.flows.NotarisationRequest import net.corda.core.flows.NotarisationRequestSignature @@ -38,6 +37,7 @@ import java.nio.charset.Charset import java.nio.charset.StandardCharsets.UTF_8 import java.nio.file.* import java.nio.file.attribute.FileAttribute +import java.nio.file.attribute.FileTime import java.security.KeyPair import java.security.PrivateKey import java.security.cert.X509Certificate @@ -130,6 +130,7 @@ fun Path.moveTo(target: Path, vararg options: CopyOption): Path = Files.move(thi fun Path.isRegularFile(vararg options: LinkOption): Boolean = Files.isRegularFile(this, *options) fun Path.isDirectory(vararg options: LinkOption): Boolean = Files.isDirectory(this, *options) inline val Path.size: Long get() = Files.size(this) +fun Path.lastModifiedTime(vararg options: LinkOption): FileTime = Files.getLastModifiedTime(this, *options) inline fun Path.list(block: (Stream) -> R): R = Files.list(this).use(block) fun Path.deleteIfExists(): Boolean = Files.deleteIfExists(this) fun Path.reader(charset: Charset = UTF_8): BufferedReader = Files.newBufferedReader(this, charset) @@ -257,6 +258,13 @@ fun IntProgression.stream(parallel: Boolean = false): IntStream = StreamSupport. // When toArray has filled in the array, the component type is no longer T? but T (that may itself be nullable): inline fun Stream.toTypedArray(): Array = uncheckedCast(toArray { size -> arrayOfNulls(size) }) +inline fun Stream.mapNotNull(crossinline transform: (T) -> R?): Stream { + return flatMap { + val value = transform(it) + if (value != null) Stream.of(value) else Stream.empty() + } +} + fun Class.castIfPossible(obj: Any): T? = if (isInstance(obj)) cast(obj) else null /** Returns a [DeclaredField] wrapper around the declared (possibly non-public) static field of the receiver [Class]. */ diff --git a/node/src/main/kotlin/net/corda/node/services/network/NodeInfoWatcher.kt b/node/src/main/kotlin/net/corda/node/services/network/NodeInfoWatcher.kt index 5059276ed5..bbcbb37e89 100644 --- a/node/src/main/kotlin/net/corda/node/services/network/NodeInfoWatcher.kt +++ b/node/src/main/kotlin/net/corda/node/services/network/NodeInfoWatcher.kt @@ -8,6 +8,7 @@ import net.corda.core.serialization.internal.SerializationEnvironmentImpl import net.corda.core.serialization.internal._contextSerializationEnv import net.corda.core.serialization.serialize import net.corda.core.utilities.contextLogger +import net.corda.core.utilities.debug import net.corda.core.utilities.seconds import net.corda.nodeapi.internal.NodeInfoAndSigned import net.corda.nodeapi.internal.SignedNodeInfo @@ -17,13 +18,12 @@ import net.corda.nodeapi.internal.serialization.SerializationFactoryImpl import net.corda.nodeapi.internal.serialization.amqp.AMQPServerSerializationScheme import rx.Observable import rx.Scheduler -import java.io.IOException import java.nio.file.Path import java.nio.file.Paths import java.nio.file.StandardCopyOption.REPLACE_EXISTING +import java.nio.file.attribute.FileTime import java.time.Duration import java.util.concurrent.TimeUnit -import java.util.stream.Stream import kotlin.streams.toList /** @@ -58,20 +58,14 @@ class NodeInfoWatcher(private val nodePath: Path, } } - private val nodeInfoDirectory = nodePath / CordformNode.NODE_INFO_DIRECTORY - + private val nodeInfosDir = nodePath / CordformNode.NODE_INFO_DIRECTORY + private val nodeInfoFiles = HashMap() private val _processedNodeInfoHashes = HashSet() val processedNodeInfoHashes: Set get() = _processedNodeInfoHashes init { require(pollInterval >= 5.seconds) { "Poll interval must be 5 seconds or longer." } - if (!nodeInfoDirectory.isDirectory()) { - try { - nodeInfoDirectory.createDirectories() - } catch (e: IOException) { - logger.info("Failed to create $nodeInfoDirectory", e) - } - } + nodeInfosDir.createDirectories() } /** @@ -93,42 +87,32 @@ class NodeInfoWatcher(private val nodePath: Path, return Companion.saveToFile(nodePath, nodeInfoAndSigned) } - /** - * Loads all the files contained in a given path and returns the deserialized [NodeInfo]s. - * Signatures are checked before returning a value. - * - * @return a list of [NodeInfo]s - */ private fun loadFromDirectory(): List { - if (!nodeInfoDirectory.isDirectory()) { - return emptyList() - } - val result = nodeInfoDirectory.list { paths -> + val result = nodeInfosDir.list { paths -> paths .filter { it.isRegularFile() } - .flatMap { path -> - val nodeInfo = processFile(path)?.let { - if (_processedNodeInfoHashes.add(it.signed.raw.hash)) it.nodeInfo else null + .filter { file -> + val lastModifiedTime = file.lastModifiedTime() + val previousLastModifiedTime = nodeInfoFiles[file] + val newOrChangedFile = previousLastModifiedTime == null || lastModifiedTime > previousLastModifiedTime + nodeInfoFiles[file] = lastModifiedTime + newOrChangedFile + } + .mapNotNull { file -> + logger.debug { "Reading SignedNodeInfo from $file" } + try { + NodeInfoAndSigned(file.readObject()) + } catch (e: Exception) { + logger.warn("Unable to read SignedNodeInfo from $file", e) + null } - if (nodeInfo != null) Stream.of(nodeInfo) else Stream.empty() } .toList() } - if (result.isNotEmpty()) { - logger.info("Successfully read ${result.size} NodeInfo files from disk.") - } - return result - } - private fun processFile(file: Path): NodeInfoAndSigned? { - return try { - logger.info("Reading NodeInfo from file: $file") - val signedNodeInfo = file.readObject() - NodeInfoAndSigned(signedNodeInfo) - } catch (e: Exception) { - logger.warn("Exception parsing NodeInfo from file. $file", e) - null - } + logger.debug { "Read ${result.size} NodeInfo files from $nodeInfosDir" } + _processedNodeInfoHashes += result.map { it.signed.raw.hash } + return result.map { it.nodeInfo } } } From ef703c50be7584a20e0144945f549d7020ce0892 Mon Sep 17 00:00:00 2001 From: Chris Rankin Date: Mon, 5 Mar 2018 12:07:36 +0000 Subject: [PATCH 5/5] Configure buildSrc to be multi-module. (#2658) * Configure buildSrc to be multi-module. * Declare canonicalizer plugin's descriptor via build.gradle. --- buildSrc/build.gradle | 7 +++++- buildSrc/canonicalizer/build.gradle | 22 +++++++++++++++++++ .../main/groovy/CanonicalizerPlugin.groovy | 0 buildSrc/settings.gradle | 2 ++ 4 files changed, 30 insertions(+), 1 deletion(-) create mode 100644 buildSrc/canonicalizer/build.gradle rename buildSrc/{ => canonicalizer}/src/main/groovy/CanonicalizerPlugin.groovy (100%) create mode 100644 buildSrc/settings.gradle diff --git a/buildSrc/build.gradle b/buildSrc/build.gradle index bd4f497c0b..01d3adced1 100644 --- a/buildSrc/build.gradle +++ b/buildSrc/build.gradle @@ -6,11 +6,16 @@ buildscript { } apply plugin: 'maven' +apply plugin: 'java' repositories { + mavenLocal() mavenCentral() } dependencies { - compile "com.google.guava:guava:$guava_version" + // Add the top-level projects ONLY to the host project. + runtime project.childProjects.values().collect { + project(it.path) + } } diff --git a/buildSrc/canonicalizer/build.gradle b/buildSrc/canonicalizer/build.gradle new file mode 100644 index 0000000000..4a78d5f9e6 --- /dev/null +++ b/buildSrc/canonicalizer/build.gradle @@ -0,0 +1,22 @@ +plugins { + id 'groovy' + id 'java-gradle-plugin' +} + +repositories { + mavenLocal() + mavenCentral() +} + +gradlePlugin { + plugins { + canonicalizerPlugin { + id = 'net.corda.plugins.canonicalizer' + implementationClass = 'CanonicalizerPlugin' + } + } +} + +dependencies { + compile "com.google.guava:guava:$guava_version" +} diff --git a/buildSrc/src/main/groovy/CanonicalizerPlugin.groovy b/buildSrc/canonicalizer/src/main/groovy/CanonicalizerPlugin.groovy similarity index 100% rename from buildSrc/src/main/groovy/CanonicalizerPlugin.groovy rename to buildSrc/canonicalizer/src/main/groovy/CanonicalizerPlugin.groovy diff --git a/buildSrc/settings.gradle b/buildSrc/settings.gradle new file mode 100644 index 0000000000..c46d96de90 --- /dev/null +++ b/buildSrc/settings.gradle @@ -0,0 +1,2 @@ +rootProject.name = 'buildSrc' +include 'canonicalizer'