Add attachement demo and documentation

Add a demo of attachments on transactions as a worked example for others to use, along with documentation
on how to run it, and how it functions.

build.gradle

@@ -124,6 +124,15 @@ dependencies {
 }
 
 // Package up the demo programs.
+
+task getAttachmentDemo(type: CreateStartScripts) {
+    mainClassName = "com.r3corda.demos.attachment.AttachmentDemoKt"
+    applicationName = "attachment-demo"
+    defaultJvmOpts = ["-javaagent:${configurations.quasar.singleFile}"]
+    outputDir = new File(project.buildDir, 'scripts')
+    classpath = jar.outputs.files + project.configurations.runtime
+}
+
 task getRateFixDemo(type: CreateStartScripts) {
     mainClassName = "com.r3corda.demos.RateFixDemoKt"
     applicationName = "get-rate-fix"
@@ -193,6 +202,7 @@ tasks.withType(Test) {
 quasarScan.dependsOn('classes', 'core:classes', 'contracts:classes', 'node:classes')
 
 applicationDistribution.into("bin") {
+    from(getAttachmentDemo)
     from(getRateFixDemo)
     from(getIRSDemo)
     from(getTraderDemo)

docs/source/index.rst

@@ -51,6 +51,7 @@ Read on to learn:
    tutorial-clientrpc-api
    protocol-state-machines
    oracles
+   tutorial-attachments
    event-scheduling
    secure-coding-guidelines
 

docs/source/running-the-demos.rst

@@ -143,4 +143,34 @@ Now open your web browser to this URL:
 
 To use the demos click the "Create Deal" button, fill in the form, then click the "Submit" button. Now you will be
 able to use the time controls at the top left of the home page to run the fixings. Click any individual trade in the
-blotter to view it.
+blotter to view it.
+
+
+
+Attachment demo
+----------------
+
+Open two terminals, and in the first run:
+
+**Windows**::
+
+    gradlew.bat & .\build\install\r3prototyping\bin\attachment-demo --role=RECIPIENT
+
+**Other**::
+
+    ./gradlew installDist && ./build/install/r3prototyping/bin/attachment-demo --role=RECIPIENT
+
+It will compile things, if necessary, then create a directory named attachment-demo/buyer with a bunch of files inside and
+start the node. You should see it waiting for a trade to begin.
+
+In the second terminal, run:
+
+**Windows**::
+
+    .\build\install\r3prototyping\bin\attachment-demo --role=SENDER
+
+**Other**::
+
+    ./build/install/r3prototyping/bin/attachment-demo --role=SENDER
+
+You should see some log lines scroll past, and within a few seconds the message "File received - we're happy!" should be printed.

docs/source/tutorial-attachments.rst

@@ -0,0 +1,99 @@
+.. highlight:: kotlin
+.. raw:: html
+
+Using attachments
+=================
+
+Attachments are (typically large) Zip/Jar files referenced within a transaction, but not included in the transaction
+itself. These files can be requested from the originating node as needed, although in many cases will be cached within
+nodes already. Examples include:
+
+* Contract executable code
+* Metadata about a transaction, such as PDF version of an invoice being settled
+* Shared information to be permanently recorded on the ledger
+
+To add attachments the file must first be added to the node's storage service using ``StorageService.importAttachment()``,
+which returns a unique ID that can be added using ``TransactionBuilder.addAttachment()``. Attachments can also be
+uploaded and downloaded via HTTP, to enable integration with external systems. For instructions on HTTP upload/download
+please see ":doc:`node-administration`".
+
+Normally attachments on transactions are fetched automatically via the ``ResolveTransactionsProtocol`` when verifying
+received transactions. Attachments are needed in order to validate a transaction (they include, for example, the
+contract code), so must be fetched before the validation process can run. ``ResolveTransactionsProtocol`` calls
+``FetchTransactionsProtocol`` to perform the actual retrieval.
+
+It is encouraged that where possible attachments are reusable data, so that nodes can meaningfully cache them.
+
+Attachments demo
+----------------
+
+There is a worked example of attachments, which relays a simple document from one node to another. The "two party
+trade protocol" also includes an attachment, however it is a significantly more complex demo, and less well suited
+for a tutorial.
+
+The demo code is in the file "src/main/kotlin/com/r3corda/demos/attachment/AttachmentDemo.kt", with the core logic
+contained within the two functions ``runRecipient()`` and ``runSender()``. We'll look at the recipient function first;
+this subscribes to notifications of new validated transactions, and if it receives a transaction containing attachments,
+loads the first attachment from storage, and checks it matches the expected attachment ID. ``ResolveTransactionsProtocol``
+has already fetched all attachments from the remote node, and as such the attachments are available from the node's
+storage service. Once the attachment is verified, the node shuts itself down.
+
+.. sourcecode:: kotlin
+
+    private fun runRecipient(node: Node) {
+        val serviceHub = node.services
+
+        // Normally we would receive the transaction from a more specific protocol, but in this case we let [FinalityProtocol]
+        // handle receiving it for us.
+        serviceHub.storageService.validatedTransactions.updates.subscribe { event ->
+            // When the transaction is received, it's passed through [ResolveTransactionsProtocol], which first fetches any
+            // attachments for us, then verifies the transaction. As such, by the time it hits the validated transaction store,
+            // we have a copy of the attachment.
+            val tx = event.tx
+            if (tx.attachments.isNotEmpty()) {
+                val attachment = serviceHub.storageService.attachments.openAttachment(tx.attachments.first())
+                assertEquals(PROSPECTUS_HASH, attachment?.id)
+
+                println("File received - we're happy!\n\nFinal transaction is:\n\n${Emoji.renderIfSupported(event.tx)}")
+                thread {
+                    node.stop()
+                }
+            }
+        }
+    }
+
+The sender correspondingly builds a transaction with the attachment, then calls ``FinalityProtocol`` to complete the
+transaction and send it to the recipient node:
+
+
+.. sourcecode:: kotlin
+
+    private fun runSender(node: Node, otherSide: Party) {
+        val serviceHub = node.services
+        // Make sure we have the file in storage
+        if (serviceHub.storageService.attachments.openAttachment(PROSPECTUS_HASH) == null) {
+            com.r3corda.demos.Role::class.java.getResourceAsStream("bank-of-london-cp.jar").use {
+                val id = node.storage.attachments.importAttachment(it)
+                assertEquals(PROSPECTUS_HASH, id)
+            }
+        }
+
+        // Create a trivial transaction that just passes across the attachment - in normal cases there would be
+        // inputs, outputs and commands that refer to this attachment.
+        val ptx = TransactionType.General.Builder(notary = null)
+        ptx.addAttachment(serviceHub.storageService.attachments.openAttachment(PROSPECTUS_HASH)!!.id)
+
+        // Despite not having any states, we have to have at least one signature on the transaction
+        ptx.signWith(ALICE_KEY)
+
+        // Send the transaction to the other recipient
+        val tx = ptx.toSignedTransaction()
+        serviceHub.startProtocol(LOG_SENDER, FinalityProtocol(tx, emptySet(), setOf(otherSide))).success {
+            thread {
+                Thread.sleep(1000L) // Give the other side time to request the attachment
+                node.stop()
+            }
+        }.failure {
+            println("Failed to relay message ")
+        }
+    }

src/main/kotlin/com/r3corda/demos/TraderDemo.kt

@@ -46,7 +46,7 @@ import kotlin.test.assertEquals
 
 // TRADING DEMO
 //
-// Please see docs/build/html/running-the-trading-demo.html
+// Please see docs/build/html/running-the-demos.html
 //
 // This program is a simple driver for exercising the two party trading protocol. Until Corda has a unified node server
 // programs like this are required to wire up the pieces and run a demo scenario end to end.

src/main/kotlin/com/r3corda/demos/attachment/AttachmentDemo.kt

@@ -0,0 +1,208 @@
+package com.r3corda.demos.attachment
+
+import com.google.common.net.HostAndPort
+import com.google.common.util.concurrent.ListenableFuture
+import com.r3corda.contracts.testing.fillWithSomeTestCash
+import com.r3corda.core.contracts.Amount
+import com.r3corda.core.contracts.DOLLARS
+import com.r3corda.core.contracts.TransactionType
+import com.r3corda.core.crypto.Party
+import com.r3corda.core.crypto.SecureHash
+import com.r3corda.core.crypto.sha256
+import com.r3corda.core.failure
+import com.r3corda.core.logElapsedTime
+import com.r3corda.core.node.services.ServiceInfo
+import com.r3corda.core.node.services.ServiceType
+import com.r3corda.core.success
+import com.r3corda.core.transactions.SignedTransaction
+import com.r3corda.core.utilities.Emoji
+import com.r3corda.core.utilities.LogHelper
+import com.r3corda.node.internal.Node
+import com.r3corda.node.services.api.AbstractNodeService
+import com.r3corda.node.services.config.NodeConfiguration
+import com.r3corda.node.services.config.NodeConfigurationFromConfig
+import com.r3corda.node.services.messaging.NodeMessagingClient
+import com.r3corda.node.services.network.NetworkMapService
+import com.r3corda.node.services.persistence.NodeAttachmentService
+import com.r3corda.node.services.transactions.SimpleNotaryService
+import com.r3corda.node.utilities.databaseTransaction
+import com.r3corda.protocols.FinalityProtocol
+import com.r3corda.testing.ALICE_KEY
+import joptsimple.OptionParser
+import org.bouncycastle.cms.Recipient
+import org.slf4j.Logger
+import org.slf4j.LoggerFactory
+import java.nio.file.Paths
+import java.util.*
+import kotlin.concurrent.thread
+import kotlin.system.exitProcess
+import kotlin.test.assertEquals
+
+// ATTACHMENT DEMO
+//
+// Please see docs/build/html/running-the-demos.html and docs/build/html/tutorial-attachments.html
+//
+// This program is a simple demonstration of sending a transaction with an attachment from one node to another, and
+// then accessing the attachment on the remote node.
+//
+// The different roles in the scenario this program can adopt are:
+
+enum class Role(val legalName: String, val port: Int) {
+    SENDER("Bank A", 31337),
+    RECIPIENT("Bank B", 31340);
+
+    val other: Role
+        get() = when (this) {
+            SENDER -> RECIPIENT
+            RECIPIENT -> SENDER
+        }
+}
+
+// And this is the directory under the current working directory where each node will create its own server directory,
+// which holds things like checkpoints, keys, databases, message logs etc.
+val DEFAULT_BASE_DIRECTORY = "./build/attachment-demo"
+
+val LOG_RECIPIENT = "demo.recipient"
+val LOG_SENDER = "demo.sender"
+
+val PROSPECTUS_HASH = SecureHash.parse("decd098666b9657314870e192ced0c3519c2c9d395507a238338f8d003929de9")
+
+private val log: Logger = LoggerFactory.getLogger("AttachmentDemo")
+
+fun main(args: Array<String>) {
+    val parser = OptionParser()
+
+    val roleArg = parser.accepts("role").withRequiredArg().ofType(Role::class.java).required()
+    val myNetworkAddress = parser.accepts("network-address").withRequiredArg().defaultsTo("localhost")
+    val theirNetworkAddress = parser.accepts("other-network-address").withRequiredArg().defaultsTo("localhost")
+    val apiNetworkAddress = parser.accepts("api-address").withRequiredArg().defaultsTo("localhost")
+    val baseDirectoryArg = parser.accepts("base-directory").withRequiredArg().defaultsTo(DEFAULT_BASE_DIRECTORY)
+
+    val options = try {
+        parser.parse(*args)
+    } catch (e: Exception) {
+        log.error(e.message)
+        printHelp(parser)
+        exitProcess(1)
+    }
+
+    val role = options.valueOf(roleArg)!!
+
+    val myNetAddr = HostAndPort.fromString(options.valueOf(myNetworkAddress)).withDefaultPort(role.port)
+    val theirNetAddr = HostAndPort.fromString(options.valueOf(theirNetworkAddress)).withDefaultPort(role.other.port)
+    val apiNetAddr = HostAndPort.fromString(options.valueOf(apiNetworkAddress)).withDefaultPort(myNetAddr.port + 1)
+
+    val baseDirectory = options.valueOf(baseDirectoryArg)!!
+
+    // Suppress the Artemis MQ noise, and activate the demo logging.
+    //
+    // The first two strings correspond to the first argument to StateMachineManager.add() but the way we handle logging
+    // for protocols will change in future.
+    LogHelper.setLevel("+${LOG_RECIPIENT}", "+${LOG_SENDER}", "-org.apache.activemq")
+
+    val directory = Paths.get(baseDirectory, role.name.toLowerCase())
+    log.info("Using base demo directory $directory")
+
+    // Override the default config file (which you can find in the file "reference.conf") to give each node a name.
+    val config = run {
+        val myLegalName = role.legalName
+        NodeConfigurationFromConfig(NodeConfiguration.loadConfig(directory, allowMissingConfig = true, configOverrides = mapOf("myLegalName" to myLegalName)))
+    }
+
+    // Which services will this instance of the node provide to the network?
+    val advertisedServices: Set<ServiceInfo>
+
+    // One of the two servers needs to run the network map and notary services. In such a trivial two-node network
+    // the map is not very helpful, but we need one anyway. So just make the recipient side run the network map as it's
+    // the side that sticks around waiting for the sender.
+    val networkMapId = if (role == Role.SENDER) {
+        advertisedServices = setOf(ServiceInfo(NetworkMapService.Type), ServiceInfo(SimpleNotaryService.Type))
+        null
+    } else {
+        advertisedServices = emptySet()
+        NodeMessagingClient.makeNetworkMapAddress(theirNetAddr)
+    }
+
+    // And now construct then start the node object. It takes a little while.
+    val node = logElapsedTime("Node startup", log) {
+        Node(myNetAddr, apiNetAddr, config, networkMapId, advertisedServices).setup().start()
+    }
+
+    // What happens next depends on the role. The recipient sits around waiting for a transaction. The sender role
+    // will contact the recipient and actually make something happen.
+    when (role) {
+        Role.RECIPIENT -> runRecipient(node)
+        Role.SENDER -> {
+            node.networkMapRegistrationFuture.success {
+                // Pause a moment to give the network map time to update
+                Thread.sleep(100L)
+                val party = node.netMapCache.getNodeByLegalName(Role.RECIPIENT.legalName)?.identity ?: throw IllegalStateException("Cannot find other node?!")
+                runSender(node, party)
+            }
+        }
+    }
+
+    node.run()
+}
+
+private fun runRecipient(node: Node) {
+    val serviceHub = node.services
+
+    // Normally we would receive the transaction from a more specific protocol, but in this case we let [FinalityProtocol]
+    // handle receiving it for us.
+    serviceHub.storageService.validatedTransactions.updates.subscribe { event ->
+        // When the transaction is received, it's passed through [ResolveTransactionsProtocol], which first fetches any
+        // attachments for us, then verifies the transaction. As such, by the time it hits the validated transaction store,
+        // we have a copy of the attachment.
+        val tx = event.tx
+        if (tx.attachments.isNotEmpty()) {
+            val attachment = serviceHub.storageService.attachments.openAttachment(tx.attachments.first())
+            assertEquals(PROSPECTUS_HASH, attachment?.id)
+
+            println("File received - we're happy!\n\nFinal transaction is:\n\n${Emoji.renderIfSupported(event.tx)}")
+            thread {
+                node.stop()
+            }
+        }
+    }
+}
+
+private fun runSender(node: Node, otherSide: Party) {
+    val serviceHub = node.services
+    // Make sure we have the file in storage
+    // TODO: We should have our own demo file, not share the trader demo file
+    if (serviceHub.storageService.attachments.openAttachment(PROSPECTUS_HASH) == null) {
+        com.r3corda.demos.Role::class.java.getResourceAsStream("bank-of-london-cp.jar").use {
+            val id = node.storage.attachments.importAttachment(it)
+            assertEquals(PROSPECTUS_HASH, id)
+        }
+    }
+
+    // Create a trivial transaction that just passes across the attachment - in normal cases there would be
+    // inputs, outputs and commands that refer to this attachment.
+    val ptx = TransactionType.General.Builder(notary = null)
+    ptx.addAttachment(serviceHub.storageService.attachments.openAttachment(PROSPECTUS_HASH)!!.id)
+
+    // Despite not having any states, we have to have at least one signature on the transaction
+    ptx.signWith(ALICE_KEY)
+
+    // Send the transaction to the other recipient
+    val tx = ptx.toSignedTransaction()
+    serviceHub.startProtocol(LOG_SENDER, FinalityProtocol(tx, emptySet(), setOf(otherSide))).success {
+        thread {
+            Thread.sleep(1000L) // Give the other side time to request the attachment
+            node.stop()
+        }
+    }.failure {
+        println("Failed to relay message ")
+    }
+}
+
+private fun printHelp(parser: OptionParser) {
+    println("""
+    Usage: attachment-demo --role [RECIPIENT|SENDER] [options]
+    Please refer to the documentation in docs/build/index.html for more info.
+
+    """.trimIndent())
+    parser.printHelpOn(System.out)
+}