Merge remote-tracking branch 'origin/master'

docs/source/api-flows.rst

@@ -744,3 +744,35 @@ We then update the progress tracker's current step as we progress through the fl
         :start-after: DOCSTART 18
         :end-before: DOCEND 18
         :dedent: 12
+
+
+Concurrency, Locking and Waiting
+--------------------------------
+This is an advanced topic.  Because Corda is designed to:
+
+* run many flows in parallel,
+* may persist flows to storage and resurrect those flows much later,
+* (in the future) migrate flows between JVMs,
+
+flows should avoid use of locks and typically not even attempt to interact with objects shared between flows (except
+``ServiceHub`` and other carefully crafted services such as Oracles.  See :doc:`oracles`).
+Locks will significantly reduce the scalability of the node, in the best case, and can cause the node to deadlock if they
+remain locked across flow context switch boundaries (such as sending and receiving
+from peers discussed above, and the sleep discussed below).
+
+If you need activities that are scheduled, you should investigate the use of ``SchedulableState``.
+However, we appreciate that Corda support for some more advanced patterns is still in the future, and if there is a need
+for brief pauses in flows then you should use ``FlowLogic.sleep`` in place of where you might have used ``Thread.sleep``.
+Flows should expressly not use ``Thread.sleep``, since this will prevent the node from processing other flows
+in the meantime, significantly impairing the performance of the node.
+Even ``FlowLogic.sleep`` is not to be used to create long running flows, since the Corda ethos is for short lived flows
+(otherwise upgrading nodes or CorDapps is much more complicated), or as a substitute to using the ``SchedulableState`` scheduler.
+
+Currently the ``finance`` package uses ``FlowLogic.sleep`` to make several attempts at coin selection, where necessary,
+when many states are soft locked and we wish to wait for those, or other new states in their place, to become unlocked.
+
+    .. literalinclude:: ../../finance/src/main/kotlin/net/corda/finance/contracts/asset/cash/selection/AbstractCashSelection.kt
+        :language: kotlin
+        :start-after: DOCSTART CASHSELECT 1
+        :end-before: DOCEND CASHSELECT 1
+        :dedent: 8

docs/source/corda-configuration-file.rst

@@ -1,64 +1,47 @@
 Node configuration
 ==================
 
+.. contents::
+
 File location
 -------------
+When starting a node, the ``corda.jar`` file defaults to reading the node's configuration from a ``node.conf`` file in
+the directory from which the command to launch Corda is executed. There are two command-line options to override this
+behaviour:
 
-The Corda all-in-one ``corda.jar`` file is generated by the ``gradle buildCordaJAR`` task and defaults to reading configuration
-from a ``node.conf`` file in the present working directory. This behaviour can be overidden using the ``--config-file``
-command line option to target configuration files with different names, or different file location (relative paths are
-relative to the current working directory). Also, the ``--base-directory`` command line option alters the Corda node
-workspace location and if specified a ``node.conf`` configuration file is then expected in the root of the workspace.
+* The ``--config-file`` command line option allows you to specify a configuration file with a different name, or at
+  different file location. Paths are relative to the current working directory
 
-The configuration file templates used for the ``gradle deployNodes`` task are to be found in the ``/config/dev`` folder.
-Also note that there is a basic set of defaults loaded from the built in resource file ``/node/src/main/resources/reference.conf``
-of the ``:node`` gradle module. All properties in this can be overidden in the file configuration and for rarely changed
-properties this defaulting allows the property to be excluded from the configuration file.
+* The ``--base-directory`` command line option allows you to specify the node's workspace location. A ``node.conf``
+  configuration file is then expected in the root of this workspace
+
+If you specify both command line arguments at the same time, the node will fail to start.
 
 Format
 ------
+The Corda configuration file uses the HOCON format which is superset of JSON. Please visit
+`<https://github.com/typesafehub/config/blob/master/HOCON.md>`_ for further details.
 
-The Corda configuration file uses the HOCON format which is superset of JSON. It has several features which makes it
-very useful as a configuration format. Please visit their `page <https://github.com/typesafehub/config/blob/master/HOCON.md>`_
-for further details.
-
-Examples
+Defaults
 --------
+A set of default configuration options are loaded from the built-in resource file ``/node/src/main/resources/reference.conf``.
+This file can be found in the ``:node`` gradle module of the `Corda repository <https://github.com/corda/corda>`_. Any
+options you do not specify in your own ``node.conf`` file will use these defaults.
 
-General node configuration file for hosting the IRSDemo services.
+Here are the contents of the ``reference.conf`` file:
 
-.. literalinclude:: example-code/src/main/resources/example-node.conf
+.. literalinclude:: ../../node/src/main/resources/reference.conf
     :language: javascript
 
-Simple Notary configuration file.
-
-.. parsed-literal::
-
-    myLegalName : "O=Notary Service,OU=corda,L=London,C=GB"
-    keyStorePassword : "cordacadevpass"
-    trustStorePassword : "trustpass"
-    p2pAddress : "localhost:12345"
-    rpcSettings = {
-        useSsl = false
-        standAloneBroker = false
-        address : "my-corda-node:10003"
-        adminAddress : "my-corda-node:10004"
-    }
-    webAddress : "localhost:12347"
-    notary : {
-        validating : false
-    }
-    devMode : true
-    compatibilityZoneURL : "https://cz.corda.net"
-
 Fields
 ------
+The available config fields are listed below. ``baseDirectory`` is available as a substitution value and contains the
+absolute path to the node's base directory.
 
-The available config fields are listed below. ``baseDirectory`` is available as a substitution value, containing the absolute
-path to the node's base directory.
-
-:myLegalName: The legal identity of the node acts as a human readable alias to the node's public key and several demos use
-    this to lookup the NodeInfo.
+:myLegalName: The legal identity of the node. This acts as a human-readable alias to the node's public key and can be used with
+    the network map to look up the node's info. This is the name that is used in the node's certificates (either when requesting them
+    from the doorman, or when auto-generating them in dev mode). At runtime, Corda checks whether this name matches the
+    name in the node's certificates.
 
 :keyStorePassword: The password to unlock the KeyStore file (``<workspace>/certificates/sslkeystore.jks``) containing the
     node certificate and private key.
@@ -195,4 +178,33 @@ path to the node's base directory.
             Otherwise defaults to 10MB
 
 :attachmentCacheBound: Optionally specify how many attachments should be cached locally. Note that this includes only the key and
-            metadata, the content is cached separately and can be loaded lazily. Defaults to 1024.
+            metadata, the content is cached separately and can be loaded lazily. Defaults to 1024.
+
+Examples
+--------
+
+General node configuration file for hosting the IRSDemo services:
+
+.. literalinclude:: example-code/src/main/resources/example-node.conf
+   :language: javascript
+
+Simple notary configuration file:
+
+.. parsed-literal::
+
+    myLegalName : "O=Notary Service,OU=corda,L=London,C=GB"
+    keyStorePassword : "cordacadevpass"
+    trustStorePassword : "trustpass"
+    p2pAddress : "localhost:12345"
+    rpcSettings = {
+        useSsl = false
+        standAloneBroker = false
+        address : "my-corda-node:10003"
+        adminAddress : "my-corda-node:10004"
+    }
+    webAddress : "localhost:12347"
+    notary : {
+        validating : false
+    }
+    devMode : true
+    compatibilityZoneURL : "https://cz.corda.net"

docs/source/node-database.rst

@@ -1,16 +1,14 @@
 Node database
 =============
 
-Currently, nodes store their data in an H2 database. In the future, we plan to support a wide range of databases.
-
-You can connect directly to a running node's database to see its stored states, transactions and attachments as
-follows:
+Default in-memory database
+--------------------------
+By default, nodes store their data in an H2 database. You can connect directly to a running node's database to see its
+stored states, transactions and attachments as follows:
 
 * Download the `h2 platform-independent zip <http://www.h2database.com/html/download.html>`_, unzip the zip, and
   navigate in a terminal window to the unzipped folder
-* Change directories to the bin folder:
-
-  ``cd h2/bin``
+* Change directories to the bin folder: ``cd h2/bin``
 
 * Run the following command to open the h2 web console in a web browser tab:
 
@@ -25,4 +23,34 @@ follows:
 * Paste this string into the JDBC URL field and click ``Connect``, using the default username and password.
 
 You will be presented with a web interface that shows the contents of your node's storage and vault, and provides an
-interface for you to query them using SQL.
+interface for you to query them using SQL.
+
+PostgreSQL
+----------
+Nodes also have untested support for PostgreSQL 9.6, using PostgreSQL JDBC Driver 42.1.4.
+
+.. warning:: This is an experimental community contribution, and is currently untested. We welcome pull requests to add
+   tests and additional support for this feature.
+
+Configuration
+~~~~~~~~~~~~~
+Here is an example node configuration for PostgreSQL:
+
+.. sourcecode:: groovy
+
+    dataSourceProperties = {
+        dataSourceClassName = "org.postgresql.ds.PGSimpleDataSource"
+        dataSource.url = "jdbc:postgresql://[HOST]:[PORT]/postgres"
+        dataSource.user = [USER]
+        dataSource.password = [PASSWORD]
+    }
+    database = {
+        transactionIsolationLevel = READ_COMMITTED
+        schema = [SCHEMA]
+    }
+
+Note that:
+
+* The ``database.schema`` property is optional
+* The value of ``database.schema`` is not wrapped in double quotes and Postgres always treats it as a lower-case value
+  (e.g. ``AliceCorp`` becomes ``alicecorp``)

docs/source/quickstart-index.rst

@@ -1,10 +1,7 @@
 Quickstart
 ==========
 
-.. toctree::
-   :maxdepth: 1
-
-   getting-set-up
-   tutorial-cordapp
-   Other CorDapps <https://www.corda.net/samples/>
-   Utilities <https://www.corda.net/utilities/>
+* :doc:`Set up your machine for CorDapp development <getting-set-up>`
+* :doc:`Run the Example CorDapp <tutorial-cordapp>`
+* `View CorDapps in Corda Explore <http://explore.corda.zone/>`_
+* `Download sample CorDapps <https://www.corda.net/samples/>`_

docs/source/shell.rst

@@ -184,7 +184,7 @@ Starting a flow
 
 We would start the ``CashIssue`` flow as follows:
 
-``flow start CashIssue amount: $1000, issueRef: 1234, recipient: "O=Bank A,L=London,C=GB", notary: "O=Notary Service,OU=corda,L=London,C=GB"``
+``flow start CashIssueFlow amount: $1000, issuerBankPartyRef: 1234, notary: "O=Controller, L=London, C=GB"``
 
 This breaks down as follows:
 

finance/src/main/kotlin/net/corda/finance/contracts/asset/cash/selection/AbstractCashSelection.kt

@@ -12,7 +12,10 @@ import net.corda.core.node.ServiceHub
 import net.corda.core.node.services.StatesNotAvailableException
 import net.corda.core.utilities.*
 import net.corda.finance.contracts.asset.Cash
-import java.sql.*
+import java.sql.Connection
+import java.sql.DatabaseMetaData
+import java.sql.ResultSet
+import java.sql.SQLException
 import java.util.*
 import java.util.concurrent.atomic.AtomicReference
 import java.util.concurrent.locks.ReentrantLock
@@ -101,6 +104,7 @@ abstract class AbstractCashSelection {
                                         withIssuerRefs: Set<OpaqueBytes> = emptySet()): List<StateAndRef<Cash.State>> {
         val stateAndRefs = mutableListOf<StateAndRef<Cash.State>>()
 
+        // DOCSTART CASHSELECT 1
         for (retryCount in 1..MAX_RETRIES) {
             if (!attemptSpend(services, amount, lockId, notary, onlyFromIssuerParties, withIssuerRefs, stateAndRefs)) {
                 log.warn("Coin selection failed on attempt $retryCount")
@@ -116,6 +120,7 @@ abstract class AbstractCashSelection {
                 break
             }
         }
+        // DOCEND CASHSELECT 1
         return stateAndRefs
     }
 

node-api/src/main/kotlin/net/corda/nodeapi/internal/ClassloaderUtils.kt

@@ -26,6 +26,7 @@ fun scanJarForContracts(cordappJarPath: String): List<ContractClassName> {
     // Only keep instantiable contracts
     val classLoader = URLClassLoader(arrayOf(File(cordappJarPath).toURL()), currentClassLoader)
     val concreteContracts = contracts.map(classLoader::loadClass).filter { !it.isInterface && !Modifier.isAbstract(it.modifiers) }
+    classLoader.close()
     return concreteContracts.map { it.name }
 }
 

node/src/main/kotlin/net/corda/node/internal/ServicesForResolutionImpl.kt

@@ -23,10 +23,10 @@ data class ServicesForResolutionImpl(
 
     @Throws(TransactionResolutionException::class)
     override fun loadStates(stateRefs: Set<StateRef>): Set<StateAndRef<ContractState>> {
-        return stateRefs.groupBy { it.txhash }.map {
+        return stateRefs.groupBy { it.txhash }.flatMap {
             val stx = validatedTransactions.getTransaction(it.key) ?: throw TransactionResolutionException(it.key)
             val baseTx = stx.resolveBaseTransaction(this)
             it.value.map { StateAndRef(baseTx.outputs[it.index], it) }
-        }.flatMap { it }.toSet()
+        }.toSet()
     }
-}
+}

node/src/main/kotlin/net/corda/node/services/transactions/BFTSMaRt.kt

@@ -248,7 +248,7 @@ object BFTSMaRt {
         /** Generates a transaction signature over the specified transaction [txId]. */
         protected fun sign(txId: SecureHash): TransactionSignature {
             val signableData = SignableData(txId, SignatureMetadata(services.myInfo.platformVersion, Crypto.findSignatureScheme(notaryIdentityKey).schemeNumberID))
-            return services.keyManagementService.sign(signableData, notaryIdentityKey)
+            return services.database.transaction { services.keyManagementService.sign(signableData, notaryIdentityKey) }
         }
 
         // TODO:

node/src/test/kotlin/net/corda/node/ArgsParserTest.kt

@@ -5,15 +5,29 @@ import net.corda.core.internal.div
 import net.corda.nodeapi.internal.crypto.X509KeyStore
 import org.assertj.core.api.Assertions.assertThat
 import org.assertj.core.api.Assertions.assertThatExceptionOfType
+import org.junit.BeforeClass
 import org.junit.Test
 import org.slf4j.event.Level
+import java.nio.file.Files
+import java.nio.file.Path
 import java.nio.file.Paths
 import kotlin.test.assertEquals
 import kotlin.test.assertNotNull
 
 class ArgsParserTest {
     private val parser = ArgsParser()
-    private val workingDirectory = Paths.get(".").normalize().toAbsolutePath()
+
+    companion object {
+        private lateinit var workingDirectory: Path
+        private lateinit var buildDirectory: Path
+
+        @BeforeClass
+        @JvmStatic
+        fun initDirectories() {
+            workingDirectory = Paths.get(".").normalize().toAbsolutePath()
+            buildDirectory = workingDirectory.resolve("build")
+        }
+    }
 
     @Test
     fun `no command line arguments`() {
@@ -113,17 +127,21 @@ class ArgsParserTest {
 
     @Test
     fun `initial-registration`() {
-        val truststorePath = workingDirectory / "truststore" / "file.jks"
+        // Create this temporary file in the "build" directory so that "clean" can delete it.
+        val truststorePath = buildDirectory / "truststore" / "file.jks"
         assertThatExceptionOfType(IllegalArgumentException::class.java).isThrownBy {
             parser.parse("--initial-registration", "--network-root-truststore", "$truststorePath", "--network-root-truststore-password", "password-test")
         }.withMessageContaining("Network root trust store path").withMessageContaining("doesn't exist")
 
         X509KeyStore.fromFile(truststorePath, "dummy_password", createNew = true)
-
-        val cmdLineOptions = parser.parse("--initial-registration", "--network-root-truststore", "$truststorePath", "--network-root-truststore-password", "password-test")
-        assertNotNull(cmdLineOptions.nodeRegistrationConfig)
-        assertEquals(truststorePath.toAbsolutePath(), cmdLineOptions.nodeRegistrationConfig?.networkRootTrustStorePath)
-        assertEquals("password-test", cmdLineOptions.nodeRegistrationConfig?.networkRootTrustStorePassword)
+        try {
+            val cmdLineOptions = parser.parse("--initial-registration", "--network-root-truststore", "$truststorePath", "--network-root-truststore-password", "password-test")
+            assertNotNull(cmdLineOptions.nodeRegistrationConfig)
+            assertEquals(truststorePath.toAbsolutePath(), cmdLineOptions.nodeRegistrationConfig?.networkRootTrustStorePath)
+            assertEquals("password-test", cmdLineOptions.nodeRegistrationConfig?.networkRootTrustStorePassword)
+        } finally {
+            Files.delete(truststorePath)
+        }
     }
 
     @Test

samples/irs-demo/src/system-test/kotlin/IRSDemoDockerTest.kt

@@ -72,15 +72,9 @@ class IRSDemoDockerTest {
 
         //Wait for deals to appear in a rows table
         val dealsList = driverWait.until<WebElement>({
-            makeScreenshot(driver, "second")
             it?.findElement(By.cssSelector("table#deal-list tbody tr"))
         })
 
         assertNotNull(dealsList)
     }
-
-    private fun makeScreenshot(driver: PhantomJSDriver, name: String) {
-        val screenshotAs = driver.getScreenshotAs(OutputType.FILE)
-        Files.copy(screenshotAs.toPath(), Paths.get("/Users", "maksymilianpawlak", "phantomjs", name + System.currentTimeMillis() + ".png"), StandardCopyOption.REPLACE_EXISTING)
-    }
 }