CORDA-1510 - Allow Doorman and NetworkMap to be configured independently (#3485)

* CORDA-1510 - Allow Doorman and NetworkMap to be configured independently (#3220)

* CORDA-1510 - Allow Doorman and NetworkMap to be configured independently

Currently only one compatabilityZoneURL can be specified, however the
two services can be run on as separate servers. Allow nodes to be
configured in this manner

* Partial review comments

* Review comments

* review comments

* Test fix

* spell fix

docs/source/changelog.rst

@@ -1,8 +1,14 @@
 Changelog
 =========
 
-Here are brief summaries of what's changed between each snapshot release. This includes guidance on how to upgrade code
+.. _changelog_v3.2:
-from the previous milestone release.
+
+Version 3.2
+-----------
+
+* Doorman and NetworkMap URLs can now be configured individually rather than being assumed to be
+  the same server. Current ``compatibilityZoneURL`` configurations remain valid. See both :doc:`corda-configuration-file`
+  and :doc:`permissioning` for details.
 
 .. _changelog_v3.1:
 

docs/source/corda-configuration-file.rst

@@ -169,7 +169,16 @@ path to the node's base directory.
     interfaces, and then by sending an IP discovery request to the network map service. Set to ``false`` to disable.
 
 :compatibilityZoneURL: The root address of Corda compatibility zone network management services, it is used by the Corda node to register with the network and
-    obtain Corda node certificate, (See :doc:`permissioning` for more information.) and also used by the node to obtain network map information.
+    obtain Corda node certificate, (See :doc:`permissioning` for more information.) and also used by the node to obtain network map information. Cannot be
+    set at the same time as the ``networkServices`` option.
+
+:networkServices: If the Corda compatibility zone services, both network map and registration (doorman), are not running on the same endpoint
+    and thus have different URLs then this option should be used in place of the ``compatibilityZoneURL`` setting.
+
+    :doormanURL: Root address of the network registration service.
+    :networkMapURL: Root address of the network map service.
+
+.. note:: Only one of ``compatibilityZoneURL`` or ``networkServices`` should be used.
 
 :jvmArgs: An optional list of JVM args, as strings, which replace those inherited from the command line when launching via ``corda.jar``
     only. e.g. ``jvmArgs = [ "-Xmx220m", "-Xms220m", "-XX:+UseG1GC" ]``
@@ -195,4 +204,4 @@ path to the node's base directory.
             Otherwise defaults to 10MB
 
 :attachmentCacheBound: Optionally specify how many attachments should be cached locally. Note that this includes only the key and
-            metadata, the content is cached separately and can be loaded lazily. Defaults to 1024.
+            metadata, the content is cached separately and can be loaded lazily. Defaults to 1024.

docs/source/example-code/src/main/resources/example-node-with-networkservices.conf

@@ -0,0 +1,25 @@
+myLegalName : "O=Bank A,L=London,C=GB"
+keyStorePassword : "cordacadevpass"
+trustStorePassword : "trustpass"
+crlCheckSoftFail: true
+dataSourceProperties : {
+    dataSourceClassName : org.h2.jdbcx.JdbcDataSource
+    dataSource.url : "jdbc:h2:file:"${baseDirectory}"/persistence"
+    dataSource.user : sa
+    dataSource.password : ""
+}
+p2pAddress : "my-corda-node:10002"
+rpcSettings = {
+    useSsl = false
+    standAloneBroker = false
+    address : "my-corda-node:10003"
+    adminAddress : "my-corda-node:10004"
+}
+rpcUsers : [
+    { username=user1, password=letmein, permissions=[ StartFlow.net.corda.protocols.CashProtocol ] }
+]
+devMode : false
+networkServices : {
+    doormanURL = "https://registration.corda.net"
+    networkMapURL = "https://cz.corda.net"
+}

docs/source/permissioning.rst

@@ -192,21 +192,25 @@ This can be overridden with the additional ``--network-root-truststore`` flag.
 The certificate signing request will be created based on node information obtained from the node configuration.
 The following information from the node configuration file is needed to generate the request.
 
-:myLegalName: Your company's legal name as an X.500 string. X.500 allows differentiation between entities with the same
+* **myLegalName** Your company's legal name as an X.500 string. X.500 allows differentiation between entities with the same
-          name as the legal name needs to be unique on the network. If another node has already been permissioned with this
+  name as the legal name needs to be unique on the network. If another node has already been permissioned with this
-          name then the permissioning server will automatically reject the request. The request will also be rejected if it
+  name then the permissioning server will automatically reject the request. The request will also be rejected if it
-          violates legal name rules, see :ref:`node_naming` for more information.
+  violates legal name rules, see :ref:`node_naming` for more information.
 
-:emailAddress: e.g. "admin@company.com"
+* **emailAddress** e.g. "admin@company.com"
 
-:devMode: must be set to false
+* **devMode** must be set to false
 
-:compatibilityZoneURL: Corda compatibility zone network management service root URL.
+* **networkServices or compatibilityZoneURL** The Corda compatibility zone services must be configured. This must be either:
 
-      A new pair of private and public keys generated by the Corda node will be used to create the request.
+  * **compatibilityZoneURL** The Corda compatibility zone network management service root URL.
+  * **networkServices** Replaces the ``compatibilityZoneURL`` when the Doorman and Network Map services
+    are configured to operate on different URL endpoints. The ``doorman`` entry is used for registration.
 
-      The utility will submit the request to the doorman server and poll for a result periodically to retrieve the certificates.
+A new pair of private and public keys generated by the Corda node will be used to create the request.
-      Once the request has been approved and the certificates downloaded from the server, the node will create the keystore and trust store using the certificates and the generated private key.
+
+The utility will submit the request to the doorman server and poll for a result periodically to retrieve the certificates.
+Once the request has been approved and the certificates downloaded from the server, the node will create the keystore and trust store using the certificates and the generated private key.
 
 .. note:: You can exit the utility at any time if the approval process is taking longer than expected. The request process will resume on restart.
 

node/src/integration-test/kotlin/net/corda/node/services/network/NetworkMapTest.kt

@@ -13,26 +13,25 @@ import net.corda.core.utilities.seconds
 import net.corda.nodeapi.internal.network.NETWORK_PARAMS_FILE_NAME
 import net.corda.nodeapi.internal.network.SignedNetworkParameters
 import net.corda.testing.common.internal.testNetworkParameters
-import net.corda.testing.core.ALICE_NAME
+import net.corda.testing.core.*
-import net.corda.testing.core.BOB_NAME
-import net.corda.testing.core.SerializationEnvironmentRule
 import net.corda.testing.driver.NodeHandle
-import net.corda.testing.driver.PortAllocation
 import net.corda.testing.driver.internal.RandomFree
-import net.corda.testing.node.internal.CompatibilityZoneParams
+import net.corda.testing.node.internal.*
-import net.corda.testing.node.internal.internalDriver
 import net.corda.testing.node.internal.network.NetworkMapServer
 import org.assertj.core.api.Assertions.assertThat
 import org.junit.After
 import org.junit.Before
 import org.junit.Rule
 import org.junit.Test
+import org.junit.runner.RunWith
+import org.junit.runners.Parameterized
 import java.net.URL
 import java.time.Instant
 import kotlin.streams.toList
 import kotlin.test.assertEquals
 
-class NetworkMapTest {
+@RunWith(Parameterized::class)
+class NetworkMapTest(var initFunc: (URL, NetworkMapServer) -> CompatibilityZoneParams) {
     @Rule
     @JvmField
     val testSerialization = SerializationEnvironmentRule(true)
@@ -43,13 +42,37 @@ class NetworkMapTest {
     private lateinit var networkMapServer: NetworkMapServer
     private lateinit var compatibilityZone: CompatibilityZoneParams
 
+    companion object {
+        @JvmStatic
+        @Parameterized.Parameters(name = "{0}")
+        fun runParams() = listOf(
+                { addr: URL, nms: NetworkMapServer ->
+                    SharedCompatibilityZoneParams(
+                            addr,
+                            publishNotaries = {
+                                nms.networkParameters = testNetworkParameters(it, modifiedTime = Instant.ofEpochMilli(random63BitValue()), epoch = 2)
+                            }
+                    )
+                },
+                { addr: URL, nms: NetworkMapServer ->
+                    SplitCompatibilityZoneParams(
+                            doormanURL = URL("http://I/Don't/Exist"),
+                            networkMapURL = addr,
+                            publishNotaries = {
+                                nms.networkParameters = testNetworkParameters(it, modifiedTime = Instant.ofEpochMilli(random63BitValue()), epoch = 2)
+                            }
+                    )
+                }
+
+        )
+    }
+
+
     @Before
     fun start() {
         networkMapServer = NetworkMapServer(cacheTimeout, portAllocation.nextHostAndPort())
         val address = networkMapServer.start()
-        compatibilityZone = CompatibilityZoneParams(URL("http://$address"), publishNotaries = {
+        compatibilityZone = initFunc(URL("http://$address"), networkMapServer)
-            networkMapServer.networkParameters = testNetworkParameters(it, modifiedTime = Instant.ofEpochMilli(random63BitValue()), epoch = 2)
-        })
     }
 
     @After

node/src/integration-test/kotlin/net/corda/node/utilities/registration/NodeRegistrationTest.kt

@@ -22,6 +22,7 @@ import net.corda.testing.driver.PortAllocation
 import net.corda.testing.internal.DEV_ROOT_CA
 import net.corda.testing.node.NotarySpec
 import net.corda.testing.node.internal.CompatibilityZoneParams
+import net.corda.testing.node.internal.SharedCompatibilityZoneParams
 import net.corda.testing.node.internal.internalDriver
 import net.corda.testing.node.internal.network.NetworkMapServer
 import org.assertj.core.api.Assertions.assertThat
@@ -77,7 +78,7 @@ class NodeRegistrationTest {
 
     @Test
     fun `node registration correct root cert`() {
-        val compatibilityZone = CompatibilityZoneParams(
+        val compatibilityZone = SharedCompatibilityZoneParams(
                 URL("http://$serverHostAndPort"),
                 publishNotaries = { server.networkParameters = testNetworkParameters(it) },
                 rootCert = DEV_ROOT_CA.certificate)

node/src/main/kotlin/net/corda/node/ArgsParser.kt

@@ -91,11 +91,14 @@ data class CmdLineOptions(val baseDirectory: Path,
                           val noLocalShell: Boolean,
                           val sshdServer: Boolean,
                           val justGenerateNodeInfo: Boolean,
-                          val bootstrapRaftCluster: Boolean) {
+                          val bootstrapRaftCluster: Boolean
+) {
     fun loadConfig(): NodeConfiguration {
         val config = ConfigHelper.loadConfig(baseDirectory, configFile).parseAsNodeConfiguration()
         if (isRegistration) {
-            requireNotNull(config.compatibilityZoneURL) { "Compatibility Zone Url must be provided in registration mode." }
+            require(config.compatibilityZoneURL != null || config.networkServices != null) {
+                "compatibilityZoneURL or networkServices must be present in the node configuration file in registration mode."
+            }
             requireNotNull(networkRootTruststorePath) { "Network root trust store path must be provided in registration mode." }
             requireNotNull(networkRootTruststorePassword) { "Network root trust store password must be provided in registration mode." }
         }

node/src/main/kotlin/net/corda/node/internal/AbstractNode.kt

@@ -197,7 +197,7 @@ abstract class AbstractNode(val configuration: NodeConfiguration,
         val (identity, identityKeyPair) = obtainIdentity(notaryConfig = null)
         val identityService = makeIdentityService(identity.certificate)
 
-        networkMapClient = configuration.compatibilityZoneURL?.let { NetworkMapClient(it, identityService.trustRoot) }
+        networkMapClient = configuration.networkServices?.let { NetworkMapClient(it.networkMapURL, identityService.trustRoot) }
 
         val networkParameters = NetworkParametersReader(identityService.trustRoot, networkMapClient, configuration.baseDirectory).networkParameters
         check(networkParameters.minimumPlatformVersion <= versionInfo.platformVersion) {

node/src/main/kotlin/net/corda/node/internal/NodeStartup.kt

@@ -188,8 +188,10 @@ open class NodeStartup(val args: Array<String>) {
         return !(!cmdlineOptions.isRegistration || compatibilityZoneURL == null)
     }
 
-    open protected fun registerWithNetwork(conf: NodeConfiguration, networkRootTruststorePath: Path, networkRootTruststorePassword: String) {
+    protected open fun registerWithNetwork(conf: NodeConfiguration, networkRootTruststorePath: Path, networkRootTruststorePassword: String) {
-        val compatibilityZoneURL = conf.compatibilityZoneURL!!
+        val compatibilityZoneURL = conf.networkServices?.doormanURL ?: throw RuntimeException(
+                "compatibilityZoneURL or networkServices must be configured!")
+
         println()
         println("******************************************************************")
         println("*                                                                *")

node/src/main/kotlin/net/corda/node/services/config/NodeConfiguration.kt

@@ -32,6 +32,7 @@ interface NodeConfiguration : NodeSSLConfiguration {
     val devMode: Boolean
     val devModeOptions: DevModeOptions?
     val compatibilityZoneURL: URL?
+    val networkServices: NetworkServicesConfig?
     val certificateChainCheckPolicies: List<CertChainPolicyConfig>
     val verifierType: VerifierType
     val messageRedeliveryDelaySeconds: Int
@@ -107,6 +108,25 @@ data class BridgeConfiguration(val retryIntervalMs: Long,
 
 data class ActiveMqServerConfiguration(val bridge: BridgeConfiguration)
 
+/**
+ * Used as an alternative to the older compatibilityZoneURL to allow the doorman and network map
+ * services for a node to be configured as different URLs. Cannot be set at the same time as the
+ * compatibilityZoneURL, and will be defaulted (if not set) to both point at the configured
+ * compatibilityZoneURL.
+ *
+ * @property doormanURL The URL of the tls certificate signing service.
+ * @property networkMapURL The URL of the Network Map service.
+ * @property inferred Non user setting that indicates weather the Network Services configuration was
+ * set explicitly ([inferred] == false) or weather they have been inferred via the compatibilityZoneURL parameter
+ * ([inferred] == true) where both the network map and doorman are running on the same endpoint. Only one,
+ * compatibilityZoneURL or networkServices, can be set at any one time.
+ */
+data class NetworkServicesConfig(
+        val doormanURL: URL,
+        val networkMapURL: URL,
+        val inferred : Boolean = false
+)
+
 fun Config.parseAsNodeConfiguration(): NodeConfiguration = parseAs<NodeConfigurationImpl>()
 
 data class NodeConfigurationImpl(
@@ -118,6 +138,7 @@ data class NodeConfigurationImpl(
         override val trustStorePassword: String,
         override val dataSourceProperties: Properties,
         override val compatibilityZoneURL: URL? = null,
+        override var networkServices: NetworkServicesConfig? = null,
         override val rpcUsers: List<User>,
         override val security : SecurityConfiguration? = null,
         override val verifierType: VerifierType,
@@ -156,6 +177,7 @@ data class NodeConfigurationImpl(
             explicitAddress != null -> {
                 require(settings.address == null) { "Can't provide top-level rpcAddress and rpcSettings.address (they control the same property)." }
                 logger.warn("Top-level declaration of property 'rpcAddress' is deprecated. Please use 'rpcSettings.address' instead.")
+
                 settings.copy(address = explicitAddress)
             }
             else -> settings
@@ -165,6 +187,7 @@ data class NodeConfigurationImpl(
     override fun validate(): List<String> {
         val errors = mutableListOf<String>()
         errors += validateRpcOptions(rpcOptions)
+        errors += validateNetworkServices()
         return errors
     }
 
@@ -179,6 +202,17 @@ data class NodeConfigurationImpl(
     }
 
     override val exportJMXto: String get() = "http"
+
+    private fun validateNetworkServices(): List<String> {
+        val errors = mutableListOf<String>()
+
+        if (compatibilityZoneURL != null && networkServices != null && !(networkServices!!.inferred)) {
+            errors += "Cannot configure both compatibilityZoneUrl and networkServices simultaneously"
+        }
+
+        return errors
+    }
+
     override val transactionCacheSizeBytes: Long
         get() = transactionCacheSizeMegaBytes?.MB ?: super.transactionCacheSizeBytes
     override val attachmentContentCacheSizeBytes: Long
@@ -192,6 +226,10 @@ data class NodeConfigurationImpl(
         require(security == null || rpcUsers.isEmpty()) {
             "Cannot specify both 'rpcUsers' and 'security' in configuration"
         }
+
+        if (compatibilityZoneURL != null && networkServices == null) {
+            networkServices = NetworkServicesConfig(compatibilityZoneURL, compatibilityZoneURL, true)
+        }
     }
 }
 

node/src/test/kotlin/net/corda/node/services/config/NodeConfigurationImplTest.kt

@@ -4,8 +4,12 @@ import net.corda.core.internal.div
 import net.corda.core.utilities.NetworkHostAndPort
 import net.corda.testing.core.ALICE_NAME
 import net.corda.testing.node.MockServices.Companion.makeTestDataSourceProperties
-import org.assertj.core.api.Assertions.assertThatThrownBy
+import org.assertj.core.api.Assertions.*
+import org.junit.Assert.assertEquals
+import org.junit.Assert.assertNotNull
 import org.junit.Test
+import java.net.URI
+import java.net.URL
 import java.nio.file.Paths
 import java.util.*
 import kotlin.test.assertFalse
@@ -29,6 +33,34 @@ class NodeConfigurationImplTest {
         assertFalse { configDebugOptions(true, DevModeOptions(true)).shouldCheckCheckpoints() }
     }
 
+    @Test
+    fun `validation has error when both compatibilityZoneURL and networkServices are configured`() {
+        val configuration = testConfiguration.copy(
+                devMode = false,
+                compatibilityZoneURL = URL("https://r3.com"),
+                networkServices = NetworkServicesConfig(
+                        URL("https://r3.com.doorman"),
+                        URL("https://r3.com/nm")))
+
+        val errors = configuration.validate()
+
+        assertThat(errors).hasOnlyOneElementSatisfying {
+            error -> error.contains("Cannot configure both compatibilityZoneUrl and networkServices simultaneously")
+        }
+    }
+
+    @Test
+    fun `compatiilityZoneURL populates NetworkServices`() {
+        val compatibilityZoneURL = URI.create("https://r3.com").toURL()
+        val configuration = testConfiguration.copy(
+                devMode = false,
+                compatibilityZoneURL = compatibilityZoneURL)
+
+        assertNotNull(configuration.networkServices)
+        assertEquals(compatibilityZoneURL, configuration.networkServices!!.doormanURL)
+        assertEquals(compatibilityZoneURL, configuration.networkServices!!.networkMapURL)
+    }
+
     private fun configDebugOptions(devMode: Boolean, devModeOptions: DevModeOptions?): NodeConfiguration {
         return testConfiguration.copy(devMode = devMode, devModeOptions = devModeOptions)
     }

testing/node-driver/src/main/kotlin/net/corda/testing/node/internal/DriverDSLImpl.kt

@@ -200,7 +200,7 @@ class DriverDSLImpl(
         }
         val registrationFuture = if (compatibilityZone?.rootCert != null) {
             // We don't need the network map to be available to be able to register the node
-            startNodeRegistration(name, compatibilityZone.rootCert, compatibilityZone.url)
+            startNodeRegistration(name, compatibilityZone.rootCert, compatibilityZone.doormanURL())
         } else {
             doneFuture(Unit)
         }
@@ -225,7 +225,15 @@ class DriverDSLImpl(
         val rpcAdminAddress = portAllocation.nextHostAndPort()
         val webAddress = portAllocation.nextHostAndPort()
         val users = rpcUsers.map { it.copy(permissions = it.permissions + DRIVER_REQUIRED_PERMISSIONS) }
-        val czUrlConfig = if (compatibilityZone != null) mapOf("compatibilityZoneURL" to compatibilityZone.url.toString()) else emptyMap()
+        val czUrlConfig = when (compatibilityZone) {
+            null -> emptyMap()
+            is SharedCompatibilityZoneParams ->
+                mapOf("compatibilityZoneURL" to compatibilityZone.doormanURL().toString())
+            is SplitCompatibilityZoneParams ->
+                mapOf("networkServices.doormanURL" to compatibilityZone.doormanURL().toString(),
+                        "networkServices.networkMapURL" to compatibilityZone.networkMapURL().toString())
+        }
+
         val config = NodeConfig(ConfigHelper.loadConfig(
                 baseDirectory = baseDirectory(name),
                 allowMissingConfig = true,
@@ -413,7 +421,7 @@ class DriverDSLImpl(
                 startNotaryIdentityGeneration()
             } else {
                 // With a root cert specified we delegate generation of the notary identities to the CZ.
-                startAllNotaryRegistrations(compatibilityZone.rootCert, compatibilityZone.url)
+                startAllNotaryRegistrations(compatibilityZone.rootCert, compatibilityZone.doormanURL())
             }
             notaryInfosFuture.map { notaryInfos ->
                 compatibilityZone.publishNotaries(notaryInfos)
@@ -1013,15 +1021,49 @@ fun <DI : DriverDSL, D : InternalDriverDSL, A> genericDriver(
 
 /**
  * Internal API to enable testing of the network map service and node registration process using the internal driver.
- * @property url The base CZ URL for registration and network map updates
+ *
  * @property publishNotaries Hook for a network map server to capture the generated [NotaryInfo] objects needed for
  * creating the network parameters. This is needed as the network map server is expected to distribute it. The callback
  * will occur on a different thread to the driver-calling thread.
  * @property rootCert If specified then the nodes will register themselves with the doorman service using [url] and expect
  * the registration response to be rooted at this cert. If not specified then no registration is performed and the dev
  * root cert is used as normal.
+ *
+ * @see SharedCompatibilityZoneParams
+ * @see SplitCompatibilityZoneParams
  */
-data class CompatibilityZoneParams(val url: URL, val publishNotaries: (List<NotaryInfo>) -> Unit, val rootCert: X509Certificate? = null)
+sealed class CompatibilityZoneParams(
+        val publishNotaries: (List<NotaryInfo>) -> Unit,
+        val rootCert: X509Certificate? = null
+) {
+    abstract fun networkMapURL(): URL
+    abstract fun doormanURL(): URL
+}
+
+/**
+ * Represent network management services, network map and doorman, running on the same URL
+ */
+class SharedCompatibilityZoneParams(
+        private val url: URL,
+        publishNotaries: (List<NotaryInfo>) -> Unit,
+        rootCert: X509Certificate? = null
+) : CompatibilityZoneParams(publishNotaries, rootCert) {
+    override fun doormanURL() = url
+    override fun networkMapURL() = url
+}
+
+/**
+ * Represent network management services, network map and doorman, running on different URLs
+ */
+class SplitCompatibilityZoneParams(
+        private val doormanURL: URL,
+        private val networkMapURL: URL,
+        publishNotaries: (List<NotaryInfo>) -> Unit,
+        rootCert: X509Certificate? = null
+) : CompatibilityZoneParams(publishNotaries, rootCert) {
+    override fun doormanURL() = doormanURL
+    override fun networkMapURL() = networkMapURL
+}
 
 fun <A> internalDriver(
         isDebug: Boolean = DriverParameters().isDebug,

testing/node-driver/src/main/kotlin/net/corda/testing/node/internal/InternalMockNetwork.kt

@@ -464,6 +464,7 @@ private fun mockNodeConfiguration(): NodeConfiguration {
         doReturn(true).whenever(it).devMode
         doReturn(null).whenever(it).compatibilityZoneURL
         doReturn(emptyList<CertChainPolicyConfig>()).whenever(it).certificateChainCheckPolicies
+        doReturn(null).whenever(it).networkServices
         doReturn(VerifierType.InMemory).whenever(it).verifierType
         doReturn(5).whenever(it).messageRedeliveryDelaySeconds
         doReturn(5.seconds.toMillis()).whenever(it).additionalNodeInfoPollingFrequencyMsec