Merge pull request #820 from corda/mnesbit-bridge-docs

Initial very basic docs for the bridge

docs/source/bridge-configuration-file.rst

@@ -0,0 +1,453 @@
+Bridge configuration
+====================
+
+.. contents::
+
+File location
+-------------
+When starting a standalone bridge, or float, the ``corda-bridgeserver.jar`` file defaults to reading the bridge's configuration from a ``bridge.conf`` file in
+the directory from which the command to launch the process is executed. There are two command-line options to override this
+behaviour:
+
+* The ``--config-file`` command line option allows you to specify a configuration file with a different name, or at
+  different file location. Paths are relative to the current working directory
+
+* The ``--base-directory`` command line option allows you to specify the bridge's workspace location. A ``bridge.conf``
+  configuration file is then expected in the root of this workspace
+
+If you specify both command line arguments at the same time, the bridge will fail to start.
+
+Format
+------
+The Bridge configuration file uses the HOCON format which is superset of JSON. Please visit
+`<https://github.com/typesafehub/config/blob/master/HOCON.md>`_ for further details.
+
+.. warning::  Please do NOT use double quotes (``"``) in configuration keys.
+
+              Bridge setup will log `Config files should not contain \" in property names. Please fix: [key]` as error
+              when it founds double quotes around keys.
+              This prevents configuration errors when mixing keys containing ``.`` wrapped with double quotes and without them
+
+Defaults
+--------
+A set of default configuration options are loaded from the built-in resource file ``/bridge/src/main/resources/bridgedefault.conf``.
+This file can be found in the ``:bridge`` gradle module of the `Enterprise Corda repository <https://github.com/corda/enterprise>`_. Any
+options you do not specify in your own ``bridge.conf`` file will use these defaults.
+
+Here are the contents of the ``bridgedefault.conf`` file:
+
+.. literalinclude:: ../../bridge/src/main/resources/bridgedefault.conf
+    :language: javascript
+   
+Bridge Operating Modes
+----------------------
+.. note:: By default, the Corda node assumes that it will carry out the peer-to-peer functions of the bridge internally!
+          Before running a dedicated bridge process, it is essential to turn off the dev mode component by setting the
+          ``enterpriseConfiguration.externalBridge`` property of the ``node.conf`` file to ``true``.
+          If the ``externalBridge`` flag is not ``true``, there will be unexpected behaviour as the node will try to send peer-to-peer messages directly!
+
+Assuming that an external bridge is to be used, the ``corda-bridgeserver.jar`` operates in one of three basic operating modes.
+The particular mode is selected via the required ``bridgeMode`` configuration property inside ``bridge.conf``:
+
+:SenderReceiver: selects a single process bridge solution to isolate the node and Artemis broker from direct Internet contact.
+  It is still assumed that the bridge process is behind a firewall, but both the message sending and receiving paths will pass via the ``bridge``.
+  In this mode the ``outboundConfig`` and ``inboundConfig`` configuration sections of ``bridge.conf`` must be provided,
+  the ``bridgeInnerConfig`` and ``floatOuterConfig`` sections should not be present.
+
+:BridgeInner: mode runs this instance of the ``corda-bridgeserver.jar`` as the trusted portion of the peer-to-peer firewall float.
+  Specifically, this process runs the complete outbound message processing. For the inbound path it operates only the filtering and durable storing portions of the message processing.
+  The process expects to connect through a firewall to a matched ``FloatOuter`` instance running in the DMZ as the actual ``TLS/AMQP 1.0`` termination point.
+
+:FloatOuter: causes this instance of the ``corda-bridgeserver.jar`` to run as a protocol break proxy for inbound message path. The process
+  will initialise a ``TLS`` control port and await connection from the ``BridgeInner``. Once the control connection is successful the ``BridgeInner`` will securely provision
+  the ``TLS`` socket server key and certificates into the ``FloatOuter``. The process will then start listening for inbound connection from peer nodes.
+
+Fields
+------
+The available config fields are listed below. ``baseDirectory`` is available as a substitution value and contains the
+absolute path to the bridge's base directory.
+
+:bridgeMode: Determines operating mode of the bridge. See above.
+
+:keyStorePassword: The password to unlock the TLS KeyStore file (``<workspace>/certificates/sslkeystore.jks``) containing the
+    node certificate and private key. The private key password must be the same due to limitations in the Artemis libraries.
+
+    .. note:: This is the non-secret value for the development certificates automatically generated during the first node run.
+        Longer term these keys will be managed in secure hardware devices.
+
+:trustStorePassword: The password to unlock the Trust store file (``<workspace>/certificates/truststore.jks``) containing
+    the Corda network root certificate. This is the non-secret value for the development certificates automatically
+    generated during the first node run.
+
+    .. note:: Longer term these keys will be managed in secure hardware devices.
+
+
+:networkParametersPath: This is the file path to a copy of the ``network-parameters`` as copied from a node after it has fetched the latest version from the network-map via http.
+   It is used to correctly configure the maximum allowed message size. The maximum message size limit is already enforced by the P2P Artemis inside the ``node``,
+   but the ``bridge`` also enforces this before forwarding messages to remote peers and also the ``float`` enforces this on received packets.
+   If the size limit is breached these messages will be consumed and discarded, so that they are not replayed forever.
+
+:outboundConfig:  This section is used to configure the processing of outbound messages. It is required for ``SenderReceiver`` and ``BridgeInner`` modes and must be absent for ``FloatOuter`` mode:
+
+   :artemisBrokerAddress: The primary host and port for peer-to-peer Artemis broker. This may be running inside to the node, in which case it will hosted on the port of the ``p2pAddress``,
+        or the ``messagingServerAddress`` if that is defined and ``messagingServerExternal`` is ``false``. Otherwise, it could be an independently run Artemis broker.
+
+   :alternateArtemisBrokerAddresses: Optionally if there are multiple Artemis broker address e.g. for hot-cold node deployment, then additional hosts and ports may be included in a list.
+
+   :customSSLConfiguration:  The default behaviour is that the outgoing ``TLS/AMQP 1.0`` connections present certificate details from (``<workspace>/certificates/sslkeystore.jks``)
+        and validate against (``<workspace>/certificates/truststore.jks``), using the passwords defined in the root config. However, distinct KeyStores may be configured in this section:
+
+        :keyStorePassword: The password for the TLS KeyStore and private keys within the KeyStore.
+
+        :trustStorePassword: The password for TLS TrustStore.
+
+        :sslKeystore: The path to the KeyStore file to use in outgoing ``TLS/AMQP 1.0`` connections.
+
+        :trustStoreFile: The path to the TrustStore file to use in outgoing ``TLS/AMQP 1.0`` connections.
+
+   :socksProxyConfig:  This section is optionally present if outgoing peer connections should go via a SOCKS4, or SOCKS5 proxy:
+
+        :version: Either SOCKS4, or SOCKS5 to define the protocol version used in connecting to the SOCKS proxy.
+
+        :proxyAddress: Host and port of the SOCKS proxy.
+
+        :userName: Optionally a user name that will be presented to the SOCKS proxy after connect.
+
+        :password: Optionally, a password to present to the SOCKS5 Proxy. It is not valid for SOCKS4 proxies and it should always be combined with [userName].
+
+:inboundConfig:  This section is used to configure the properties of the listening port. It is required for ``SenderReceiver`` and ``FloatOuter`` modes and must be absent for ``BridgeInner`` mode:
+
+        :listeningAddress: The host and port to bind to as ``TLS/AMQP 1.0`` listener. This may be a specific network interface on multi-homed machines.
+            It may also differ from the externally exposed public ``p2pAddress`` of the port if the firewalls, or load balancers transparently reroute the traffic.
+
+   :customSSLConfiguration:  The default behaviour is that the inbound ``TLS/AMQP 1.0`` connections present certificate details from (``<workspace>/certificates/sslkeystore.jks``)
+        and validate against (``<workspace>/certificates/truststore.jks``), using the passwords defined in the root config. However, distinct KeyStores may be configured in this section:
+
+        :keyStorePassword: The password for the TLS KeyStore and private keys within the KeyStore.
+
+        :trustStorePassword: The password for TLS TrustStore.
+
+        :sslKeystore: The path to the KeyStore file to use in inbound ``TLS/AMQP 1.0`` connections.
+
+        :trustStoreFile: The path to the TrustStore file to use in inbound ``TLS/AMQP 1.0`` connections.
+
+:bridgeInnerConfig:  This section is required for ``BridgeInner`` mode and configures the tunnel connection to the ``FloatOuter`` (s) in the DMZ. The section should be absent in ``SenderReceiver`` and ``FloatOuter`` modes:
+
+        :floatAddresses: The list of host and ports to connect the available ``FloatOuter`` instances. At least one must be present.
+            The active ``BridgeInner`` will round-robin over available ``FloatOuter`` addresses until it can connect and activate one.
+
+        :expectedCertificateSubject: The X500 Subject name that will be presented in client certificates from the remote ``FloatOuter`` instances.
+        
+        :customSSLConfiguration:   .. note:: For ease of use the TLS default control tunnel connections present certificate details from (``<workspace>/certificates/sslkeystore.jks``)
+                                             and validate against (``<workspace>/certificates/truststore.jks``), using the passwords defined in the root config.
+                                             However, it is strongly recommended that distinct KeyStores should be configured in this section to use locally valid certificates only, so that compromise of the DMZ machines does not give access to the node's primary TLS keys.
+
+            :keyStorePassword: The password for the TLS KeyStore and private keys within the KeyStore.
+
+            :trustStorePassword: The password for TLS TrustStore.
+
+            :sslKeystore: The path to the KeyStore file to use in control tunnel connections.
+
+            :trustStoreFile: The path to the TrustStore file to use in control tunnel connections.
+            
+        :customFloatOuterSSLConfiguration: The keys and certificates for the ``FloatOuter`` are provisioned dynamically from the ``BridgeInner`` over the control tunnel and are not loaded from disk in the DMZ.
+            By default, they are taken from (``<workspace>/certificates/sslkeystore.jks``)
+            and validate against (``<workspace>/certificates/truststore.jks``), using the passwords defined in the root config. However, alternate sources may be defined in this section.
+
+            :keyStorePassword: The password for the TLS KeyStore and private keys within the KeyStore.
+
+            :trustStorePassword: The password for TLS TrustStore.
+
+            :sslKeystore: The path to the KeyStore file to use in the ``FloatOuter`` when it activates the peer listening socket.
+
+            :trustStoreFile: The path to the TrustStore file to use in the ``FloatOuter`` when it activates the peer listening socket.
+
+:floatOuterConfig:   This section is required for ``FloatOuter`` mode and configures the control tunnel listening socket. It should be absent for ``SenderReceiver`` and ``BridgeInner`` modes:
+
+        :floatAddress: The host and port to bind the control tunnel listener socket to. This can be for a specific interface if used on a multi-homed machine.
+
+        :expectedCertificateSubject: The X500 Subject name that will be presented in client certificates from the ``BridgeInner`` when it connects to this ``FloatOuter`` instance.
+
+        :customSSLConfiguration:   .. note:: For ease of use the TLS default control tunnel connection presents certificate details from (``<workspace>/certificates/sslkeystore.jks``)
+                                             and validate against (``<workspace>/certificates/truststore.jks``), using the passwords defined in the root config.
+                                             However, it is strongly recommended that distinct KeyStores should be configured in this section to use locally valid certificates only, so that compromise of the DMZ machines does not give access to the node's primary TLS keys.
+
+            :keyStorePassword: The password for the TLS KeyStore and private keys within the KeyStore.
+
+            :trustStorePassword: The password for TLS TrustStore.
+
+            :sslKeystore: The path to the KeyStore file to use in control tunnel connections.
+
+            :trustStoreFile: The path to the TrustStore file to use in control tunnel connections.
+            
+:haConfig: Optionally the ``SenderReceiver`` and ``BridgeInner`` modes can be run in a hot-warm configuration, which determines the active instance using an external master election service.
+    Currently, only Zookeeper can be used as master elector. Eventually other electors may be supported e.g. ``etcd``. This configuration section controls these options:
+
+    :haConnectionString:  A string containing the connection details of the master electors as a comma delimited list of connection string in the format ``zk://<host>:<port>``.
+        In future it intended that other schemes such as ``etcd`` are supported.
+
+    :haPriority: The implementation uses a prioritise leader election algorithm, so that a preferred master instance can be set. The highest priority is 0 and larger integers have lower priority.
+        At the same level of priority, it is random which instance wins the leader election. If a bridge instance dies another will have the opportunity to become master in instead.
+
+    :haTopic: Sets the zookeeper topic prefix that the nodes used in resolving the election and must be the same for all ``bridge``
+        instances competing for master status. This is available to allow a single zookeeper cluster to be reused with multiple
+        sets of ``bridges`` (e.g. in test environments).
+        The default value is ``bridge/ha`` and would not normally need to be changed if the cluster is not shared.
+
+:artemisReconnectionIntervalMin: If connection to the local Artemis server fails the initial reconnection attempt will be
+    after [artemisReconnectionIntervalMin] ms. The default interval is 5000 ms.
+    Subsequent retries will take be exponentially backed off until they reach [artemisReconnectionIntervalMax] ms.
+
+:artemisReconnectionIntervalMax: The worst case Artemis retry period after repeated failure to connect is [artemisReconnectionIntervalMax] ms. The default interval is 60000 ms.
+
+:enableAMQPPacketTrace: Set this developer flag to true if very detailed logs are required for connectivity debugging. Note that the logging volume is substantial, so do not enable in production systems.
+
+Fully Worked Example
+-------
+As an example to show all features the following is a walk through of the configuration for a pair of HA hot-cold Nodes,
+connected to by a HA hot-warm set of ``BridgeInner`` and ``FloatOuter`` that use some simple certificates to secure the
+control tunnel and a SOCKS5 proxy for outgoing connectivity (see diagram).
+This is also the recommended full enterprise deployment pattern, although there are plenty of alternative deployment options.
+
+.. image:: resources/bridge/ha_bridge_float_socks.png
+     :scale: 100%
+     :align: center
+
+In this example it is assumed that the corda nodes are deployed on ``nodeserver1`` and ``nodeserver2`` using Azure SQL Server ``nodeda.database.windows.net/corda`` as clustered storage. The ``BridgeInner`` instances run on ``bridgeserver1`` and ``bridgeserver2``.
+The SOCKS5 proxy is at ``socksproxy`` port 1234. The ``FloatOuter`` instances run in the DMZ with dual homed machines with addresses ``dmzinternal1`` and ``dmzinternal2`` as the side exposed to the
+internal trusted zone. The externally accessible addresses of the DMZ servers are ``dmzexternal1`` and ``dmzexternal2``, which the Internet facing firewall/load balancer maps to ``banka.com``.
+There is also a zookeeper (must be version ``3.5.3-beta``) cluster on ``zookeep1``, ``zookeep2`` and ``zookeep3``.
+
+First, the nodes need to be configured in hot-cold mode, with external bridge mode enabled. The nodes will host the P2P Artemis internally, with the journal folder replicated between hot and cold nodes.
+It is essential that the node registration process be followed on one node and the resulting certificates, keys and network-parameters nodeInfo files are synchronised across the setup. In particular the 
+``BridgeInner`` setup needs a certificates folder containing the ``sslkeystore.jks`` and ``truststore.jks``. copied from the node and a copied ``network-parameters`` file in the workspace folder.
+The ``FloatOuter`` instances needs a copied ``network-parameters`` file only as the public facing SSL is provisioned from the ``BridgeInner``.
+
+In this example the tunnel connection uses local certs which can be generated with Java keytool from the SDK. An example script would be:
+
+..  code-block:: bash
+
+    keytool.exe -genkeypair -keyalg EC -keysize 256 -alias floatroot -validity 1000 -dname "CN=Float Root,O=Local Only,L=London,C=GB" -ext bc:ca:true,pathlen:1 -keystore floatca.jks -storepass capass -keypass cakeypass
+    keytool.exe -genkeypair -keyalg EC -keysize 256 -alias bridgecert -validity 1000 -dname "CN=Bridge Local,O=Local Only,L=London,C=GB" -ext bc:ca:false -keystore bridge.jks -storepass bridgepass -keypass bridgepass
+    keytool.exe -genkeypair -keyalg EC -keysize 256 -alias floatcert -validity 1000 -dname "CN=Float Local,O=Local Only,L=London,C=GB" -ext bc:ca:false -keystore float.jks -storepass floatpass -keypass floatpass
+
+    keytool.exe -exportcert -rfc -alias floatroot -keystore floatca.jks -storepass capass -keypass cakeypass > root.pem
+    keytool.exe -importcert -noprompt -file root.pem -alias root -keystore trust.jks -storepass trustpass
+
+    keytool.exe -certreq -alias bridgecert -keystore bridge.jks -storepass bridgepass -keypass bridgepass |keytool.exe -gencert -ext ku:c=dig,keyEncipherment -ext: eku:true=serverAuth,clientAuth -rfc -keystore floatca.jks -alias floatroot -storepass capass -keypass cakeypass > bridge.pem
+    cat root.pem bridge.pem >> bridgechain.pem
+    keytool.exe -importcert -noprompt -file bridgechain.pem -alias bridgecert -keystore bridge.jks -storepass bridgepass -keypass bridgepass
+
+    keytool.exe -certreq -alias floatcert -keystore float.jks -storepass floatpass -keypass floatpass |keytool.exe -gencert -ext ku:c=dig,keyEncipherment -ext: eku::true=serverAuth,clientAuth -rfc -keystore floatca.jks -alias floatroot -storepass capass -keypass cakeypass > float.pem
+    cat root.pem float.pem >> floatchain.pem
+    keytool.exe -importcert -noprompt -file floatchain.pem -alias floatcert -keystore float.jks -storepass floatpass -keypass floatpass
+
+The resulting ``trust.jks`` and ``bridge.jks`` should be copied to a ``<workspace>/bridgecerts`` folder on the ``BridgeInner`` instances. The ``trust.jks`` and ``float.jks`` should be copied to a ``<workspace>/floatcerts`` folder on the ``FloatOuter`` instances.
+
+Typical configuration for ``nodeserver1`` would be a ``node.conf`` files containing:
+
+..  code-block:: javascript
+
+    myLegalName = "O=Bank A,L=London,C=GB"
+    p2pAddress = "banka.com:10005" // Host and port exposed by Internet facing firewall/load balancer in front of float servers in DMZ.
+    messagingServerAddress = "nodeserver1:11005" // specifying a different port to the advertised port as an example
+    messagingServerExternal = false // override default behaviour and ensure that Artemis runs in process.
+    rpcSettings {
+        address = "nodeserver1:10006"
+        adminAddress = "nodeserver1:10026"
+    }
+    dataSourceProperties { // Point at clustered Azure SQL Server
+        dataSourceClassName = "com.microsoft.sqlserver.jdbc.SQLServerDataSource"
+        dataSource.url = "jdbc:sqlserver://nodedb.database.windows.net:1433;databaseName=corda;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30"
+        dataSource.user = Corda
+        dataSource.password = password
+    }
+    database {
+        transactionIsolationLevel = READ_COMMITTED
+        runMigration = false
+        schema = dbo
+    }
+    security {
+        authService {
+            dataSource {
+                type = INMEMORY
+                users = [
+                    {
+                        password = password
+                        permissions = [
+                            ALL
+                        ]
+                        username=user
+                    }
+                ]
+        }
+        }
+    }
+    useTestClock = false
+    enterpriseConfiguration = {
+        externalBridge = true // Ensure node doesn't run P2P AMQP bridge, instead delegate to the BridgeInner.
+        mutualExclusionConfiguration = { // Enable the protective heartbeat logic so that only one node instance is ever running.
+            on = true
+            machineName = "nodeserver1"
+            updateInterval = 20000
+            waitInterval = 40000
+        }
+    }
+    compatibilityZoneURL = "http://r3-doorman:10001"
+    devMode = false // Turn off things like key autogeneration and require proper doorman registration.
+
+Typical configuration for ``nodeserver2`` would be a ``node.conf`` files containing:
+
+..  code-block:: javascript
+
+    myLegalName = "O=Bank A,L=London,C=GB"
+    p2pAddress = "banka.com:10005" // Host and port exposed by Internet facing firewall/load balancer in front of float servers in DMZ.
+    messagingServerAddress = "nodeserver2:11005" // specifying a different port to the advertised port as an example
+    messagingServerExternal = false // override default behaviour and ensure that Artemis runs in process.
+    rpcSettings {
+        address = "nodeserver2:10006"
+        adminAddress = "nodeserver2:10026"
+    }
+    dataSourceProperties { // Point at clustered Azure SQL Server
+        dataSourceClassName = "com.microsoft.sqlserver.jdbc.SQLServerDataSource"
+        dataSource.url = "jdbc:sqlserver://nodedb.database.windows.net:1433;databaseName=corda;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30"
+        dataSource.user = Corda
+        dataSource.password = password
+    }
+    database {
+        transactionIsolationLevel = READ_COMMITTED
+        runMigration = false
+        schema = dbo
+    }
+    security {
+        authService {
+            dataSource {
+                type = INMEMORY
+                users = [
+                    {
+                        password = password
+                        permissions = [
+                            ALL
+                        ]
+                        username=user
+                    }
+                ]
+        }
+        }
+    }
+    useTestClock = false
+    enterpriseConfiguration = {
+        externalBridge = true // Ensure node doesn't run P2P AMQP bridge, instead delegate to the BridgeInner.
+        mutualExclusionConfiguration = { // Enable the protective heartbeat logic so that only one node instance is ever running.
+            on = true
+            machineName = "nodeserver2"
+            updateInterval = 20000
+            waitInterval = 40000
+        }
+    }
+    compatibilityZoneURL = "http://r3-doorman:10001"
+    devMode = false // Turn off things like key autogeneration and require proper doorman registration.
+
+
+Configuration in ``bridge.conf`` for ``bridgeserver1``:
+
+..  code-block:: javascript
+    
+    bridgeMode = BridgeInner // Set the mode the  corda-bridgeserver.jar runs as appropriately.
+    outboundConfig { // Required section
+        artemisBrokerAddress = "nodeserver1:11005" // point at primary Artemis address in the node
+        alternateArtemisBrokerAddresses = [ "nodeserver2:11005" ] // List any other HA Artemis addresses
+        socksProxyConfig { // Enable SOCKS proxying by specifying this section
+            version = SOCKS5
+            proxyAddress = "proxyserver:12345"
+            username = "proxyuser"
+            password = "password"
+        }
+    }
+    bridgeInnerConfig { // Required section
+        floatAddresses = ["dmzinternal1:12005", "dmzinternal2:12005"] // The BridgeInner initiates a connection to one of this pool of FloatOuter and round-robins
+        expectedCertificateSubject = "CN=Float Local,O=Tunnel,L=London,C=GB" // This must align with the certificate subject used by the FloatOuter control.
+        customSSLConfiguration { // Use a tunnel specific set of certificates distinct from the node's sslkeystore.jks and truststore.jks.
+               keyStorePassword = "bridgepass"
+               trustStorePassword = "trustpass"
+               sslKeystore = "./bridgecerts/bridge.jks"
+               trustStoreFile = "./bridgecerts/trust.jks"
+        }
+    }
+    haConfig { // Enable HA pointing at Zookeeper cluster for master selection.
+        haConnectionString = "zk://zookeep1:11105,zk://zookeep2:11105 ,zk://zookeep3:11105"
+    }
+    networkParametersPath = network-parameters // The network-parameters file is expected to be copied from the node after registration and here is expected in the workspace folder.
+
+Configuration in ``bridge.conf`` for ``bridgeserver2``:
+
+..  code-block:: javascript  
+
+    bridgeMode = BridgeInner // Set the mode the  corda-bridgeserver.jar runs as appropriately.
+    outboundConfig { // Required section
+        artemisBrokerAddress = "nodeserver2:11005" // point at primary Artemis address in the node
+        alternateArtemisBrokerAddresses = [ "nodeserver1:11005" ] // List any other HA Artemis addresses
+        socksProxyConfig { // Enable SOCKS proxying by specifying this section
+            version = SOCKS5
+            proxyAddress = "proxyserver:12345"
+            username = "proxyuser"
+            password = "password"
+        }
+    }
+    bridgeInnerConfig { // Required section
+        floatAddresses = ["dmzinternal2:12005", "dmzinternal1:12005"] // The BridgeInner initiates a connection to one of this pool of FloatOuter and round-robins
+        expectedCertificateSubject = "CN=Float Local,O=Tunnel,L=London,C=GB" // This must align with the certificate subject used by the FloatOuter control.
+        customSSLConfiguration { // Use a tunnel specific set of certificates distinct from the node's sslkeystore.jks and truststore.jks.
+               keyStorePassword = "bridgepass"
+               trustStorePassword = "trustpass"
+               sslKeystore = "./bridgecerts/bridge.jks"
+               trustStoreFile = "./bridgecerts/trust.jks"
+        }
+    }
+    haConfig { // Enable HA pointing at Zookeeper cluster for master selection.
+        haConnectionString = "zk://zookeep1:11105,zk://zookeep2:11105 ,zk://zookeep3:11105"
+    }
+    networkParametersPath = network-parameters // The network-parameters file is expected to be copied from the node after registration and here is expected in the workspace folder.
+
+    
+Configuration in ``bridge.conf`` for ``floatserver1``:
+
+..  code-block:: javascript  
+
+    bridgeMode = FloatOuter // Set the mode the  corda-bridgeserver.jar runs as appropriately.
+    inboundConfig { // Required section
+        listeningAddress = "dmzexternal1:10005" // expose the listening port on the out NIC
+    }
+    floatOuterConfig { // Required section
+        floatAddress = "dmzinternal1:12005" // await control instructions on inner NIC
+        expectedCertificateSubject = "CN=Bridge Local,O=Tunnel,L=London,C=GB" // Must line up with X500 Subject of certificates on BridgeInner
+        customSSLConfiguration {
+               keyStorePassword = "floatpass"
+               trustStorePassword = "trustpass"
+               sslKeystore = "./floatcerts/float.jks"
+               trustStoreFile = "./floatcerts/trust.jks"
+        }
+    }
+    networkParametersPath = network-parameters // The network-parameters file is expected to be copied from the node after registration and here is expected in the workspace folder.
+    
+Configuration in ``bridge.conf`` for ``floatserver2``:
+
+..  code-block:: javascript  
+
+    bridgeMode = FloatOuter // Set the mode the  corda-bridgeserver.jar runs as appropriately.
+    inboundConfig { // Required section
+        listeningAddress = "dmzexternal2:10005" // expose the listening port on the out NIC
+    }
+    floatOuterConfig { // Required section
+        floatAddress = "dmzinternal2:12005" // await control instructions on inner NIC
+        expectedCertificateSubject = "CN=Bridge Local,O=Tunnel,L=London,C=GB" // Must line up with X500 Subject of certificates on BridgeInner
+        customSSLConfiguration {
+               keyStorePassword = "floatpass"
+               trustStorePassword = "trustpass"
+               sslKeystore = "./floatcerts/float.jks"
+               trustStoreFile = "./floatcerts/trust.jks"
+        }
+    }
+    networkParametersPath = network-parameters // The network-parameters file is expected to be copied from the node after registration and here is expected in the workspace folder.
+
+

docs/source/corda-bridge-component.rst

@@ -0,0 +1,186 @@
+Corda Bridge Component Overview
+===============================
+
+.. contents::
+
+Introduction
+------------
+The Corda ``bridge``/``float`` component is designed for enterprise deployments and acts as an application level
+firewall and protocol break on all internet facing endpoints. The ``corda-bridgeserver.jar`` encapsulates the peer
+network functionality of the basic Corda ``node``, so that this can be operated separately from the security sensitive
+JVM runtime of the ``node``. This gives separation of functionality and ensures that the legal identity keys are not
+used in the same process as the internet TLS connections. Also, it adds support for enterprise deployment requirements,
+such as High Availability (HA) and SOCKS proxy support.
+
+This document is intended to provide an overview of the architecture and options available.
+
+Terminology
+-----------
+The component referred to here as the ``bridge`` is the library of code responsible for managing outgoing links to peer
+nodes and implements the AMQP 1.0 protocol over TLS 1.0 between peers to provide reliable flow message delivery. This
+component can be run as a simple integrated feature of the ``node``. However, for enhanced security and features on R3
+Enterprise Corda the in-node version should be turned off and a standalone and HA version can be run from the
+``corda-bridgeserver.jar``, possibly integrating with a SOCKS proxy too.
+
+The ``float`` component refers to the inbound socket listener, packet filtering and DMZ compatible component. In the
+simple all-in-one ``node`` all inbound peer connections terminate directly onto an embedded Artemis broker component
+hosted within the ``node``. The connection authentication and packet the filtering is managed directly via Artemis
+permission controls managed directly inside the ``node`` JVM. For R3 Enterprise Corda deployments we provide a more
+secure and configurable isolation component that is available using code inside ``corda-bridgeserver.jar``. This
+component is designed to provide a clear protocol break and thus prevents the ``node`` and Artemis server ever being
+directly exposed to peers. For simpler deployments with no DMZ the ``float`` and ``bridge`` logic can also be run as a
+single application behind the firewall, but still protecting the ``node`` and hosted Artemis. In future we may also host
+the Artemis server out of process and shared across nodes, but this will be transparent to peers as the interchange
+protocol will continue to be AMQP 1.0 over TLS.
+
+.. Note:: All deployment modes of the ``bridge``, ``float``, or all-in-one ``node`` are transparently interoperable, if correctly configured.
+
+Message Path Between Peer Nodes
+-------------------------------
+When a flow within a ``node`` needs to send a message to a peer there is a carefully orchestrated sequence of steps to ensure
+correct secure routing based upon the network map information and to ensure safe, restartable delivery to the remote flow.
+Adding the ``bridge`` and ``float`` to this process adds some extra steps and security checking of the messages.
+The complete sequence is therefore:
+
+1.   The flow calls ``send``, or ``sendAndReceive`` to propagate a message to a peer. This leads to checkpointing
+     of the flow fiber within the ``StateMachine`` and posting the message to the internal ``MessagingService``. This ensures that
+     the send activity will be retried if there are any errors before further durable transmission of the message.
+
+2.   The ``MessagingService`` checks if this is a new destination ``node`` and if an existing out queue and bridge exists in Artemis.
+     If the durable out queue does not exist then this will need to be created in Artemis:
+
+     a.   First the durable queue needs to be created in the peer-to-peer Artemis. Each queue is uniquely named based upon the hash of the
+          legal identity ``PublicKey`` of the target ``node``.
+
+     b.   Once the queue creation is complete a bridge creation request is also published onto the Artemis bus via the bridge control protocol.
+          This message uses information from the network map to link the out queue to the target host and port and TLS credentials.
+          The flow does not need to wait for any response at this point and can carry on to send messages to the Artemis out queue.
+
+     c.   The message when received by the ``bridge`` process opens a TLS connection to the remote peer (optionally, this
+          connection can be made via a SOCKS4/5 proxy). On connect the two ends of the TLS link exchange certificate details
+          and confirm that the certificate path is anchored at the network root certificate and that the X500 subject matches
+          the expected target as specified in the create bridge message using details contained in the network map.
+          The links are long lived so as to reduce the setup cost of the P2P messaging.
+          In future, there may also be DOS protection measures applied.
+
+     d.   If the outgoing TLS 1.2 link is created successfully then the ``bridge`` opens a consumer on the Artemis out queue.
+          The pending messages will then be transferred to the remote destination using AMQP 1.0, with final removal from the
+          out queue only occurring when the remote end fully acknowledges safe message receipt. This ensures at least once
+          delivery semantics.
+
+     e.   Note that at startup of either the ``node``, or the ``bridge`` the bridge control protocol resynchronises the bridging state,
+          so that all out queues have an active bridge.
+
+3.   Assuming an out queue exists the message can be posted to Artemis and the ``bridge`` should eventually deliver this
+     message to the remote system.
+
+4.   On receipt of a message acknowledge from Artemis the ``StateMachine`` can continue flow if it is not awaiting a response
+     i.e. a ``send`` operation. Otherwise it remains suspended waiting for the reply.
+
+5.   The receiving end of the ``bridge`` TLS/AMQP 1.0 link might be the Artemis broker of a remote ``node``,
+     but for now we assume it is an enterprise deployment that is using a ``float`` process running behind a firewall.
+     The receiver will already  have confirmed the validity of the TLS originator when it accepted the TLS handshake.
+     However, the ``float`` does some further basic checking of received messages and their associated headers.
+     For instance the message must be targeted at an inbox address and must be below the network parameters defined ``maxMessageSize``.
+
+6.   Having passed initial checks on the message the ``float`` bundles up the message and originator as a payload to be
+     sent across the DMZ internal firewall. This inbound message path uses a separate AMQP 1.0/TLS control tunnel.
+     (N.B. This link is initiated from the local master ``bridge`` in the trusted zone to the ``float`` in the DMZ. This allows a
+     simple firewall rule to be configured which blocks any attempts to probe the internal network from the DMZ.)
+     Once the message is forwarded the ``float`` keeps track of the delivery acknowledgements,
+     so that the original sender will consume the message in the source queue, only on final delivery to the peer inbox.
+     Any disconnections, or problems will send a reject status leading to redelivery from source.
+
+7.   The ``bridge`` process having now received custody of the message does further checks that the message is good. At the
+     minute the checks are essentially of well formedness of the message and that the source and destination are valid.
+     However, future enhancements may include deep inspection of the message payload for CorDapp blacklisting, and other purposes.
+     Any problems and the message is acknowledged to prevent further redelivery, logged to audit and dropped.
+
+8.   Assuming this is a normal message it is passed onto the Artemis inbox and on acknowledgment of delivery
+     is cascaded back. Thus, Artemis acknowledgement, leads to acknowledgement of the tunnel AMQP packet,
+     which acknowledges the AMQP back to the sending ``bridge`` and that finally marks the Artemis out queue item as consumed.
+     To prevent this leading to very slow one after the other message delivery the AMQP channels using sliding window flow control.
+     (Currently, a practical default is set internally and the window size is not user configurable.)
+
+9.   The ``MessagingService`` on the peer ``node`` will pick up the message from inbox on Artemis, carry out any necessary
+     deduplication. This deduplication is needed as the distributed restartable logic of the Corda wire protocol only
+     offers 'at least once' delivery guarantees.
+     The resulting unique messages are then passed to the ``StateMachine`` so that the remote flow can be woken up.
+
+10.  The reply messages use the authenticated originator flag attached by the ``float`` to route the replies back to the
+     correct originator.
+
+     .. Note::   That the message reply path is not via the inbound path, but instead is via a separately validated route
+        from the local ``bridge`` to the original ``node``'s ``float`` and then on to the original ``node`` via Artemis.
+
+Operating modes of the Bridge and Float
+---------------------------------------
+
+Embedded Developer Node (node + artemis + internal bridge, no float, no DMZ)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The simplest development deployment of the bridge is to just use the embedded Peer-to-Peer Artemis with the node as TLS endpoint
+and to have the outgoing packets use the internal bridge functionality. Typically this should only be used for easy development,
+or for organisations evaluating on Open Source Corda, where this is the only available option:
+
+.. image:: resources/bridge/node_embedded_bridge.png
+     :scale: 100%
+     :align: center
+
+Node + Bridge (no float, no DMZ)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The next simplest deployment is to turn off the built in bridge using the ``externalBridge`` enterprise config property
+and to run a single combined ``bridge``/``float`` process. This might be suitable for a test environment, to conserve VM's.
+
+ .. note::  Note that to run the ``bridge`` and the ``node`` on the same machine there could be a port conflict with a naive setup,
+            but by using the ``messagingServerAddress`` property to specify the bind address and port plus setting
+            ``messagingServerExternal = false``
+            the embedded Artemis P2P broker can be set to listen on a different port rather than the advertised ``p2paddress`` port.
+            Then configure an all-in-one ``bridge`` to point at this ``node``:
+
+.. image:: resources/bridge/simple_bridge.png
+     :scale: 100%
+     :align: center
+
+DMZ ready (node + bridge + float)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To familiarize oneself with the a more complete deployment including a DMZ and separated inbound and outbound paths
+the ``bridgeMode`` property in the ``bridge.conf`` should be set to ``BridgeInner`` for the ``bridge`` and
+``FloatOuter`` for the DMZ ``float``. The diagram below shows such a non-HA deployment. This would not be recommended
+for production, unless used as part of a cold DR type standby.
+
+.. note::  Note that whilst the ``bridge`` needs access to the official TLS private
+        key, the tunnel link should use a private set of link specific keys and certificates. The ``float`` will be provisioned
+        dynamically with the official TLS key when activated via the tunnel and this key will never be stored in the DMZ:
+
+.. image:: resources/bridge/bridge_and_float.png
+     :scale: 100%
+     :align: center
+
+DMZ ready with outbound SOCKS
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Some organisations require dynamic outgoing connections to operate via a SOCKS proxy. The code supports this option
+by adding extra information to the ``outboundConfig`` section of the ``bridge`` process. An simplified example deployment is shown here
+to highlight the option:
+
+.. image:: resources/bridge/bridge_with_socks.png
+     :scale: 100%
+     :align: center
+
+Full production HA DMZ ready mode (hot/cold node, hot/warm bridge)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Finally, we show a full HA solution as recommended for production. This does require adding an external ``Zookeeper``
+cluster to provide ``bridge`` master selection and extra instances of the ``bridge`` and ``float``. This allows
+hot-warm operation of all the ``bridge`` and ``float`` instances. The R3 enterprise Corda ``node`` should be run as hot-cold HA too.
+Highlighted in the diagram is the addition of the ``haConfig`` section to point at ``zookeeper`` and also the use of secondary
+addresses in the ``alternateArtemisAddresses`` to allow ``node`` failover and in the ``floatAddresses`` to point at a
+pool of DMZ ``float`` processes.:
+
+.. image:: resources/bridge/ha_bridge_float.png
+     :scale: 100%
+     :align: center
+

docs/source/getting-set-up.rst

@@ -28,6 +28,10 @@ Please note:
 
 * IntelliJ IDEA is recommended due to the strength of its Kotlin integration.
 
+* If an HA Bridge/Float deployment is required then a ``Zookeeper 3.5.3-Beta`` cluster will be required.
+  Refer to :doc:`High Availability <high-availability>` and :doc:`Bridge configuration <bridge-configuration-file>`
+  for more deployment information.
+
 Following these software recommendations will minimize the number of errors you encounter, and make it easier for
 others to provide support. However, if you do use other tools, we'd be interested to hear about any issues that arise.
 

docs/source/operations-guide.rst

@@ -2,3 +2,9 @@ Operations Guide
 ================
 
 * :doc:`Database Migration <database-migration>`
+
+* :doc:`High Availability <high-availability>`
+
+* :doc:`Bridge Configuration <bridge-configuration-file>`
+
+* :doc:`Corda Bridge Component Overview <corda-bridge-component>`