From bc93275f00badba594b32a6251f1bbe94d880ba6 Mon Sep 17 00:00:00 2001 From: Thomas Schroeter Date: Wed, 13 Jun 2018 13:18:44 +0100 Subject: [PATCH] Add HA notary setup tutorial (#937) --- .../installing-percona.rst | 180 +++++++++++++++++ ...alling-the-notary-service-bootstrapper.rst | 33 ++++ ...stalling-the-notary-service-monitoring.rst | 6 + .../installing-the-notary-service-netman.rst | 155 +++++++++++++++ .../installing-the-notary-service.rst | 39 ++++ .../running-a-notary-cluster/introduction.rst | 181 ++++++++++++++++++ .../operating-percona.rst | 129 +++++++++++++ .../resources/doorman-light.png | Bin 0 -> 7199 bytes .../resources/ha-notary-overview2.png | Bin 0 -> 18717 bytes .../resources/node.conf | 42 ++++ .../resources/percona-colocated.png | Bin 0 -> 4678 bytes .../resources/wsrep.cnf | 48 +++++ .../running-a-notary-cluster/toctree.rst | 13 ++ docs/source/tutorials-index.rst | 3 +- 14 files changed, 828 insertions(+), 1 deletion(-) create mode 100644 docs/source/running-a-notary-cluster/installing-percona.rst create mode 100644 docs/source/running-a-notary-cluster/installing-the-notary-service-bootstrapper.rst create mode 100644 docs/source/running-a-notary-cluster/installing-the-notary-service-monitoring.rst create mode 100644 docs/source/running-a-notary-cluster/installing-the-notary-service-netman.rst create mode 100644 docs/source/running-a-notary-cluster/installing-the-notary-service.rst create mode 100644 docs/source/running-a-notary-cluster/introduction.rst create mode 100644 docs/source/running-a-notary-cluster/operating-percona.rst create mode 100644 docs/source/running-a-notary-cluster/resources/doorman-light.png create mode 100644 docs/source/running-a-notary-cluster/resources/ha-notary-overview2.png create mode 100644 docs/source/running-a-notary-cluster/resources/node.conf create mode 100644 docs/source/running-a-notary-cluster/resources/percona-colocated.png create mode 100644 docs/source/running-a-notary-cluster/resources/wsrep.cnf create mode 100644 docs/source/running-a-notary-cluster/toctree.rst diff --git a/docs/source/running-a-notary-cluster/installing-percona.rst b/docs/source/running-a-notary-cluster/installing-percona.rst new file mode 100644 index 0000000000..3157e91649 --- /dev/null +++ b/docs/source/running-a-notary-cluster/installing-percona.rst @@ -0,0 +1,180 @@ +================================ +Percona, the underlying Database +================================ + +Percona's `documentation page `__ explains the installation in detail. + +In this section we're setting up a +three-node Percona cluster. A three-node cluster can tolerate one crash +fault. In production, you probably want to run five nodes, to be able to +tolerate up to two faults. + +Host names and IP addresses used in the example are listed in the table below. + +========= ======== +Host IP +========= ======== +percona-1 10.1.0.1 +percona-2 10.1.0.2 +percona-3 10.1.0.3 +========= ======== + +Installation +============ + +Percona provides repositories for the YUM and APT package managers. +Alternatively you can install from source. For simplicity, we are going to +install Percona using the default data directory ``/var/lib/mysql``. + +.. note:: + + The steps below should be run on all your Percona nodes, unless otherwise + mentioned. You should write down the host names or IP addresses of all your + Percona nodes before starting the installation, to configure the data + replication and later to configure the JDBC connection of your notary + cluster. + +Run the commands below on all nodes of your Percona cluster to configure the +Percona repositories and install the service. + +.. code:: sh + + wget https://repo.percona.com/apt/percona-release_0.1-4.$(lsb_release -sc)_all.deb + sudo dpkg -i percona-release_0.1-4.$(lsb_release -sc)_all.deb + sudo apt-get update + sudo apt-get install percona-xtradb-cluster-57 + +The service will start up automatically after the installation, you can confirm that the service is +running with ``service mysql status``, start the service with ``sudo service mysql start`` and stop with +``sudo service mysql stop``. + +Configuration +============= + +Configure the MySQL Root Password (if necessary) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Some distros allow root access to the database through a Unix domain socket, others +require you to find the temporary password in the log file and change it upon +first login. + +Stop the Service +^^^^^^^^^^^^^^^^ + +.. code:: sh + + sudo service mysql stop + +Setup replication +^^^^^^^^^^^^^^^^^ + +Variables you need to change from the defaults are listed in the table below. + +====================== =========================================================== ========================================================== +Variable Name Example Description +====================== =========================================================== ========================================================== +wsrep_cluster_address gcomm://10.1.0.1,10.1.0.2,10.1.0.3 The addresses of all the cluster nodes (host and port) +wsrep_node_address 10.1.0.1 The address of the Percona node +wsrep_cluster_name notary-cluster-1 The name of the Percona cluster +wsrep_sst_auth username:password The credentials for SST +wsrep_provider_options "gcache.size=8G" Replication options +====================== =========================================================== ========================================================== + +Configure all replicas via +``/etc/mysql/percona-xtradb-cluster.conf.d/wsrep.cnf`` as shown in the template +below. + +.. literalinclude:: resources/wsrep.cnf + :caption: wsrep.cnf + :name: wsrep-cnf + +The file ``/etc/mysql/percona-xtradb-cluster.conf.d/mysqld.cnf`` contains additional settings like the data directory. We're assuming +you keep the default ``/var/lib/mysql``. + +Configure AppArmor, SELinux or other Kernel Security Module +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you're changing the location of the database data directory, you might need to +configure your security module accordingly. + +On the first Percona node +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Start the Database +~~~~~~~~~~~~~~~~~~ + +.. code:: sh + + sudo /etc/init.d/mysql bootstrap-pxc + + +Watch the logs using ``tail -f /var/log/mysqld.log``. Look for a log entry like +``WSREP: Setting wsrep_ready to true``. + +Create the Corda User +~~~~~~~~~~~~~~~~~~~~~ + +.. code:: sql + + CREATE USER corda IDENTIFIED BY '{{ password }}'; + +Create the Database and Tables +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code:: sql + + CREATE DATABASE corda; + + CREATE TABLE IF NOT EXISTS corda.notary_committed_states ( + issue_transaction_id BINARY(32) NOT NULL, + issue_transaction_output_id INT UNSIGNED NOT NULL, + consuming_transaction_id BINARY(32) NOT NULL, + CONSTRAINT id PRIMARY KEY (issue_transaction_id, issue_transaction_output_id) + ); + + GRANT SELECT, INSERT ON corda.notary_committed_states TO 'corda'; + + CREATE TABLE IF NOT EXISTS corda.notary_request_log ( + consuming_transaction_id BINARY(32) NOT NULL, + requesting_party_name TEXT NOT NULL, + request_signature BLOB NOT NULL, + request_date TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP, + request_id INT UNSIGNED NOT NULL AUTO_INCREMENT, + CONSTRAINT rid PRIMARY KEY (request_id) + ); + + GRANT INSERT ON corda.notary_request_log TO 'corda'; + +Create the SST User +~~~~~~~~~~~~~~~~~~~ + +.. code:: sql + + CREATE USER ‘{{ sst_user }}’@’localhost’ IDENTIFIED BY ‘{{ sst_pass }}‘; + GRANT RELOAD, LOCK TABLES, PROCESS, REPLICATION CLIENT ON *.* TO ‘{{ sst_user }}’@’localhost’; + FLUSH PRIVILEGES; + + +On all other Nodes +^^^^^^^^^^^^^^^^^^ + +Once you have updated the ``wsrep.cnf`` on all nodes, start MySQL on all the +remaining nodes of your cluster. Run this command on all nodes of your cluster, +except the first one. The config file is shown `above <#wsrep-cnf>`__. + +.. code:: sh + + service mysql start + +Watch the logs using ``tail -f /var/log/mysqld.log``. Make sure you can start +the MySQL client on the command line and access the ``corda`` database on all +nodes. + +.. code:: sh + + mysql + mysql> use corda; + # The output should be `Database changed`. + +In the next section, we're :doc:`installing-the-notary-service`. You can read about :doc:`operating-percona` in a later section of this tutorial. + diff --git a/docs/source/running-a-notary-cluster/installing-the-notary-service-bootstrapper.rst b/docs/source/running-a-notary-cluster/installing-the-notary-service-bootstrapper.rst new file mode 100644 index 0000000000..0ef0b9e825 --- /dev/null +++ b/docs/source/running-a-notary-cluster/installing-the-notary-service-bootstrapper.rst @@ -0,0 +1,33 @@ +Using the Bootstrapper +++++++++++++++++++++++ + +You can skip this section when you're setting up or joining a cluster with +doorman and network map. + +Once the database is set up, you can prepare your configuration files of your notary +nodes and use the bootstrapper to create a Corda network, see +:doc:`../setting-up-a-corda-network`. Remember to configure +``notary.serviceLegalName`` in addition to ``myLegalName`` for all members of +your cluster. + +You can find the documentation of the bootstrapper at :doc:`../setting-up-a-corda-network`. + +Expected Outcome +~~~~~~~~~~~~~~~~ + +You will go from a set of configuration files to a directory tree containing a fully functional Corda network. + +The notaries will be visible and available on the network. You can list available notaries using the node shell. + +.. code:: sh + + run notaryIdentities + +The output of the above command should include the ``notary.serviceLegalName`` +you have configured, e.g. ``O=HA Notary, L=London, C=GB``. + +CorDapp developers should select the notary service identity from the network map cache. + +.. code:: kotlin + + serviceHub.networkMapCache.getNotary(CordaX500Name("HA Notary", "London", "GB")) diff --git a/docs/source/running-a-notary-cluster/installing-the-notary-service-monitoring.rst b/docs/source/running-a-notary-cluster/installing-the-notary-service-monitoring.rst new file mode 100644 index 0000000000..6764008373 --- /dev/null +++ b/docs/source/running-a-notary-cluster/installing-the-notary-service-monitoring.rst @@ -0,0 +1,6 @@ + +Monitoring +++++++++++ + +The notary health-check CorDapp monitors all notaries of the network and +serves a metrics website (TBD where the documentation is hosted). diff --git a/docs/source/running-a-notary-cluster/installing-the-notary-service-netman.rst b/docs/source/running-a-notary-cluster/installing-the-notary-service-netman.rst new file mode 100644 index 0000000000..c660616843 --- /dev/null +++ b/docs/source/running-a-notary-cluster/installing-the-notary-service-netman.rst @@ -0,0 +1,155 @@ +In a network with Doorman and Network map ++++++++++++++++++++++++++++++++++++++++++ + +You can skip this section if you're not setting up or joining a network with +doorman and network map service. + +Expected Outcome +~~~~~~~~~~~~~~~~ + +You will go from a set of configuration files to a fully functional Corda network. +The network map will be advertising the service identity of the Notary. Every +notary replica has obtained its own identity and the shared service identity +from the doorman. + +Using the registration tool, you will obtain the serivce identity of your notary +cluster and distribute it to the keystores of all replicas of your notary +cluster. + +The individual node identity is used by the messaging layer to route requests to +individual notary replicas. The notary replicas sign using the service identity +private key. + +======== =========================== =========================== +Host Individual identity Service identity +======== =========================== =========================== +notary-1 O=Replica 1, L=London, C=GB O=HA Notary, L=London, C=GB +notary-2 O=Replica 2, L=London, C=GB O=HA Notary, L=London, C=GB +notary-3 O=Replica 3, L=London, C=GB O=HA Notary, L=London, C=GB +======== =========================== =========================== + +The notaries will be visible and available on the network. To list available notary +identities using the Corda node shell + +.. code:: sh + + run notaryIdentities + +The output of the above command should include ``O=HA Notary, L=London, C=GB``. + +CorDapp developers should select the notary service identity from the network map cache. + +.. code:: kotlin + + serviceHub.networkMapCache.getNotary(CordaX500Name("HA Notary", "London", "GB")) + +Every notary replica's keystore contains the private key of the replica and the +private key of the notary service (with aliases ``identity-private-key`` and +``distributed-notary-private key`` in the keystore). We're going to create and +populate the node's keystores later in this tutorial. + +The Notary, the Doorman and the Network Map +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The notary is an essential component of every Corda network, therefore the +Notary identity needs to be created first, before other nodes can join the +network, since the Notary identity is part of the network parameters. +Adding a Corda notary to an existing network is covered in +the network services documentation (TBD where this is hosted). Removing a notary from a network +is currently not supported. + +.. image:: resources/doorman-light.png + :scale: 70% + + +The notary sends a certificate signing request (CSR) to the doorman for +approval. Once approved, the notary obtains a signed certificate from the +doorman. The notary can then produce a signed node info file that contains the +P2P addresses, the legal identity, certificate and public key. The node infos +of all notaries that are part of the network are included in the network +parameters. Therefore, the notary node info files have to be present when the +network parameters are created. + +Registering with the Doorman +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Obtaining the individual Node Identities +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Write the configuration files for your replicas as described in :doc:`installing-the-notary-service`. + +Register all the notary replicas with the doorman using the ``--initial-registration`` flag. + +.. code:: sh + + java -jar corda.jar --initial-registration \ + --network-root-truststore-password '{{ root-truststore-password }}' \ + --network-root-truststore network-root-truststore.jks + +Obtaining the distributed Service Identity +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Once you completed the initial registration for all notary nodes, you can use +the registration tool to submit the certificate signing request (CSR) for the +service identity of your notary cluster. Read the documentation about the +`registration tool `__ +for detailed instructions. + +Use the configuration file template below to configure the registration tool. + +:: + + legalName = "{{ X500 name of the notary service }}" + email = "test@email.com" + compatibilityZoneURL = "https://{{ host }}:{{ port }}" + networkRootTrustStorePath = "network-root-truststore.jks" + + networkRootTrustStorePassword = "" + keyStorePassword = "" + trustStorePassword = "" + + crlCheckSoftFail = true + +Run the command below to obtain the service identity of the notary cluster. + +.. code:: sh + + java -jar registration-tool.jar --config-file '{{ registation-config-file }}' + +The service identity will be stored in a file +``certificates/notaryidentitykeystore.jks``. Distribute the +``distributed-notary-private-key`` into the keystores of all notary nodes that +are part of the cluster as follows: + +* Copy the notary service identity to all notary nodes, placing it in the same directory as the ``nodekeystore.jks`` file and run the following command to import the service identity into the node's keystore: + +.. code:: sh + + registration-tool.jar --importkeystore \ + --srcalias distributed-notary-private-key \ + --srckeystore certificates/notaryidentitykeystore.jks \ + --destkeystore certificates/nodekeystore.jks + +* Check that the private keys are available in the keystore using the following command + +.. code:: sh + + keytool -list -v -keystore certificates/nodekeystore.jks | grep Alias + + # Output: + # Alias name: cordaclientca + # Alias name: identity-private-key + # Alias name: distributed-notary-private-key + +Network Map: Setting the Network Parameters +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This step is only applicable if you're the operator of the network map service. +In case the network map is operated by somebody else, you might need to send +them the node-info file of one of your notary nodes for inclusion in the +network parameters. + +Copy the node info file of one of the notary replicas to the network map to +include the service identity in the network parameters. Follow the +instructions in the manual of the network services to generate the network +parameters (TBD where the documentation is hosted). diff --git a/docs/source/running-a-notary-cluster/installing-the-notary-service.rst b/docs/source/running-a-notary-cluster/installing-the-notary-service.rst new file mode 100644 index 0000000000..1d7bf68aeb --- /dev/null +++ b/docs/source/running-a-notary-cluster/installing-the-notary-service.rst @@ -0,0 +1,39 @@ +============================= +Setting up the Notary Service +============================= + +In the previous section of this tutorial we set up a Percona cluster. + +On top of the Percona cluster we're deploying three Corda notary nodes ``notary-{1,2,3}`` and +a single regular Corda node ``node-1`` that runs the notary health-check CorDapp. + +If you're deploying VMs in your environment you might need to adjust the host names accordingly. + +Configuration Files ++++++++++++++++++++ + +Below is a template for the notary configuration. Notice the parameters +``rewriteBatchedStatements=true&useSSL=false&failOverReadOnly=false`` of the +JDBC URL. See :doc:`../corda-configuration-file` for a complete reference. + +Put the IP address or host name of the nearest Percona server first in the JDBC +URL. When running a Percona and a Notary replica on a single machine, list the +local IP first. + +.. literalinclude:: resources/node.conf + :caption: node.conf + :name: node-conf + +.. note:: + + Omit ``compatibilityZoneURL`` and set ``devMode = true`` when using the bootstrapper. + + +Next Steps +++++++++++ + +.. toctree:: + :maxdepth: 1 + + installing-the-notary-service-bootstrapper + installing-the-notary-service-netman diff --git a/docs/source/running-a-notary-cluster/introduction.rst b/docs/source/running-a-notary-cluster/introduction.rst new file mode 100644 index 0000000000..8be2e2e7ef --- /dev/null +++ b/docs/source/running-a-notary-cluster/introduction.rst @@ -0,0 +1,181 @@ +===================================== +Highly Available Notary Service Setup +===================================== + +About the HA Notary Installation +================================ + +In this chapter you'll learn how to set up, configure and start a highly +available (HA) Corda Notary from scratch. If you're targeting an environment +with doorman and network map, you require the registration tool. If you don't +require the doorman and network map, and you don't want to join an existing +network, the bootstrapper allows you to set up a cluster of nodes from a set of +configuration files. + +The HA Notary relies on a Percona/XtraDB (Percona) cluster. How to set up Percona +is described below. + +This guide assumes you're running a Debian-based Linux OS. + +Double curly braces ``{{ }}`` are used to represent placeholder values +throughout this guide. + +Overview +======== + +.. image:: resources/ha-notary-overview2.png + :scale: 75 % + +The figure above displays the Corda nodes in green on the top, then the Corda +notaries in red in the middle and on the bottom are the Percona nodes in blue. + +Corda nodes that request a notarisation by the service name of the Notary, +will connect to the available Notary nodes in a round-robin fashion. + +Since our Notary cluster consists of several Percona nodes and several Corda +Notary nodes, we achieve high availability (HA). Individual nodes of the +Percona and Notary clusters can fail, while clients are still able to +notarise transactions. The Notary cluster remains available. A three-node +Percona cluster as shown in the figure above can tolerate one crash fault. + +.. note:: + + In production you should consider running five nodes or more, to be able to + tolerate more than one simultaneous crash fault. One single Corda Notary + replica is enough to serve traffic in principal, although its capacity might + not be sufficient, depending on your throughput and latency requirements. + +Colocating Percona and the Notary Service ++++++++++++++++++++++++++++++++++++++++++ + +.. image:: resources/percona-colocated.png + :scale: 50% + +You can run a Percona DB service and a Corda Notary service on the same machine. + +Summary ++++++++ + +* Corda nodes communicate with the Notary cluster via P2P messaging, the messaging layer handles selecting an appropriate Notary replica by the service legal name. +* Corda nodes connect to the Notary cluster members round-robin. +* The notaries communicate with the underlying Percona cluster via JDBC. +* The Percona nodes communicate with each other via group communication (GComm). +* The Percona replicas should only be reachable from each other and from the Notary nodes. +* The Notary P2P ports should be reachable from the internet (or at least from the rest of the Corda network you're building or joining). +* We recommend running the notaries and the Percona service in a joined private subnet, opening up the P2P ports of the notaries for external traffic. + +Legal Names and Identities +++++++++++++++++++++++++++ + +Every Notary replica has two legal names. Its own legal name, specified by +``myLegalName``, e.g ``O=Replica 1, C=GB, L=London`` and the service legal name +specified in configuration by ``notary.serviceLegalName``, e.g. ``O=HA Notary, +C=GB, L=London``. Only the service legal name is included in the network +parameters. CorDapp developers should select the Notary service identity from the network map cache. + +.. code:: kotlin + + serviceHub.networkMapCache.getNotary(CordaX500Name("HA Notary", "London", "GB")) + +Every Notary replica's keystore contains the private key of the replica and the +private key of the Notary service (with aliases ``identity-private-key`` and +``distributed-notary-private key`` in the keystore). We're going to create and +populate the node's keystores later in this tutorial. + + +Choosing Installation Path +========================== + +.. note:: + + If you want to connect to a Corda network with a doorman and network map service, + use the registration tool to create your service identity. In case you want + to set up a test network for development or a private network without doorman and + network map, using the bootstrapper is recommended. + +Expected Data Volume +==================== + +For non-validating notaries the Notary stores roughly one kilobyte per transaction. + +Prerequisites +============= + +* Java runtime +* Corda JAR +* Notary Health-Check JAR +* Bootstrapper JAR (only required when setting up network without doorman and network map) +* Network Registration tool (only required when setting up a network with doorman and network map) +* Root access to a Linux machine or VM to install Percona +* The private IP addresses of your DB hosts (where we're going to install Percona) +* The public IP addresses of your Notary hosts (in order to advertise these IPs for P2P traffic) + +Your Corda distribution should contain all the JARs listed above. + + +Security +======== + +Credentials ++++++++++++ + +Make sure you have the following credentials available, create them if necessary and always +keep them safe. + +================================ ============================================================================================================ +Password or Keystore Description +================================ ============================================================================================================ +database root password used to create the Corda user, setting up the DB and tables (only required for some installation methods) +Corda DB user password used by the Notary service to access the DB +SST DB user password used by the Percona cluster for data replication (SST stands for state snapshot transfer) +Network root truststore password (not required when using the bootstrapper) +Node keystore password (not required when using the bootstrapper) +The network root truststore (not required when using the bootstrapper) +================================ ============================================================================================================ + +Networking +++++++++++ + +Percona Cluster +~~~~~~~~~~~~~~~ +===== ======================= +Port Purpose +===== ======================= +3306 MySQL client connections (from the Corda Notary nodes) +4444 SST via rsync and Percona XtraBackup +4567 Write-set replication traffic (over TCP) and multicast replication (over TCP and UDP) +4568 IST (Incremental State Transfer) +===== ======================= + +Follow the `Percona documentation `__ +if you need to encrypt the traffic between your Corda nodes and Percona and between Percona nodes. + +Corda Node +~~~~~~~~~~ + +========= ======= ============================== +Port Example Purpose +========= ======= ============================== +P2P Port 10002 P2P traffic (external) +RPC Port 10003 RPC traffic (internal only) +========= ======= ============================== + +Later in the tutorial we're covering the Notary service configuration in details, in :doc:`installing-the-notary-service`. + +Keys and Certificates ++++++++++++++++++++++ + +Keys are stored the same way as for regular Corda nodes in the ``certificates`` +directory. If you're interested in the details you can find out +more in the :doc:`../permissioning` document. + + +Next Steps +========== + +.. toctree:: + :maxdepth: 1 + + installing-percona + installing-the-notary-service + operating-percona diff --git a/docs/source/running-a-notary-cluster/operating-percona.rst b/docs/source/running-a-notary-cluster/operating-percona.rst new file mode 100644 index 0000000000..1f6802cbe2 --- /dev/null +++ b/docs/source/running-a-notary-cluster/operating-percona.rst @@ -0,0 +1,129 @@ +============================== +Percona Monitoring, Backup and Restore (Advanced) +============================== + +Monitoring +========== + +Percona Monitoring and Management (PMM) is a platform for managing and +monitoring your Percona cluster. See the `PMM documentation +`__. + +Running PMM Server +^^^^^^^^^^^^^^^^^^ + +Install PMM Server on a single machine of your cluster. + +.. code:: sh + + docker run -d \ + -p 80:80 \ + --volumes-from pmm-data \ + --name pmm-server \ + percona/pmm-server:latest + +Installing PMM Client +^^^^^^^^^^^^^^^^^^^^^^ + +You need to configure the Percona repositories first, as described above. +Install and configure PMM Client on all the machines that are running Percona. + +.. code:: sh + + sudo apt-get install pmm-client + sudo pmm-admin config --server ${PMM_HOST}:${PMM_PORT} + + + +Backup +====== + +You can take backups with the ``XtraBackup`` tool. The command below creates a +backup in ``/data/backups``. + +.. code:: sh + + xtrabackup --backup --target-dir=/data/backups/ + + +Restore +======= + +Stop the Cluster +^^^^^^^^^^^^^^^^ + +Stop the Percona cluster by shutting down nodes one by one. Prepare the backup to restore using + +.. code:: sh + + xtrabackup --prepare --target-dir=/data/backups/ + +Restore from a Backup +^^^^^^^^^^^^^^^^^^^^^ + +.. code:: sh + + mv '{{ data-directory }}' '{{ data-directory-backup }}' + xtrabackup --copy-back --target-dir=/data/backups/ + sudo chown -R mysql:mysql '{{ data-directory }}' + +Note that you might need the data in ``{{ data-direcotry-backup }}`` in case you +need to repair and replay from the binlog, as described below. + +Start the first Node +^^^^^^^^^^^^^^^^^^^^ + +.. code:: sh + + /etc/init.d/mysql bootstrap-pxc + +Repair +====== + +You can recover from some accidents, e.g. a table drop, by restoring the last +backup and then applying the binlog up to the offending statement. + +Replay the Binary Log +^^^^^^^^^^^^^^^^^^^^^ + +XtraBackup records the binlog position of the backup in +``xtrabackup_binlog_info``. Use this positon to start replaying the binlog from +your data directory (e.g. ``/var/lib/mysql``, or the target directory of the move command +used in the backup step above). + +.. code:: sh + + mysqlbinlog '{{ binlog-file }}' --start-position= > binlog.sql + +In case there are offending statements, such as +accidental table drops, you can open ``binlog.sql`` for examination. + +Optionally can also pass ``--base64-output=decode-rows`` to decode every statement into a human readable format. + +.. code:: sh + + mysqlbinlog $BINLOG_FILE --start-position=$START_POS --stop-position=$STOP_POS > binlog.sql + # Replay the binlog + mysql -u root -p < binlog.sql + +Start remaining Nodes +^^^^^^^^^^^^^^^^^^^^^ + +Finally, start the remaining nodes of the cluster. + +Restarting a Cluster +==================== +When all nodes of the cluster are down, manual intervention is needed to bring +the cluster back up. On the node with the most advanced replication index, +``set safe_to_bootstrap: 1`` in the file ``grastate.dat`` in the data directory. +You can use ``SHOW GLOBAL STATUS LIKE 'wsrep_last_committed';`` to find out the +sequence number of the last committed transaction. Start the first node using +``/etc/init.d/mysql bootstrap-pxc``. Bring back one node at a time and watch +the logs. If a SST is required, the first node can only +serve as a donor for one node a time. + +See the documentation of the safe to bootstrap feature. Similar to restoring +from backup, restarting the entire cluster is an operation that deserves +practice. See the `documentation +`__ +of this feature. diff --git a/docs/source/running-a-notary-cluster/resources/doorman-light.png b/docs/source/running-a-notary-cluster/resources/doorman-light.png new file mode 100644 index 0000000000000000000000000000000000000000..2305e9682a294a13c57b45b36d255d483cdad1e1 GIT binary patch literal 7199 zcmds6cU05Mmrv*f5v7PAO#<=^QWa5ZKqYh{1dtjrpb(mXl+cS<>Ae$$KnNlhinQ>g zcYF{)K|s0$A@q_^Ha>UH+1+#Yx9^>^=XZAZkMEh8JKsAq_s;!(?`J;q6mAUT1POuw z005_+?sZcDfB_Ex05{p_0szTBJds{7-P1DC0s!8|bL=}X)7xSwT~i|fAn+Uj5dIJV zpwdO*-vIz$c>sWX8vszr008*!=hWU*qjxaeF@RkM96gTjX{5K<@9SFm005`6e?CAx z->4M2koAF{(GAvV7A7t^MV)2a;{d=JZoTVTh=9@M+}^~3o`LAWn0E})_h07x&Z@R6 z{^Wy53VVyvw|BvoSKvXzm8g&DR;~F(Loy{Wws#Xg;zg`_ypOMG5JWExzGRQ&ebT1^ zWDC>*@R^l$2SEB+J2)m#81ICYl+u}fHG#;=-BalofaoA|L!meF$1=yDJ;9E9W}0{DoKU?3#e z{D1e0lyy!Gp#_>TJl8;qJia8gKcY49$}^+^Sroc8_&qz4uvS16?dkS)OUJ7U(7v4xf*aMVOilUc<}sq(r}uE-FDhj; zL7Y}1+HJ3^V%2(Q^v3pQrX^SVG}Y#rLH-3bfhaxm82Cs?pu-pMvKW6gRcE)0hA6Z4 z1J-+`vi3P1AoGj1SLOKH9x9PPj}lU+azHzcR|M`Gle@HLl}fxI4Y!FFJEHG}`Rud!(! zcMI=iQ6mktZZu9NeNB7bJ5=;i`q^i7FHux%MF#$Xw*!@9L2|5(+JcW&2%fhfJO2nP zjJ&2si$i(E)?^IXB1E;yKFsIYl0%<2b$ZG0ISIK7D|u@Xkh39j_?SEspep2GHu}0; zF$Jfq6}l=^E?g8DsU|x)9rREnQV`7Bl56maT-DmT0X}-w_D9m1nVd=6lBwD3XBMK$ zR`iBq8ZHN&GnX*s@KMnzFaP*Ekjo5HLy<$N7laUQZJ;QN0W*u>Pr`#-_?|C!JQ!$d z%-iY3Qd?f;yZxZex$|q~M#czSzMB-vyn6*EG|^JNiEK|awY7NeU#zyuAC zm6XGj{#MLJ@b^OoH=d!HF#2BN<;I)7PQ=7ux7DVvTE1H7k`!*Lo|$}3gnd`QUoS9W z7erUf7!#}feHI|&PhD)w*hp^^Cg9}LT9v*GLk6T!tbO2USz8HytF&cm|P~p!@r|;ftICUeD~je|sM>+D#j1Y~Nv1XR4g(Lm4K!tyoo+ z*}&~aH&fsMYT-%s(2au zK3XVTGI}A$*3gV=@hyj>2~tq6AzW~~nC4I4l0RVh;dYpC`N^&TaM*Y)E4k(3CCCTk zLRJ#y6B}oK?V_o-l+)78=igFONAASZI1y7@kI%oD~*azGzV5%6pth zNurr~s5O+vyBTe5AWt!_uqLp@n7S&4tA?(L`lmw>?wlW>(h;q_+i3C4&0 z)I`f0zcMxhV%9v7A1hw|?AHRB@g1e&a?l(0d>9~PozTGn|HrR--o-wu;^_9HSc{4t z(}3(BzDe&5Vh55ehj`bQnTwu|o%2wgsMr`0{Cs$^7zqDqCR^z_NL?QL@&{MR!N;Ej z2+X83RT5F0YFG^Vv%lj~CF0on4)l_Ih5J`oY2Z-1@ce>zXOIApG~(EaE7S ztDXU8l^(t}RNil>7wXBa0Ty1r8liI8s~6ESMjfgOxAriptP3hlP@H3e_-9R&9sTC? zBgb_iEJe}8xS{=mdFeU>Uzf#COt(L(N8cso{CIdd_}A7Kmv|oSJ6@*LBbnjm9WqDT zbI7EfQUC!|6VT|_kDkCZg0O^y>?8N#wtsDW0+GA^ttvul`B{a4;YaS&-)}S|00=s% z%Ya|rcWDyh!xBD$(9z~ciS+2ZYQ#<=jA^X(sZuO~2tIGe?UcLuf6lch*BB`C=Mqvu ze?Nd{f25q)jQI<&b(ihum-BZ6^H&USMUr#FmO?H2gY1ALRkG_fTq+W~(4jtbrx7`s z;%Ik-sB@hi5(}rA3oAIM;j@aqHJcgvVTW6KIdA$?B=VaNn=$5W*idt=yf7A@^h(+& zw5;SlxS#o0$g-I_w878b4tv~jn&P+pHCqP&yXSVILul`I2{%Lg>seXLZ4M=?#z>VD zIE@d|brtD}4MM)d`}zBUK4CTWDopUkF4Gz}P4IO25nEvytd3{Hmq9!F+3^D-7GI^e5hjW~{`>8% zE=|YK4Wi@K4W`}q((j7)m1+aF;vN+xP<+Td*1kfo&Eg^7I(;}flY6d8uY}H-*NHjr zpn`;pfd@w2((`mz1R`IQ7V_{`fRaj_g}rz_4fyD#EdbsHz2Cz3`X#uRm(AVSu(tJ;&du0YKZ2T_*LxmJ zxB@Hpwyd;76J+5SD-$OaC!b54AV{(5m3ROL-tcW=zZI+oV}8rp@?g-p+N7wefg4|- z;CDq;c(WlTLSUZA}>9Xq5%4J!wKDK|vkjn(2 zLcc5J0@27bL4F>(nHq;D3HwOyy(j9ZDKH6as3G9^LmUTCl5=7~*IJqYlvL-K(4Wmf z@C{QPY}zJefK`?popKUyw6@baQpjaX;y4e6ryps*16>Q#=>n1TG_FH{=(sLUgoU#~ zcvh|=lP|Z047Aq(6RFawN&qN zd2B&wJXpG__@##G9)$_se^YEGGdqcZyk2pA+aFs+GE!lt3v&UWDu;~eyO!~4MH>Yk zg0AVr`vG`Y{>I_J2fm-9&@QZm^6Uy#ic%j|RtAg@?l{4#`!_~!G8qZv#Kq47}lVyX}d{Sdp zWtIHb5+bv^RWg zSdHP^7>8)!p6xG2W;3gHKIJ1bR=d3OB%jn95d6i+?DWy#-jz}SUm!iL#=obs!z&L) zK(hyW(s=1o(N0=sKlO#16Imm>pW07M`}9W_3yCYkRu{A2TODID&IV6~Q0iCG22AGY z`H+@%3XbuncK$?e$mBZQrrA%-Mmz_*F{7DJ$8;3eb)?4zBLeq7_=osUXteXo`Lo{z z{sRBB>MHdUNCpqb(`ll=3p#hK;$Jm3vti1AT17{ybW}ae{7^1fiq0cR{$Jsd{xR1@ zsSA$cYTqOBAQlFm2JTSVk9-r9mu!%fTjMkad--yoW}&Uhj5ym-w(gTfA8VB7LZtB5 z{WKgJ!)h1p8_vuzzmInC2S@X?w;>ZBeck>`Leg5VwAHoDtTt>c`mkh0| zvkyb)BeTK}E>VlH=zTneb;DbDOp4X6Br|O&NG$kaG1!pdTL2Gx+H0a z<@{m7_8Ggmu;iUGtO|eK{^*ErvSbTT&uE>kW*fut1uLbVXT6eFt{SM7iuL{R~Qg`xNEs6JPRYbXQ$W(od z)A?e-dsiI;ZoM(gTqurur9FB@tK8)?-Es(aI4Qx6Rw9@v z5ld}Z6B~b9bKibZv*h%`aTql;0&yt+yHNcr{lyD48$a$4{A3^-R_){CV$2@SPt?g` zt?9~swPMR2L*A$G!5$tgL)`-FPbJ~3mR2IP1GSkrN_mnAq_iClLWMb_nFuMajv`>ZdVD}1-%^dY-s9Ef-JO?|lYz$|e|$HKd^GC* zJh^;%{A-QA{~bX~X$Id5g~}-^I&k+~=oWXSC_X*UkIvMg>wVHyZnux0D=!%s`hI84 z=t8I}?r?9ep2A{&PJ;W*(=EemIyraJg}U6Z+v~(l%9^@1m)7!!jnjh?*+FUUOQ#U& zm(_7Kbt3Q@=-lOK9?5#cxzi?tq#H3syY-U`2NC+r!mW9UudKB?gH;HCJta`Pdo z_;)Q}EA7IM(21w9YME)ogTVJ?@;BDbO@cSsNS9xxQ+Y+MdsGc?OXJ-yc%Q%I3N&9C zyx(%!NM+1{h0e}B#&z_LLE$Cq=pwjR4e2l4tJ)qj|Y~|Ogm58o!y>UL= z)W}eH7P-Uh>X;>ifajKXqjqF}tBq@YMAuq26Pmvm=&gjdU)r!uKW1_+W*POIPRofR z^rq%?8qfY+nkhl0%Ls>nowpHf*D3j7Jrl}k9$<{BOH@7MtyH+%=4H^{d=kliW`%Yb z9NEaWp1U7Ro1lHj5*Fd?4n#(7X@Eg&CGwO zSR3oM?4`kkN7|dWXT1++Ov!KBI74;0X0~)KQ5pX6{&yc!g2OaEBAGCcb!a`NWWnet zO5H4x_kfH|bBWvWnk;6X1Q&84Y&72|1vXL!BPk~bSjmHta>0XfHB1qn6aNpg2W11B zhkGw8wr&skb?7yzl|8YU^%qXo|LprBYS^hb8MVn!o&@@ZndE~Tw;nh;JaAG$c{|Yy zfTFy@6&d-HbAP<^&O7hk_r6(c*6yyVuC85OUDZ{4?@%QLDQt{K7)VG+*wPRQ6(l4S zDEvQ;jsk~BymMB9U+$ZW$%`Q&RYqc78a{ycsf-~i@<>P?^hij4aN=_~$Zs79$(aKQ zY104+i9ZPmiNr3gL0J$9>7Kljoa&34n;SR{8ylOAjZI!&-sa}!+}zy7#f5}~gulQ4 z@bEAxDQV86Yee&Fi}&)5DXQN2+8@T8QwQyXs*F^u1toQ#^U6Mn@3tmqAi^cY)b*BN z=A-QF>^wX?)YR0hsi}#LjU^!=0TSUQRY5q{^&Fp!s*Z)-9wz>MW>I8f z>IYm(Xi9DvR@qoy4XDC;=o(NI>Zbs+WK8>FfPMt3lqIH1C8kF$+Dk7KI$mTlW?UBj z2e^bRzD0DFk%WBm{P{nhMYm&+Gw|h1@??x~CG}D3`?808CVy%BP)h%ab6UAXK?Yo# z_l#tvB#=TLkVD`dfE`5B5ebPj_4a>H+BqN&4n%X3mVbe^h>C{8fZN5aKY)ZJStl(a zrs_7elWvi^BznRSbU#@&nvlSdkjJ4Xf{`+EK$F=vZ$(sMpdc@NuK)U91%^M)uuI5Wy$RQJlSq4V zwE4F#&k2jhv2>*-^|fwF&?iP*2A3GOA4s6c1!4O`e|AU=E@dTQ85%yPGCbiJIZy_p zjME+vLMDW;j=Z2(upm*`FT>(BajPh96rc<4OQA>wgxJ zQ%GILF0bv9r8Q1;rk`2QDF#D5eMqBls&UsQ^%$(z+EP2~vv&08 z!BpAq-ecR=t1nC5cQ)cDBz_pec;P6oSRA_$cuj5{GBu`5)DpIy7YLC_EUQM_yIad^5%>yc{dt+*bpFXWRsJ6A32Xz><`Js}{m zJCfrZLeaXnEd5#YXmy&hDIs`0oPo0G>X*lvt3+q1j7CXIx8iK{4e`VF>=vHP9uZvI z;SnLYRQbdptd5laNO|mTV$b09y_X(5xI@~C8_g%{8`_(=muNaUTz&siR}AHV!4qtV!+f34Rsd1gptUKGf}o6)2_y_w-{f%c+iVuhKv?RPZKOk@s?Im^gS z#qwRv%p%UC(4fY|q|(*CNjb>#P5dLts$7GVTr_(|!tht_LVSh_>Vq;C#Th)xu<)+x zAcwV=Y$9P}-uv;RpMNuc0LfpV2@iooIz@JS@S3A{WSZs6o1-bMsO;xP7%+mL8#bTM z7SvkuzHV*N?|l-p<^S+0(eUdv^eOShkVru(Qo zw^VDHmQkS(zq8SX68nViDkv=8q##ftwe$;bv`A;K=&VrOLE21Gdf&dshQzEE`6@o} zcq{;17djP{9+F(C7@(2D0wU`{Gsf7*{-L}+_Su4&8u(cL88_v3<9d8L?y`)&Sbd}2 z6?NN~rrmF~a}z%AoB<`{Cm!J_kvN;4vdxk6dm|^u9|v_ZI#G)zvPXqku%<5YObOIv zDjADfrbks05oBa2%dzd3kQe!qwW$!!r39C_}x-spX< z=~ICPT>CMu2S!Uum})D;?T8L^2IVtZuyFPFg(05}ftWTa7C^p9Rex zJ<;809zXHcP9K5|F`S>&3`-Ds@OzbsB9iB+&%sZ#@-fLCN(1jmOYhHbQ=to+ zNzzQCvdHe1RIBW!noYB0F3DIRVnexiaStu};pCBA#wBg|t~J z`g<(Yb}}eL-%S_wsyw;|YOD$s| za9sBreS%?cE85}YOv>P?LBT6ORy{0A4t>>+=5SW&L1$PHC76!gH=y>cV-OD^)Yp}w zZNk7L%z|bEj((Cy1sfVn0{wGG&#;2L-5W(@Jw^h2d`>nO0u1?M%Q+|xzof2sS{%y+ zWO)twB&qPTy3gS}zr>2(D)uTL)JT(2{1Jfsx2u}cH}@G8A5 zZGYyK%PWFOCAWG18+EZ)A<4`?#LDX}iCxzDI~d`L|J8wF|DUM?MjIf5lK#a}vMd3} z$}f|85gPfYT_1#0CJHu#pIq+&lJ5Z;(Lg4splyn%sE1J%p8u_gh4Z&wZ4ZKAgyJyC zx(F0I|16Fd5|vw`DBmr>7&?haqO}{!gU$dXrxe}~-<`4HS8C;%J|t-JlX9&xgbZRNx`;Sdl>kk?IVl z#o#FKU;OBp_bR@X;K0C0+YWQLF!^`41QR(Ew;fg|1m=S)9x<{(fbMPQYxQ*dS6Jm1 z%{{laIKDy_5S;AEvbo#Mv55K12z%>|EdWl7am05{bOM^WK9+;y0rb;2zY?M(%fcSa z`x6k!gs^L&wLOZ$L+TaK4lWRD1B)au{u|;J%YzGo76D1To$NquGa;&yg_}8&8;GD%RC; z_Y>fl|O z3HC&ee3O`Fq|dzmH=jURZt3mqptw1mEYMVgIQpu{{hMw*pO5#>mu z<#ilAXVQ~`Zi{{Ws@+b~RsHa%`Bo_0F|p}hM$c4bgPvTCZ#P__Y?gc*s$*C+ITN7I z(s;A#VU40Bstr7T(zBanV+K2F9*P>adxoD{FdTSPj#Dt&uMZYV2+Td<$yvwFMP)Zx>J>@tov2EXH8st{~ehTImPv|6q)t&Q)1Zl71XDwW4!{Q+ebv1p z)~aU>OJhFSP37#kOAr9sW@V8boz5g@t_U`|^tyivhQIQx9E!o+6UWt*wT-VMF! z1t}2}$8qTG6e_66E{syoOO*hRV{?6PX1B+o4zgU+%Vah`8E`)Yy~$J(reAwOm&H#% zG8dyHXV+}PRUXuA)#yY|tflYHoLH5TK&Swxoy z-A4+a&?oEjR^7w&6e>hN9mUzmtBUjB1%a5{2?Q=8@?)3kwH#88t6*%tZDC7)~bi4$)a|xy3X#y`d(%-^T6AdP`tF z4c?gl>MVxj6cHXc++MlmfV57iQZHe+f{*eYEeP85HCvSI7y=SgdP)%#_0)@OMh^`{rAIGEvs4*mpR%HnI`1%9N zWW{|o&hP$K8pzC#)h`D@z8_I^bbgtz#ZaEc?H$N-^RFx_M72W8#u8QDZT=La)w55}sLQ{j~< za))1{;3#F*$2Fm$G~?835tZ?{O)woCD#IsHco3>AaVqL4QE;#*n93iD;|~=@29P5I z{O<$#?*owmASZOteIR5V4}uPg`cD1-;DD%N=DDLhi+*<1egNfGiHm+LR15~BqTVct z4WI`EKM;8WNpffO zZZ%(WgaVZ^{bb1A6*NPKll8_9eaNXFxQB0~Cl=3p`X=Ggb-0Z`bJZW?z0<0<$LRE# zn;;biG-!03RhiMS>{wU#%L-Aqgav7S+{j;#li-CGzqfp&-_Jv_MgFn+sY}Z^ca@ z9T_AcR4<+|%!%LXL-yS+8$^aG5ny-lQTbT#Xt$6dgDXan=BzqRZ7e%ZA@(Ld>hdvO z`DcAk&Q~{)ibZNDPe%MQ?(A%!lo!^*ym3;Ev}Gsi>?N`locW-;X= zRTS`e^g1=ET(T-clRvsv%@Lk`$ABthlvuc;`*HTM$1`+Pr3=ieT2g+fXjv=gJCHbi z{2YZ+pY?GS!bNG8jk@Uiu0Sr=_ez2^ttapNY_$Y`D*iX^Iq&X4^-iP6*(aEz$W>A? z%V+hTlfvEV!OW3r60a@Ky$3=!Mw#cuKgaJ^ipfmnL{GPxfI~0!713k+k{ML4<@R4J zzHSi4q6K|D{?#d@yeh@CtVs?_HVHFB^IXp5og`}umD>N!cx2>m4pRDIN`Y22k>N2m zWVT7xmlu{YWTiD%K(nZi=iOV=LUMR{z|K@AHVvdW z9J}|Og9-R_f zJb3~aPFF=U9_cCyaqN=>!vncji@ghPp7ZIjjt%!Qey|2)j;L!>pUYhRUKl@%J)jKB zJnI@q$k4s-o;gG!w{;;gc5z$U6m<$Fksn@lqVe@(V;l08cH9fP8J0Z)e&{rl{6bK;Y1Yk)}q zV$SuI+@;0!Vl}Ra7rk2R=~>vC#GF?y#pTX{`QQe@!;G=x@vyiW4aF$!vl&DNkUY4?d23P_K&{5ZN#eY?qyEYs0LIU?64Q2BB zqD0223hG*61ozspE<~x+Hk1^h_il&`Q?FMH1IaE8cjKg~t#56V5hkV`%f$JlYV>bR6(O&blG{dcV zB-4tYX$(ish0n6H10psrHb(|CDrd-kdRNUN+aKpYY$1*M{qkHRXuW@1=_d(|3X=rf z7|+MdQ}KF@G3EqY4@}^J}t110o-y;C}oF-!o!S{|ae#+qTAf*UAMZF9X_LUrL)oT9HJ((5E zvOJg(N|`BSd7U_8v_etPVd9_ zqVVPBR%GkxPO!eu)po!6+1%F7ygz~T6>JZtS&{ylW1V9dpUDH#1| zM_s?>0&&3cQ~Z(huVc;s2Tpk4qaZuvqZ<9c3;Rze*Y=q{G~s*yWoPm)4tU})75I^# zv|mhuu#tky^{x%^->R%c10si3k{uSVd01`3GXWzeG<(skKkyWQX6hbpJ#6fVpz=6TA5VJmjmkb|N8HNs&uY=K^ zV-}QII?An88-|sYzYY&b4m2JgtIyk-tdQ(h|MWvlW8%zeB4?V}BRHUHcb3gw!ZL##N>LU-!--4|98?$q;184wfsojm|rGvMiifmk*VH&{8@mKX|h)fnVdY8%q zH}7&;fHUI$f}s2e-HL5TiG%~ZZq+Q2Y_yi6lq?_h#zBASrH}bB3iwu2tLL!P^PI053#-isX58e z=sK>4D!@`7(wIwq5$Pb6)`!?e;{m15=}kItG={8Y8NW?m;`M~B<60!Ka;1rZ?QlP? zAJYkbFtMb6pBG9yZfu2&CmP0t?^}I(& zI6|S;qA)-D)ErLn`WS|??uq*d>tL1Uoq&+`$EV!jF-WJ8rz?xN)6pU<$eS)L5B534 z?C~zD!?Ct3<1w1~Xnz8^n==#@VQVeGh$511af`%#HVRyboKHQpa$hlt^!-wRk1bse zC1UMnJsmkefRt_D_eOSo|Im6QPNn-Em<^(89T=m1{JHJjtTxPzA2KF9$4hhwvkH58 z6%|yH-p+xITHSL5*}5}_%d6XuSM^Vye&1Zqz4!Hb1@j%CJWa$Am{_pV$nRj*(l}+mIgb=L z^VM~!=%1{RO;mR(3cNXIhj&LND`X1Pok9c2T`Q(1E2L}Tdl|FwmqB;6VRS_KIyte+ zYB?5F44(qynJl^Y)4_Gagl zMG5DthXw3sK?a1VSLr}H+VL%aF07RMSg@?t94NX#kih(RkO2ioPh@M4R&?AOmFKEArvCqrQ2JO!|8Szc+T#gUy_NTL3xkfo+w-8T#zYLW;f5c)(121*abORXoF!VKx2T zh4~8U&5m(Hc6qc{Wy2hh5j|3coSx|uD*lJ;phF}y`IHcr<6#McmX5O&@ozMtA#0X@NMhb-R2R0gZ)M*QKfp=$bOyZgOb@btk2{4h(yZic{?f_bev;*Vxr2KpuaN#X z{O1ZZ4v%i7SdxF&-TaE8T z0loN|XUO&!On^|^>sWg6_@3&=v7uk<_+HjKl~8x=fx);OletBL$gLGoJ%Wjt?(gRo zE(rJ{NPr%G?ey5DAZxm3;$Ma*J##b=uzLCz5(nCi$%1_K=>mac$(c`=n z)P30BWq%xfxU^jdp#v<5g^NPtTYiMC-A+WBsAR*NPEjVJTWe*XxnWj+4gI;W;XzyR z4!ZY2feI$0U)Jq-D<=ztv0hL>&zIZFr>Sm=J(Z`wq-)RXs{Uw6K--rHM2Pbvw&Gpj z-Cf5(nPf=9mP#%2$p+TO_p@>P-*rE=N!$nHj=PoxJ2^GgxpB>|0ATM@R)q|t*A=A&$qWAatJNVXQ4 zX?5>_ail4nneUCa5a8zbg#)HLVhx!AyEZPmh0+FkAs*}5=Vm&g(7yIWXF4xU8*0RQ z+OZo>Z{wJ*8h8WxS(g!dBah{9t?-BD5m%AHEzsXay3Rofl9U?{h=ZCEJKVzFDoEG4 zC_ywZ=?p^yvAQikwbuAU`5QnBU)mEzy!68^H3e9gdGxBUO0hwsL)3s;GOIo|%*pU^ zJ6}r_fU}bnjF@c3g1UAu<(I8BRaV3RWa3>wBZ~>Q-qPI=B7nAoU2tI27t0m|YknF- zOPozs6D~c>I|6Kgwv&=2b-~`$I)Z}&WTH5LAM;`OxXj%YU>N>Tq(}|Jf}#j8oHz2{ zzyVK(YD+awM`|UEXP0V*=B7l!0JQKqP~=7_OuvuZ zg!iu|=m51WR)<-$={OE#eK?yV*Wi0#}KDBvg!w<=<|l?|gvgm|HR$zoB=_ zhX24Y{HoRa2Wr@!*4FJ>;q(J%S%|EmyAJN!baCQmE{W}Uq^;AZg_;NST_kt_a{FnF8&sl-8;~Ghc7JT@%bGm;7|CGAX!iO*>0p|)#5dDh8p3uy_7anbduzUqg0G!PEC_s$)|{9eJ+f>hwu8=OHwO_r6a`3*{xJYZ%msK*MbvGmue;v7Vv{kkVs5NuXuZfrFvAi zG~z1Vlt)1wx6rrCX2eTR8@7>NI)uGhl_OS~-f>+%KJ0xG%r)nGb4BSQJMqJf9VgN* z^ z*UC4L#8)p6SyytZcMPKd^~gJaq~u<5$&+RbKnvpwiL;K zg!X_~Ex|5wZ&oF5PiqIUk+y8-TMlY3H(5NzrCXLaf70y#?AyGV)Xp4hnheHCDty2QN?iKZMlX z18%yt0SDH*6JKy-_>` z$4=FW`H|*FX!_T3qiY;`(+H2cXGcdi94T@smafB`gwZh-FvfSfZ+eZX8#(TR@bK^# zaH=DP?k_F32>NE{{!-=5fz@V#VVk$S)nNE5yTdrUih)L$RjaB=zGFh585Bt2Qo9VV zj<*tx!mmvALII**4tPMd`*5)ffnR#ht7WLYPl`pk3c%?Bt(VAv5_qH%R0S}knkc*; zEK=HmE6K~Tk9+0{$R)Se~MsWKBAU4L%8wz zFd$mBte%OK7GLiCCI89-g}%C-o;u5m)}p59_dsyk;)iwQhpuF* zBpLn)96WrgIy1?AW{cE66+2faj|&rnF~Jo@9{rn&H4S4|-#d<*c$3WQjG^JeztgSL zwrTZ3`!S?2hZ#t$7-%6j);hrYahs}tFJ%`IZOb4UoJ7}^2FgIE-w0kS1;1N$GxNUwx}Bao&tVB z81rG2eFfmkC@*7|V*B6h5-5saEE^D;rh#@rOw^Y#6CR$%G&H8ge3rOo5Wa^zM(H_G*&HoFuy_3$BX0NQp|>^K!-#RKJ8`68i$xqF z1J9D__B)O(_MQ$o{7=a2B92!2YZJ(%p<**sdMXnOMpRGXmhfjcf3RWkXDOuZ=BKft5pdKlI_-7q5%3}SY z5~Fu!Ogj3QyiKl5l|cbmh(rM#yST@3Q6|bSqRXN^W8R?2rZ8Bot`M~JZjO~YANJOA z$~VzAoEUN{#U#Yh=Y}|Na}|l~JcxTPqNlTy_$_s1Y|*X7qxl0*DYf(vPoDJ9{?cQ^ zg$vm&%^nK`Ik&|#44z)X$5sbtM~hbVI%@aU3ODW zjVzsxp0vZadHM$f{@ZlnM=D|H>wG{nOvj;_ab8R4aC2XpX_0j7dS%i-R^(^w@5Uon zP9kae0qUlo^nGGHbSdFBhb)%9di`i%nDc|yeG7if0_6Y;=@@3WoAvb{Xek(Us&jAE zmgccy_kK<1B;OE64_dGKo9mUV!pqFlbztotMvsD+^MxftUit=%xl`GZz{wOu1pH5mgMITe zw))<;7qwRR%|*;S<>l%{&H36Pa&ujskK!&w4H(QXqGcN9q{keb@EXnUR}mX;2o zp_N?IH()ZjlBvm8y+~_rKWG#wVyC=3zGoZ0ziAs%RDlOaKm;jFD6W4Wh|fM+8JpvE z@A2JD=_rV}xr*T)1H#R}Hc*g1@k&^&LV z$$&?a-jzR}k3>9idSq%QkIdRp+KNB)VNr)ProlxjZHGNnEbT!?3@;Nu{WVqCI(m7E7TlDh=|Ld7)d$-^5C>8+LULMcwqbk16D#Qs{I_ zwPH)0)oxZc2bvrA39HcE>=d!LV~(kJfo_M|0|&n*YTbn9Sa@> zY+0q=-y4BnHdXDDgsfAflk&YG=u=Qj-yF^nyY8;Aj&A{1-`3dUEcZ5FPMn3SeuJKM z5I8%TH6>Kk@^?R_P$Lh(_)6Y3RhG~qV@eRBjKp^O3^I~&`Px>%Hn-}2_qEzA16HjK+7e^bTwGc`JvVvx zl4w$qO|ApZv{eK>=lfxkgXr)qK|Ii0waCC1&RjjhK1V#-M-uj+)+~2OHN-=Jf?;w< z`8h35MRs$%9nB1L-F>Zr1+Q;eX87)V)kP!+v>n%KBK!@}9GE?^^(e4j= zS>LqQ0D3L$rN6&eqoR zeAQ7~sQLDCJt*rHGQ|d-en7LSgKr@j9=y*Y=s?PP21Uf&kmcbYFl8SYpv+Y;}meQ~;Jm=1X5YagC$ zq`_5W9i$?rjcNwL=F?@nh1S&ELQQG1-MiRPtRkoRO3HN%#`JYuKar}*ze$Fd#h4lS z77tC6($2$}?JVa`vB+iZu91fw=ZL-C?5}1)*KK#4^aj;MGzJJ3jfO8*!2`}rAa`m=N zB$QxxjdUCp01{b$(ay4rXEF*3>e?QHVPJSdFDd_w<8x`z?*6vhmFDT$-r^e(659mz zu%)we$MbI2&o0!4)bNK{IrI-nrPDELZh6G*~vxb+&V8rmU*bSmA_R1Wh zfespNFO)|8qE<&xN60kW8j;e_Fm;seXL-&if3!nIopTKa-z`N{l(H}US;?(VxFJ1-t^Z>G zU#}b@h+DPlr}?R!u`yQAN1VU5GC%c}P|CcP!PDN8O@Fm%{S|zR{87Jl7Sz0drR#`V zcb8$|Nje+*suL=p1o_We63G<)YRaYM%1mb+?RuIxfq*92~LD&vSB|JeW_mh~GL^k`9&5)Cmrl^YZa!gq;1R_Y}Qk z-qmF(8?2Th1e-jSze~~)!vcXB7<&nceKl#OW|Ks$Ri*<~j|)oBMq7<|?1kqNlvqPh zp2z?76E0*)^`06lA)`Jj-QBwpG^hMI$=( zE^KQxVP})QLx%Jt-y~toTpQ%&P0f8)g4x;-4g5davaW?1c{;Y}WXP631C)0Sl1PDt z5G9yY_(v(XC72Z{GkC%6baByXix>O$jpV!KVUGUoLOpnoLaz+eq>J9Wd_DiIrq15Y zyOC}ol)s=(@Y^qqW@~F7lz%dY7XOO@)a5bFW7HI{-mymG<;Lpjt1m0(bE#cjJH0#S z&2zO|uH&LG8F&Vb(9kI(!N2yu@qeRp)UC6#1N;e2Mb2-xJy)!s7qlJq@NoQb+wivx z8me~ZiH0Wr3AjUmM?SY*{)*ftB6LKer|8Z#*9@A7N)!-tHSrO(+Hu$m{bD8kp`={+|eoL-$_=J-^0Q~jZmD0TlLad6A@9gE89G5>+I6* z;E@G#rfxHy%PyGKxIAqkn>bQN{dKV-Okp2tjKYy@T-fIH--1XOfXB%c7+G#18&wfShNtZZIU|poUuat-Q>K zoS#-$7V{pWI`B8Nq#7sDU$6dYmrSCEHq~pCSW{7dA2Q=!ywAY+Nfi7c=bIIjmn8@O zZWlSLnX%ji?I;TTbCTn#A;s783$ExB$Z$ldsW@-aeBpce%Uw>N;^7~lbl%7=lfgd& ziAbn=N+%;()(v}f_qhrDS-XL@6T|ISDkf8N$DCks9J8>*0wim>_5A zIP$!_XP&ag)Tf`t_n>d~Z|=Q`q9A&G?@bJx z6P?ow4JQ*rCsTf72UGY3=^4jUURDlnRt_H3r=0xH`S_o5GIMb7b8zG|zQ+BZ23Xsg aSeUv0uLlgTey_fjRr-a3L>buN?f(F9BHXJ0 literal 0 HcmV?d00001 diff --git a/docs/source/running-a-notary-cluster/resources/node.conf b/docs/source/running-a-notary-cluster/resources/node.conf new file mode 100644 index 0000000000..502f2fc651 --- /dev/null +++ b/docs/source/running-a-notary-cluster/resources/node.conf @@ -0,0 +1,42 @@ +notary { + mysql { + connectionRetries={{ number of Percona nodes }} + dataSource { + autoCommit="false" + jdbcUrl="jdbc:mysql://{{ your cluster IPs }}/{{ DB name, e.g. corda }}?rewriteBatchedStatements=true&useSSL=false&failOverReadOnly=false" + username={{ DB username }} + password={{ DB password }} + } + } + validating=false + serviceLegalName="O=HA Notary, C=GB, L=London" +} + +compatibilityZoneURL = "https://example.com:1300" +devMode = false + +rpcSettings { + address : "localhost:18003" + adminAddress : "localhost:18004" +} +keyStorePassword = "" +trustStorePassword = "" +p2pAddress : "{{ fully qualified domain name, e.g. host.example.com (or localhost in development) }}:{{ P2P port }}" + +rpcUsers=[] +myLegalName : "O=Replica 1, C=GB, L=London" + +// We recommend using Postgres for the node database, or an other supported +// database that you already have set up. Note that the notarised states +// are written to the MySQL database configured in `notary.mysql`. +dataSourceProperties = { + dataSourceClassName = "org.postgresql.ds.PGSimpleDataSource" + dataSource.url = "jdbc:postgresql://[HOST]:[PORT]/postgres" + dataSource.user = [USER] + dataSource.password = [PASSWORD] +} +database = { + transactionIsolationLevel = READ_COMMITTED + schema = [SCHEMA] +} +jarDirs = [PATH_TO_JDBC_DRIVER_DIR] diff --git a/docs/source/running-a-notary-cluster/resources/percona-colocated.png b/docs/source/running-a-notary-cluster/resources/percona-colocated.png new file mode 100644 index 0000000000000000000000000000000000000000..f47ff09971a6d2bdc2dd8250844f1910d80d80fc GIT binary patch literal 4678 zcmeHLX;>228m7fuaH%oTGIMOxt_e7kV7Mh}YKCiRMrvwiX5xlQE?5~l<(SP_TB0@N zt{JH)D54dT(^P^>kRsxi8ZPCCOCWN&_x9)hxIbo|d7k-kJAdBue&6}d^PTU<`<-*H zd%8PosBKpR000`UE{@&+zUA$2M zK=M8S;PN#9K%y{Ro&o?8pa6jA3;=+*3jpZG6t#F+D;)Wr$BsM6<#NUJrP-mY27C6X zs2$PLW*x938=IHy1&4ule6vxhaF?mCwjOQ&!KbFM_-)@>sB7IcG;!4YMm@Df*%P2gU}U}?WVFdS5*S+zOsrEmc}*x3c64<3`}<$H zawRV>Pb?N=Fc^_Y)Y#ZqR#v8`rw0aurBbP*qhn7`Pj<;NQIXcV5I1K>zz<3WE()R= z2lhS78J%=B52mpZ1cU>LrkK+bri?e;Vqd`tK zUa6PKlwB*k9**wbVkno#r@8CwD~+BzKfLu&&C>n&AD2SoMpzW(^z-tpgno93V2Ek)|6&?o%WVD5(Ft5k0oSO69Od_ojFJ{O>dC$3+G<;GZav*xcMvjBQ-dWBEgy58RNOv z^7*B%^+EIo9nfdW9gxqvs3fvJ!iI=p$T)(a0^(s3bHWhRi#~H~)q4?17By+_7HW+= zeTar&mzN6MA3G7|GMUY+9ljq^4;ALzV@M~-gLd8!aWw}0VLFl7W>1~{t(FRgUqOE4 znpO?Xy{MR5n=616<38nl{FvlmIrPYOrIG(3bCz<89Ul-4$JgOVepsu)M_s=Lv#;{g z*T$O+TBc*QO|XN==%%}j%}X%){An&zIx~d~{!8ab(p(Jkfr#jZ@PS^${YNY%yMIoa9q^eddl$Im9kR$8=DJg1I+7<;IbR5?O)#f5_ST{ZL+Y>Aj<~%DiHWR0 z6V=k5G|T|WKYm%;vc=J%v(^Z%HVehR`XUr26eh=v?sYzkgib8sXgW}5j*tv8G5tA6 zD|JD=J|R)BCG2Ye*YkWr9MVtuOC@r2TXI(KgpDlu4N^9No}(3l`*-7Z!;=z2(ZNNJ zOEEqf$Z)ewaElu{634<;j+U%6EOvT!MKEe3i>57Xjv-r(gPRq+CQ0rPd?_np#@})uuiDJK>~N) zm~TBNLt^LmmL4uY2eXw>e8i5RO-h`q1~&%lH;NBraH=ik-hNFJF+H#ESb z!LawBBUC2d`mLq|?P#QBuE%IL_JE44L0sgn-+aTcwM|wOe+l2CYleNyU!fp&XZhq| zK9r0>FT6vQaHzq!b^s=<~BRK-9v6T0KU=?<+TO`nJJY$=uZp*w0S=!G58>Xxf6 zqSwrTbrx7Wo7lb5Ux6cyxz&pY8TX2>Armx@=5~#OUKkl{>O$J7vL=^Mi!Wud16`+o zt0rKcn+<5_GS)8K;}2IGO2(Ahqx98m=(HcCs=N#rQCPwqs5`Op_Ovq3NHxc_R>bbi z;IlodcJWNHp_utX>b%vuU6_jbC-{DGUgKHP?h?@w``g!geG_(&jB{FGTRTsN!H?DK zw_vOfCS1mn4#KGO!>a?sP9v_Tjx8QDRPVYVe^xOuW1o&U505p@m~Y0B^yBEwJHbPf z`OSM@1b6d%YX2cd>EUf>{DN#-ZL7%hHOj^%pIJp&-TukHQ%gDi7&vp};c^6jBC(%5 zmvlNyDaesp&FQutII)~2Vx2Bsm~pbSYd)$4?qw7|$li8MFU~scjZ^LEsRPvPy@An` z*HO1HnH~8-7Bl5uQwv^t0a=u5$>twqUX>ps19|ED7pEIfrlxNNOCC69Ma$Np0Tl;u zCemrrBuyhrrf3NTMrm`;vv$dlL2IDiOX=93ZXOZZvRz7Q_@cAeO9JvuZ`s-;pBT-o z;jbhv-4jZZN$~N+Ug~r{-lHr%tz{~J4%*tXdG$zP)K{7Q%bHn~2Eo_Z7@H9DpO(nQ zk@&v``GYX&EL7wfqgDc@|Nj?<4V&rc8n&Ky;!?HxGrz zc|>A+_4IUojA_Fm_-o5NkWTV`s_Tl&)m(W}3{Vj0V2!YQ{1z2D?1rl{t6^C)THDqt z@3WHfcv15{vXjkDzurtO=eRX9Y$_U7&^Ys{C2`TCwk)1SON4r=s{Y~@xY>lW zN}g*1I7NNL=sw;7;fFFDOEGM3D|I3S4n&2Hm6#C%pUv*Ie$#Ee(dWm~wi}jd z_XkM8i;#s&Bm8;Afi2 zU?t{c882ipLS!>&BePC;;~u_1so}WM7!3B@>_9v{Wu?-tj0D`ch_oX_oMP<};L?x6 zEi1?#i!7(s;U9A6?oVRQG(F;E(`{gvQn_)`%DGc^vW2O>Ux+WBDirMosZF-JaKxH${~g(JR)nL?omC=|q{J^QbKsOYm1 Y=Pv#4fbeWU+P`vib#ixnj68GopNEaxr~m)} literal 0 HcmV?d00001 diff --git a/docs/source/running-a-notary-cluster/resources/wsrep.cnf b/docs/source/running-a-notary-cluster/resources/wsrep.cnf new file mode 100644 index 0000000000..3d7f42fb7e --- /dev/null +++ b/docs/source/running-a-notary-cluster/resources/wsrep.cnf @@ -0,0 +1,48 @@ +[mysqld] +# Path to Galera library +wsrep_provider=/usr/lib/galera3/libgalera_smm.so +wsrep_provider_options="gcache.size=8G" +# TODO set options related to the timeouts for WAN: +# evs.keepalive_period=PT3s +# evs.inactive_check_period=PT10S +# evs.suspect_timeout=PT30S +# evs.install_timeout=PT1M +# evs.send_window=1024 +# evs.user_send_window=512 + +# Cluster connection URL contains IPs of nodes +#If no IP is found, this implies that a new cluster needs to be created, +#in order to do that you need to bootstrap this node +wsrep_cluster_address="gcomm://{{ your_cluster_IPs }}" + +# In order for Galera to work correctly binlog format should be ROW +binlog_format=ROW + +# MyISAM storage engine has only experimental support +default_storage_engine=InnoDB + +# Slave thread to use +wsrep_slave_threads= 8 + +wsrep_log_conflicts + +# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera +innodb_autoinc_lock_mode=2 + +# Node IP address +wsrep_node_address={{ node_address }} + +# Cluster name +wsrep_cluster_name={{ cluster_name }} + +#If wsrep_node_name is not specified, then system hostname will be used +#wsrep_node_name= + +#pxc_strict_mode allowed values: DISABLED,PERMISSIVE,ENFORCING,MASTER +pxc_strict_mode=ENFORCING + +# SST method +wsrep_sst_method=xtrabackup-v2 + +#Authentication for SST method +wsrep_sst_auth={{ sst_user }}:{{ sst_pass }} diff --git a/docs/source/running-a-notary-cluster/toctree.rst b/docs/source/running-a-notary-cluster/toctree.rst new file mode 100644 index 0000000000..667477749f --- /dev/null +++ b/docs/source/running-a-notary-cluster/toctree.rst @@ -0,0 +1,13 @@ +======================== +Running a notary cluster +======================== + +.. toctree:: + :maxdepth: 1 + + introduction + installing-percona + installing-the-notary-service + installing-the-notary-service-bootstrapper + installing-the-notary-service-netman + operating-percona diff --git a/docs/source/tutorials-index.rst b/docs/source/tutorials-index.rst index dab76fb001..8ef20b5c6a 100644 --- a/docs/source/tutorials-index.rst +++ b/docs/source/tutorials-index.rst @@ -27,9 +27,10 @@ World tutorials, and can be read in any order. flow-state-machines flow-testing running-a-notary + running-a-notary-cluster/toctree oracles tutorial-custom-notary tutorial-tear-offs tutorial-attachments event-scheduling - tutorial-observer-nodes \ No newline at end of file + tutorial-observer-nodes