diff --git a/docs/source/index.rst b/docs/source/index.rst index 43470403fb..7a6e227b95 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -88,6 +88,7 @@ application development please continue to refer to `the main project documentat design/threat-model/corda-threat-model.md design/data-model-upgrades/signature-constraints.md design/data-model-upgrades/package-namespace-ownership.md + performance-testing/toc-tree.rst .. conditional-toctree:: :caption: Operations @@ -95,4 +96,6 @@ application development please continue to refer to `the main project documentat :if_tag: htmlmode corda-nodes-index.rst - corda-networks-index.rstcertificate-revocation + corda-networks-index.rst + certificate-revocation + diff --git a/docs/source/performance-testing/installation.rst b/docs/source/performance-testing/installation.rst new file mode 100644 index 0000000000..7c6c72c876 --- /dev/null +++ b/docs/source/performance-testing/installation.rst @@ -0,0 +1,75 @@ +=================================================== +Obtaining and Installing the Performance Test Suite +=================================================== + +As a registered user of Corda Enterprise, you can download the performance test suite at **TODO** add location https://r3-cev.atlassian.net/browse/ENT-2643. + +File Contents +============= + +The performance test suite comes as a zip file containing the following files: + +``jmeter-corda.jar`` + The JAR file that contains the wrapped JMeter code to to drive performance tests. + +``perftestcordapp.jar`` + The performance test CorDapp used in for the built-in samplers and the included sample test plans. This needs to + be deployed to any node of the system under test if these test plans will be used. + +Sample Testplans + A number of test plan JMX files **TODO** describe test plans https://r3-cev.atlassian.net/browse/ENT-2644 + +Sample ``jmeter.properties`` + An example of the ``jmeter.properties`` file used to configure JMeter. If you need a custom configuration, it is + recommended to base it on this file. + +Installation +============ + +Client Installation +------------------- + +Simply create a working directory for JMeter Corda on the client machine and unzip the performance test suite to this +directory. This app requires an Oracle JRE version 1.8 build 172 or later. After unpacking, +you should immediately be able to run it from a shell by typing ``java -jar jmeter-corda.jar``. Please refer to +:doc:`running-jmeter-corda` for details and command line options. + +.. _jmeter_server: + +Installing JMeter server +------------------------ + +The same JAR file used for running the client can be run as a server as well by starting it with the ``-s`` flag as part +of the JMeter arguments. It can be installed by simply copying the JAR file to the server and having an adequate JRE +installed. + +If SSH tunneling is in use, the server will also need access to the same RMI port mappings file used on the client side. +Also, if the client side uses a customised JMeter properties file that differs from the default one baked into the JMeter +JAR file by more than just the +list of remote host names from, this needs to be used on the server side as well. See :doc:`running-jmeter-corda` +for details on the command line options required. A typical command line might look like this:: + + java -jar -XjmeterProperties -XserverRmiMappings -- -s + +If you want to use JMeter Corda remotely, it is suggested to install the JMeter server as a system service on the servers +it is required to run on. Assuming a linux system with a systemd based run control, the service to install would look +something like this:: + + [Unit] + Description=Jmeter Corda + Requires=network.target + + [Service] + Type=simple + User= + WorkingDirectory= + ExecStart=/usr/bin/java -Xmx512m -jar /jmeter-corda.jar -XjmeterProperties -XserverRmiMappings -- -s + Restart=on-failure + + [Install] + WantedBy=multi-user.target + +Sampler Client SDK +================== + +Coming soon. diff --git a/docs/source/performance-testing/introduction.rst b/docs/source/performance-testing/introduction.rst new file mode 100644 index 0000000000..6466d27205 --- /dev/null +++ b/docs/source/performance-testing/introduction.rst @@ -0,0 +1,167 @@ +============ +Introduction +============ + +Corda Performance Test Suite +============================ + +Use the Corda Enterprise performance test suite to stress/soak test a Corda installation, driving either a single +node or a small network of nodes including a notary. +It uses `Apache JMeter `_ to start flows on nodes via RPC calls, and +capture the start/return rates and thus throughput of the system under test. + +.. warning:: + This guide assumes that you have a working Corda test network or + know how to set one up - the set-up and topology of the Corda network as well as the hardware used can have a sweeping + impact on performance, so there is not much point in performance testing before these points have been considered. + +Test Architecture +================= + +A typical test architecture consists of the following components: + +- a Corda network to be tested. This should be a network of Corda nodes along with a notary that is self-contained + (i.e. does not depend on any external services). See e.g. :doc:`../corda-test-networks` for information on + setting up a network. +- a CorDapp that is to be tested and needs to be installed on the cluster +- an app to drive the test - Apache JMeter is used here + +Apache JMeter +------------- + +Apache JMeter runs tests that repeatedly trigger an action, wait for a response and record start/success/failure +timings and so on, and allow to view the result data interactively or rendered as reports in various formats. Run controls +like parallelising tasks, running tasks in a specific order and count and time based repetitions are already built in. + +The interactions with the system under test are done via so called *samplers* (see :doc:`jmeter-samplers`) that can be +triggered by JMeter and then +run an action. JMeter has a number of built-in samplers, mostly around web technology, e.g. for HTTP requests, database +queries, starting scripts and so on. It is also possible to provide custom samplers that can run Java code when invoked. + +For the Corda performance tests, a custom sampler is used that invokes one or more specific flows via remote procedure +calls (RPC), where all the required parameters for the flow and RPC call are passed to the sampler as parameters from +the test definition. + +Interactive Mode +**************** + +By default, JMeter runs in interactive mode, i.e. it brings up a graphical user interface (GUI) that allows the user to +create, view, modify and run a test definition. Tests can either be in process (i.e. the sampler runs in the GUI +process) or can be fanned out to a set of JMeter server instances that will run under the control of a JMeter client +connected to them (see :ref:`Server Mode `) + +Non-Interactive Mode +******************** + +Once a test definition is complete, it can be run in headless mode by providing the test definition and a report target +directory on the command line. + +.. _JMeter-server: + +Server Mode +*********** + +By adding the ``-s`` flag, JMeter can run as a server process that runs samplers controlled by a client connected to it +via Java Remote Method Invocation (RMI). +This allows a single client to e.g. run load from various servers for one test run and collate all the results in the +client. + +jmeter-corda +------------ + +Apache JMeter can be fairly tricky to run in a specific configuration - therefore the Corda Enterprise performance test +suite provides a wrapper around JMeter that comes in a fat JAR with all required dependencies and a default configuration, +and sets up the required directories and config files that JMeter needs to start. It is also bundled with a set of default +Corda performance test samplers. On top of that, it supports opening SSH tunnels to machines running remote JMeter server +instances. + +Performance Test CorDapp +------------------------ + +The performance test suite contains a performance test CorDapp (``perftest-cordapp.jar``) that is roughly modelled on the +finance CorDapp shipped with Corda Enterprise. It contains a number of flows that issue tokens, and pay them to other +parties. There are flows that e.g. issue and pay tokens with or without using coin selection, or create arbitrary +numbers of change output or coin input states to test the behaviour of the system when using various transaction sizes +and shapes. + +Basic Performance Test Set-Up +----------------------------- + +The typical set-up used for performance tests at R3 consists of a small Corda network of 2-4 nodes and a notary to +notarise transactions. These all run inside a datacenter or virtual network in the cloud with open connectivity (or at +least Corda P2P and RPC communication enabled between the nodes). On each of the node machines, an instance of JMeter +is running in server mode. + +The driving app sits outside the network and connects to the JMeter servers through SSH tunnels. In the basic test +measuring the throughput of a node, the test definition instructs all JMeter servers to open RPC connections to one node, +thus saturating the RPC handler and driving the node as hard as possible. The test typically e.g. issues cash on the node +(no interaction with other nodes) or sends cash to a second node which requires sending P2P messages back and forth. + +.. image:: resources/jmeter-network-overview.png + :scale: 75% + +Performance Tests +================= + +There are a number of different parts of the system that can be benchmarked with different performance tests, represented +by different test plans and/or samplers. In general, the closer a performance test is to real world load, the less it is +possible to isolate pinch points in the system under test. Hence a typical performance test run consists a of a number +of these tests that allow seeing where a performance drop off occurs. + +If the reasons for a performance bottleneck cannot be figured out using a set of performance tests, it might be necessary +to attach a remote profile app to one of the nodes and profile a manual performance run using any of the suite of +existing JVM profiling tools available on the market. + +The performance test suite contains test plans, CorDapp and sampler for the following tests: + +Performance of a Single Node +---------------------------- + +These tests stress components in a single node, without any dependencies on other nodes in the flow. + +Empty Flow +********** + +This test starts a flow that does nothing - this gives us a timing for the overhead involved in starting a flow, i.e. RPC +handling, deserialization of the request, starting/winding down a flow and sending the response. Note that a flow that +requires inputs via RPC might have a larger overhead as these might need to be deserialised. + +Issuance +******** + +A node issuing tokens to itself. In addition to the parts used above, this also loads/starts the CorDapp, creates states +in the vault and thus uses persistence to the database. + +Inter-Node Performance +---------------------- + +These are flows that are closer to modelling real world loads to varying degrees. + +Issue and Pay Flow +****************** + +This flow makes the node under test issue some cash to itself and then pays it to a second node. This involves initiating +a transaction with the target node, and then having the transaction notarised by a network notary, thus creating a load that +is similar to what a node will do under real world conditions. This flow has a few variations that can be controlled via +the test definition: + +- Use coin selection - the flow can either just pay the issued cash or use coin selection to select the cash to pay (this + is used to isolate coin selection issues from general transaction performance) +- Anonymous identities - the flow can turn on anonymous identities. This means that a new private/public key pair will be + generated for each transaction, allowing to measure the overhead this introduces. + +To test the throughput a single node can achieve, this flow is run against a single node from all JMeter servers. In order +to measure network throughput, it can also be run against all nodes from their respective JMeter server. + +Advanced Flows +************** + +The issue and pay flow creates a somewhat realistic load but still has a very uniform, artificial usage pattern of resources. +Therefore more advanced test flows/test plans have been developed that allow to issue a large amount of cash once and +then start to break it up in smaller payments, allowing the following settings to be tweaked: + +- Number of states to be transferred in one transaction +- Number of change states created per transaction (i.e. the number of output states of the transaction) +- Number of input states to a new transaction (i.e. pay a larger sums from change shards of the previous transaction). + +Advanced tests also include testing e.g. connecting to the target node via float/firewall. diff --git a/docs/source/performance-testing/jmeter-samplers.rst b/docs/source/performance-testing/jmeter-samplers.rst new file mode 100644 index 0000000000..bd984b2735 --- /dev/null +++ b/docs/source/performance-testing/jmeter-samplers.rst @@ -0,0 +1,157 @@ +=============== +JMeter Samplers +=============== + +JMeter uses samplers to interact with the system under test. It comes with a number of built-in +`samplers `_, mostly +around testing web sites and infrastructure. + +Corda Flow Sampling +=================== + +For performance testing Corda, the `Java Request sampler +`_ is used. This sampler works by calling +a Java class implementing the ``org.apache.jmeter.protocol.java.sampler.JavaSamplerClient`` interface and passing +a set of parameters:: + + public interface JavaSamplerClient { + void setupTest(JavaSamplerContext context); + + SampleResult runTest(JavaSamplerContext context); + + void teardownTest(JavaSamplerContext context); + + Arguments getDefaultParameters(); + } + +When JMeter runs a test using such a sampler, the ``setupTest`` method is called once at the beginning of the test for +each instance of the sampler. The actual test will be calling the``runTest`` method one ore more times and aggregating +the sample results. Finally, the ``teardownTest`` method will be called once per instance. As thread groups in a JMeter +test plan run all of their content in parallel, a new instance of the sampler will be created for each thread in the +group, and ``setupTest`` and ``teardownTest`` get called once per thread. Note that ``teardownTest`` will only be called +once all thread groups have run and the test plan is terminating. + +Provided Sampler Clients +======================== + +JMeter Corda provides a number of sampler client implementations that can be used with Java Request sampler. They all +share some common base infrastructure that allows them to invoke flows on a Corda node via RPC. All of these samplers +are built against a special performance test CorDapp called ``perftestcordapp``. On each call to run the sampler, one +RPC flow to the respective flow used by this sampler is made, and the run function will block until the flow result is +returned. + +``EmptyFlowSampler`` + This sampler client has the class name ``com.r3.corda.jmeter.EmptyFlowSampler`` and starts the flow + ``com.r3.corda.enterprise.perftestcordapp.flows.EmptyFlow``. As the name suggests, the ``call()`` method of this flow + is empty, it does not run any logic of its own. Invoking this flow goes through the whole overhead of invoking a flow + via RPC without adding any additional flow work, and can therefore be used to measure the pure overhead of invoking + a flow in a given set-up. + + .. image:: resources/empty-flow-sampler.png + :scale: 85% + + This sampler client requires the minimal set of properties to be required that it shares with all Corda sampler + clients documented here: + + label + A label for reporting results on this sampler - if in doubt what to put here ``${__samplername} will fill in the + sampler client class name. + + host + The host name on which the Corda node is running. The sampler client will connect to this host via RPC. This name needs + to be resolvable from where the sampler client is running (i.e. if using remote JMeter calls, this means from the + server, not the client machine). + + port + The RPC port of the Corda node. + + username + The RPC user name of the Corda node. + + password + The RPC password of the Corda node. + +``CashIssueSampler`` + This sampler client has the class name ``com.r3.corda.jmeter.CashIssueSampler`` and starts the flow + ``com.r3.corda.enterprise.perftestcordapp.flows.CashIssueFlow``. This flow will self-issue 1.1 billions + of cash tokens on the node it is running on and store it in the vault. + + .. image:: resources/cash-issue-sampler.png + :scale: 85% + + In addition to the common properties described above under ``EmptyFlowSampler``, this sampler client also requires: + + notaryName + The X500 name of the notary that will be acceptable to transfer cash tokens issued via this sampler. Issuing tokens + does not need to be notarised, and therefore invoking this sampler does not create traffic to the notary. However, + the notary is stored as part of the cash state and must be valid to do anything else with the cash state, therefore + this sampler will check the notary indentity against the network parameters of the node. + +``CashIssueAndPaySampler`` + This sampler client has the classname ``com.r3.corda.jmeter.CashIssueAndPaySampler`` and can start either the + the flow ``com.r3.corda.enterprise.perftestcordapp.flows.CashIssueAndPaymentFlow`` or + ``com.r3.corda.enterprise.perftestcordapp.flows.CashIssueAndpaymentNoSelection``, depending on its parameters. + Either way it issues 2 million dollars in tokens and then transfers the sum to a configurable other node, thus + invoking the full vault access, peer-to-peer communication and notarisation cycle. + + .. image:: resources/cash-issue-and-pay-sampler.png + :scale: 85% + + In addition to the parameters required for the ``CashIssueSampler``, this also requires: + + otherPartyName + The X500 name of the recipient node of the payment. + + useCoinSelection + Whether to use coin selection to select the tokens for paying or use the cash reference returned by the issuance + call. The value of this flag switches between the two different flows mentioned above. Coin selection adds a set + of additional problems to the processing, so it is of interest to measure its impact. + + anonymousIdentities + Switches the creation of anonymised per-transactions keys on and off. + +``CashPaySampler`` + A sampler that issues cash once per run in its ``setupTest`` method, and then generates a transaction to pay 1 dollar "numberOfStatesPerTx" times + to a specified party per sample, thus invoking the notary and the payee via P2P. + This allows us to test performance with different numbers of states per transaction, and to eliminate issuance from + each sample (unlike CashIssueAndPaySampler). + The classname of this sampler client is ``com.r3.corda.jmeter.CashPaySampler``. + + .. image:: resources/cash-pay-sampler.png + :scale: 85% + + In addition to the base requirements as in the ``CashIssueSampler``, this sampler client requires the following + parameters: + + otherPartyName + The Corda X500 name of the party receiving the payments + + numberOfStatesPerTx + The number of $1 payments that are batched up and transferred to the recipient in one transaction, thus allowing + to observe the impact of transaction size on peer to peer throughput and notarisation. + + anonymousIdentities + Switches the creation of anonymised per-transactions keys on and off. + +Custom Sampler Clients +====================== + +The sampler clients provided with JMeter Corda all target the performance test CorDapp developed along with the +performance test tool kit. In order to drive performance tests using different CorApps, custom samplers need to be +used that can drive the respective CorDapp via RPC. + +Loading a Custom Sampler Client +------------------------------- + +The JAR file for the custom sampler needs to be added to the search path of JMeter in order for the Java sampler to +be able to load it. The ``-XadditionalSearchPaths`` flag can be used to do this. + +If the custom sampler uses flows or states from another CorDapp that is not packaged with the +JMeter Corda package, this needs to be on the classpath for the JMeter instance running the sampler as the RPC interface +requires instantiating classes to serialize them. The JMeter property ``user.classpath`` can be used to set up additional +class paths to search for required classes. + +Writing a Custom Sampler Client +------------------------------- + +Coming soon diff --git a/docs/source/performance-testing/jmeter-testplans.rst b/docs/source/performance-testing/jmeter-testplans.rst new file mode 100644 index 0000000000..ed586511f1 --- /dev/null +++ b/docs/source/performance-testing/jmeter-testplans.rst @@ -0,0 +1,111 @@ +==================================== +Understanding and Creating Testplans +==================================== + +JMeter offers a very flexible and powerful tool kit to build a large variety of testplans. This document can only give +a brief overview over what testplans are typically used for Corda performance testing. To get an overview of what is +available in JMeter, and if you are looking into writing more elaborate testplans, please refer to the `JMeter user +manual `_. + +Structure of a Simple Testplan +============================== + +The testplan is a hierarchy of configuration elements. The only elements used in our simple test plans are +*Variables*, *Thread Groups*, *Samplers* and *Listeners*. Rather than creating a new testplan from scratch, +it might a good idea to take a copy of one of the provided example test plans and modify that. + +.. image:: resources/jmeter-testplan.png + +User Defined Variables + A variables element defines key/value pairs that can be used in all following following elements instead of string + literals. They can be referenced by using a ``$`` sign and curly braces, e.g. ``${varName}``. + + .. image:: resources/variables.png + +Thread Group + A thread group collects a set of actions that form one step in the test plan. All elements within a thread group + will be run in order. As the name suggest, a thread group can spin up threads to run serveral instances of its + content in parallel. Note that everything inside the group will be run in every thread (including any initialisation + or tear down steps). However, the tear down steps will only be run when all thread groups have finished at the end + of the test plan - not at the end of the thread group's execution. It cannot be used to release resources that + the next thread group is supposed to reuse. + + The thread group also allows to repeat the action that has been configured. Repitition can either be count based, or + scheduler based (e.g. for 5 minutes). Counter based repition will loop the given number of times for each thread, + so 10 thread with 10 repetitions means 100 runs of group content. + + If a test plan contains a list of thread groups, they will be run sequentially. If the runtime of the thread group + is controlled via a scheduler, a start-up delay is configurable that the process will wait between finishing the + previous thread group and starting the next one, e.g. to let the system under test finish any lingering requests, + flush queues and return to idle state. + +Sampler + Usually, there will be a sampler inside the thread group that creates the load for the Corda network via RPC + invocations. Note that the thread group will call ``setupTest`` only before the first iteration, and ``teardownTest`` + only after all tests in all thread groups have finished, but will call ``runTest`` for every loop iteration + configured in the thread group. + However, all task are run for each thread in parallel. + +Listeners + Listeners collect the output of samplers and handle it by either displaying it, storing it to file or + aggregating it in some way. Note that starting and stopping does not clear data from the listeners, and some + statistics are calculated across runs and breaks, so clearing the relevant data before starting a run is vital when + running from the GUI. The description of all built-in listeners can be found in the `documentation + `_ - here is a short list of listeners + typically used for Corda runs. + + View Results in Table + This outpug handler is useful for debugging the test system - every result will be displayed as a line in a table, + including a status and stats about sent and received bytes. The table can become very long and unwieldy quickly + when running with many threads and/or loops. + + Graph Results + This aggregator plots various stats about processing time and throughput over time. It can be handy to view + trends, but as the time window is fixed and it loops back to the beginning and plots over existing graphs, + the usefulness for long running tests is limited. + + Aggregate Report + This aggregator collects processing time stats and throughput on a per thread group basis as a table, and keeps + a running total for the run. + + Aggregate Graph + Like the Aggregate Report, but also offers to tools to plot the results. + +Included Testplans +================== + +Two testplans are included with the performance test suite, to be found in the ``Sample Testplans`` directory in the zip +file. Note that both testplans are examples how they can look and are not runnable out of the box - you need to supply +values for host names, party/notary names, RPC credentials and so on for your Corda installation. + +``Example Flow Request.jmx`` + This is a very simple testplan that has one thread group that just runs cash self issue. Everything is configured + directly on the sampler's page - just enter the details of your installed Corda node here and you should be good + to go. The structure of this test plan is a good place to start to create your own test plans. + +``NightlyBenchmarkSample.jmx`` + This is a copy of the test plan that is used for performance testing Corda Enterprise at R3. This plan has a lot of + different tests, each in its own thread group. All the thread groups will be run one after another. These different + tests show use of all the different sampler clients described in :doc:`jmeter-samplers`. + This testplan uses variables to avoid repetition of values that might require changing. You need to enter the + appropriate values for your Corda network installation here to get the plan working. + + A useful way to try out bits of the plan is to load it in an interactive JMeter session and disable most of the + thread groups in order to only run a few tests. (Thread groups can be enabled/disabled in the right-click context + menu). + +Creating Testplans +================== + +The JMeter GUI can be used to create and try out new testplans. The easiest way of doing this to take a copy of an +existing testplan and modify this as the structure of all testplans is fairly similar. + +- Optionally, there can be user variables and set-up steps defined at the top of the test plan +- Each plan needs one or more thread groups that contain the actual tests to run, i.e. samplers that exercise the system + under test. Each thread group can just contain a sampler, or furhter logical elements like repetition/decision elements + that run different samplers in various combinations. +- Thread groups can contains listeners that only process results from this thread group +- The testplan should have at least one global listener that captures results from all thread groups. + +Detailed description of the parts of a test plan can be found in the `JMeter documentation +`_. diff --git a/docs/source/performance-testing/r3-performance-runs.rst b/docs/source/performance-testing/r3-performance-runs.rst new file mode 100644 index 0000000000..98781cade5 --- /dev/null +++ b/docs/source/performance-testing/r3-performance-runs.rst @@ -0,0 +1,155 @@ +=========================================== +Reproducing the R3 performance test numbers +=========================================== + +The performance test suite contains all the code and configuration to reproduce the performance tests run at R3 and +presented or published occasionally. + +Test Network +============ + +The performance test runs in a self contained test network consisting of + +- 4 node machines, each running one Corda node +- 1 simple notary running on a separate machine +- separate database servers for all the nodes +- optionally a doorman/network map server - the test network can be set up using the bootstrapper or the network map + approach - this does not impact performance however. + +The performance network sits behind a firewall that allows incoming connections only via SSH, however the machines +within the network can connect to each other on a range of ports (Corda Remote Procedure Call (RPC), Corda peer to +peer connection (P2P) and JDBC connections to various database servers are all allowed). + +.. image:: resources/performance-cluster.png + :scale: 75% + + +A Corda JMeter server instance is running on all 4 nodes and on the notary server. The JMeter client connects to the +JMeter server instances via SSH tunnels. As the JMeter servers run within the firewalled network, they are free to +open RPC connections to any of the Corda nodes - which one they connect to is configured in the JMeter test plan. +They usually connect all to one node to drive the test flows as hard as possible, but might connect to different +nodes for different tests. + +The location of the node's working directory is important - as nodes use Artemis for peer to peer communication, it is +crucial that the Artemis spool directory is on a fast storage solution, e.g. a local SSD. It should not be on a network +mount or SAN drive. + +There is a separate database for each node - for historical reasons, the notary and node 2 share a database in the, +set-up, therefore node 2 gets the lightest load in order to not cause a slow-down by overloading the database. It +is entirely possible to run the whole network with one database that has different users/schemas for each node and the +notary, however, this set-up might not reach full performance as database throughput becomes a bottleneck. + +The default set-up uses Microsoft SQL Server instances as databases, but some testing has been done with Postgres +and Oracle 12 databases. + + +Machines in the Test Network +---------------------------- + +Microsoft Azure ++++++++++++++++ + +The initial performance network was set up on a cluster of Microsoft Azure nodes. The following specs have been chosen +for the network setup + +========= ================== =========================================== ========================================================= + Machine Type Description Notes +========= ================== =========================================== ========================================================= + node 1 Standard D32s 32 cores, 128GB memory, Ubuntu 16.04 main target node for flow invocation + node 2 Standard F8s 8 cores, 16GB memory, Ubuntu 16.04 mostly used for JMeter server, Corda node lightly used + node 3 Standard D32s 32 cores, 128GB memory, Ubuntu 16.04 target node configured with float/Corda firewall + node 4 Standard D16s 16 cores, 64GB memory, Ubuntu 16.04 recipient node for payment flows + notary Standard F8s 8 cores, 16GB memory, Ubuntu 16.04 notary node + sql 1 Standard DS12 v2 4 cores, 28GB memory, Windows Server 2016 SQL Server for node 1 + sql 2 Standard DS12 v2 4 cores, 28GB memory, Windows Server 2016 SQL Server for node 2 and notary + sql 3 Standard DS12 v2 4 cores, 28GB memory, Windows Server 2016 SQL Server for node 3 + sql 4 Standard DS12 v2 4 cores, 28GB memory, Windows Server 2016 SQL Server for node 4 +========= ================== =========================================== ========================================================= + +This set-up has been used to performance tune Corda until Autumn 2018. One of the lessons learned was that performance +testing in the cloud has a few drawbacks: + +- The performance of virtual machines in the cloud is not always consistent. How fast they go exactly depends on what + else is going on, so fluctuations in measured performance could not always be attributed to changes to Corda. +- Cloud databases are optimised for web apps etc. Therefore, they are mostly read-optimised. Corda, however, writes + a lot of checkpoints and transactions to the database which required us to install custom database instances. +- The virtual hard drives in the cloud have very limited througput, which causes bottlenecks for Artemis queues, + log file writing and database writing. Getting more performant storage in the cloud is either very expensive or + plainly not possible. + +Dedicated Hardware +++++++++++++++++++ + +Due to the observed drawbacks with performance testing in the cloud, the performance testing has been moved to +dedicated hardware in a rented data center. To build up the performance test network, the following machine types are +acquired. They are all placed in the same data center, with 10Gb/s network connectivity. + +Node Servers + Intel 2x Xeon E5-2687Wv4, 24 cores hyperthreaded, 256GB DDR4 ECC 2133 MHz, 2x 400GB NVME + +Database Servers + Intel Xeon E5-2687Wv4 - 12 cores hyperthreaded, 256GB DDR4 ECC 2133 MHz, 2 x 1GB NVME + +Notary Server + Intel Xeon E5-2687Wv4 - 12 cores hyperthreaded, 256GB DDR4 ECC 2133 MHz, 2 x 1GB NVME + +Node Configuration +------------------ + +For performance runs, the nodes run with pretty standard configuration. However, there are a few tweaks: + +- The JVM gets set up with 8 gigabyte heapsize and using the G1 garbage collector. +- The database transaction isolation level is set to ``READ_COMMITTED``. +- In the tuning part of the Enterprise configuration, a generous number of flow and RPC threads are configured. +- Dev mode is turned off for all nodes including the notary. +:: + + custom = { + jvmArgs = [ "-Xmx8g", "-Xms8g", "-XX:+UseG1GC" ] + } + database { + transactionIsolationLevel = "READ_COMMITTED" + } + + enterpriseConfiguration = { + tuning = { + rpcThreadPoolSize = 32 + flowThreadPoolSize = 128 + } + } + devMode = false + +Depending on the type of machines used, it is recommended that the ``rpcThreadPoolSize`` does not exceed the number of +virtual cores (i.e 2 x number of cores when hyperthreading is enabled), and the ``flowThreadPoolSize`` should be around +4 times the number of virtual cores. Depending on other factors like disk I/O, networking, memory etc. tweaking these +numbers can yield better performance results. + +Notary Configuration +-------------------- +When using a simple, single node notary, it has the following configuration in addition to the above:: + + notary { + className="net.corda.notary.jpa.JPANotaryService" + extraConfig { + } + validating=false + } + + +Database Configuration +---------------------- + +Each node has its own, dedicated database server running Microsoft SQL Server 2016 Standard. +Note that the disk write latency and throughput on the database machine are critical for the node's performance as the +Corda checkpointing mechanism means that many checkpoints are written to the database, and then deleted again when the +flow finishes. Checkpoints will only be read when a flow is resumed or restarted, which might not happen a lot when +running a large number of short-lived flows. Therefore, a read-optimised database server (as e.g. the hosted database +instances offered by the big cloud providers) is not ideal for a busy Corda node and will limit throughput. +It is recommended to manage the database server manually and optimise for write throughput at least as much as for read. + +Test Plan +========= + +For the test, JMeter server instances on 4 node machines and the simple notary machine all connect to one node via RPC +to drive the tests. A external JMeter client uses the ``NightlyBenchmark.jmx`` test plan to run the tests. + diff --git a/docs/source/performance-testing/resources/batch-notarise-sampler.png b/docs/source/performance-testing/resources/batch-notarise-sampler.png new file mode 100644 index 0000000000..90fd3074fd Binary files /dev/null and b/docs/source/performance-testing/resources/batch-notarise-sampler.png differ diff --git a/docs/source/performance-testing/resources/cash-issue-and-pay-sampler.png b/docs/source/performance-testing/resources/cash-issue-and-pay-sampler.png new file mode 100644 index 0000000000..e046a6bb43 Binary files /dev/null and b/docs/source/performance-testing/resources/cash-issue-and-pay-sampler.png differ diff --git a/docs/source/performance-testing/resources/cash-issue-sampler.png b/docs/source/performance-testing/resources/cash-issue-sampler.png new file mode 100644 index 0000000000..cd129edb59 Binary files /dev/null and b/docs/source/performance-testing/resources/cash-issue-sampler.png differ diff --git a/docs/source/performance-testing/resources/cash-pay-sampler.png b/docs/source/performance-testing/resources/cash-pay-sampler.png new file mode 100644 index 0000000000..be3e803c6f Binary files /dev/null and b/docs/source/performance-testing/resources/cash-pay-sampler.png differ diff --git a/docs/source/performance-testing/resources/empty-flow-sampler.png b/docs/source/performance-testing/resources/empty-flow-sampler.png new file mode 100644 index 0000000000..e5cde6151f Binary files /dev/null and b/docs/source/performance-testing/resources/empty-flow-sampler.png differ diff --git a/docs/source/performance-testing/resources/jmeter-buttons.png b/docs/source/performance-testing/resources/jmeter-buttons.png new file mode 100644 index 0000000000..2414fd9430 Binary files /dev/null and b/docs/source/performance-testing/resources/jmeter-buttons.png differ diff --git a/docs/source/performance-testing/resources/jmeter-network-overview.png b/docs/source/performance-testing/resources/jmeter-network-overview.png new file mode 100644 index 0000000000..3f306eb933 Binary files /dev/null and b/docs/source/performance-testing/resources/jmeter-network-overview.png differ diff --git a/docs/source/performance-testing/resources/jmeter-testplan.png b/docs/source/performance-testing/resources/jmeter-testplan.png new file mode 100644 index 0000000000..ece41935fe Binary files /dev/null and b/docs/source/performance-testing/resources/jmeter-testplan.png differ diff --git a/docs/source/performance-testing/resources/performance-cluster.png b/docs/source/performance-testing/resources/performance-cluster.png new file mode 100644 index 0000000000..5e6983d988 Binary files /dev/null and b/docs/source/performance-testing/resources/performance-cluster.png differ diff --git a/docs/source/performance-testing/resources/variables.png b/docs/source/performance-testing/resources/variables.png new file mode 100644 index 0000000000..09f0991e4c Binary files /dev/null and b/docs/source/performance-testing/resources/variables.png differ diff --git a/docs/source/performance-testing/running-jmeter-corda.rst b/docs/source/performance-testing/running-jmeter-corda.rst new file mode 100644 index 0000000000..f23fbdd10c --- /dev/null +++ b/docs/source/performance-testing/running-jmeter-corda.rst @@ -0,0 +1,210 @@ +==================== +Running JMeter Corda +==================== + +Jmeter Corda is distributed as a runnable "fat" JAR containing all the dependenices required to run the application. +It comes prepacked with Corda :doc:`jmeter-samplers` and a wrapper that sets up a basic configuration and allows +configuration of SSH tunnels. The arguments for the ``jmeter-corda`` command line fall into two categories: there are +a number of arguments that are consumed by the custom wrapper, followed by a double dash ``--``. Anything after this +will be passed on to the JMeter code as JMeter arguments (though the wrapper can add more arguments to this). The typical +call will be:: + + java -jar jmeter-corda-jar -- + +JMeter Corda Wrapper Arguments +============================== + +The JMeter Corda wrappper understands the following optional arguments: + +``-?``, ``-h``, ``--help`` + This will print a help text for the JMeter Corda wrapper arguments and also trigger printing of the JMeter help + text + +``-Xssh`` + This tells the wrapper to set up SSH tunnels. It takes a list of host names as argument. See :ref:`ssh_tunnel` for + details. + +``-XsshUser`` + User name to use for remote servers if creating SSH tunnels. This defaults to the current user name. + +``-XadditionalSearchPaths`` + A semicolon separated list of directories containing class files or JAR files that JMeter should consider for finding + samplers or plug-ins. The wrapper sets up a default search path that contains the samplers included in the JAR and + the required plug-ins to talk to Corda. Any custom samplers to be used in tests need to be included here. + +``-XjmeterProperties`` + Path to a JMeter properties file. JMeter always needs configuration via a properties file - there is a ``jmeter.properties`` + file included in the jmeter-corda.jar that sets sensible defaults. The wrapper will make JMeter use the pre-packed + properties if this option is not set. + +``-XserverRmiMappings`` + Port mappings for the remote method invocation tunnels when setting up SSH tunnels. See :ref:`ssh_tunnel` for details. + +These arguments have to come first on the command line. They can be followed by a double dash and JMeter arguments +that will be fed through to JMeter. The wrapper will add some arguments to the list passed on to JMeter and does disallow +some arguments. Some JMeter arguments used/discussed on this page are: + +``-p`` + This is the way of telling JMeter what properties file to use. This will **always** be set by the wrapper code and + must **not** be set on the command line. Use the ``-XjmeterProperties`` argument to the wrapper instead. + +``-s`` + This makes JMeter run in server mode, i.e. it will run headless and wait for instructions from a client via remote + method invocation. See also :ref:`jmeter_server`. + +Running the JMeter GUI +====================== + +By default, JMeter Corda will start up as client in GUI mode. The GUI allows to view/create test plans, and run tests +interactively either locally or remotely, and view results. The UI has a set of handy butons at the top that allow to +quickly create or load a testplan, control local or remote test runs, and clear any test results collected so far. +The latter is particularly important when looking at averages, as old data might skew the results being looked at. + +.. image:: resources/jmeter-buttons.png + :scale: 75% + +The *clear current view* button only clears the data in the currently viewed output collector - if a test plan has several +output listeners defined (as in the example above, we have *Aggregate Graph*, *Graph Results* and *View Results in Table*), +any collector not currently selected is not affected. *Clear all data* will clear the results from all collectors. + +See :doc:`jmeter-testplans` for details on the testplan, and how to create one. + +Running JMeter headless +======================= + +JMeter can be run in headless mode, i.e. without starting up a GUI, by giving it a test definition on the command line. +This allows to run performance tests excluding any influences from GUI rendering or interaction issues, and allows +to run performance tests in an automated manner, e.g. as part of a nightly build. This is controlled by JMeter arguments +passed through to the JMeter instances. The relevant arguments are: + +``-n`` + This will run a JMeter client in headless mode. This requires to also pass a test definition via ``-t`` and an output + file via ``-l``. + +``-t `` + Provide a testplan xml file to load and run. + +``-l `` + Path to an output jtl file to be written - this is a JMeter specific CSV format. + +``-r`` + Run via RMI against the remote servers configured in the JMeter properties file. Without this option, JMeter + will run the testplan locally. + +Example:: + + java -jar jmeter-corda.jar -Xssh node1.mydomain.com node2.mydomain.com -- -n -t /home//tesplan.jmx -l /home//results.jtl -r + +See `JMeter documentation `_ for details. + +Using remote JMeter +=================== + +JMeter can connect to JMeter server instances using Java Remote Invocation (RMI). The hosts and ports a JMeter client +instance can connect to are listed in the properties file it is using. When running the GUI, the remote hosts will be +used when using the start/stop remotely functions, while the simple start/stop will still work locally. When running +headless, the remote hosts will be used when the ``-r`` option is given. + +If no SSH tunneling is required, the ``-R`` option can be used to pass in a different list of remote hosts to the client. +If the SSH tunneling set-up from the wrapper is used, this option is **not** supported and **will fail**. + +.. _ssh_tunnel: + +Ssh Tunnel Set-Up +----------------- + +The RMI traffic requires non-standard ports to be open, therefore the standard practice is to +route it via SSH tunnels. Therefore, the remote hosts in the default configuration are all specified as ``127.0.0.1``, +i.e. localhost, with different port numbers, for which SSH tunnels will be opened to the hosts specified on the command +line using the ``-Xssh`` flag. +Due to the Java RMI architecture, three tunnels need to be opened for each remote JMeter instace - one for the connection +to the RMI server, and an outbound and inbound tunnel. For this to work, it is crucial that the JMeter client and the +server instances agree on the ports they use, and the hostnames used on the command line and the server RMI mapping +do match exactly. + +The Corda JMeter executable contains code to set up the tunnels and orchestrate the connections on the client and server +side. All the tunnels are set up from the client side. The information required to set up the tunnels comes from three +places: + +- The number of hosts to create tunnels for and their client side RMI server ports are read from the ``jmeter.properties`` + file that is used to run this instance of JMeter. For using the SSH tunnel, localhost addresses are listed here as the + actual remoting is done via SSH. By default, the five remote addresses shown below are listed in the properties file + included in the JAR. This configuration is only relevant on the client side:: + + #--------------------------------------------------------------------------- + # Remote hosts and RMI configuration + #--------------------------------------------------------------------------- + + # Remote Hosts - comma delimited + remote_hosts=127.0.0.1:20100,127.0.0.1:20101,127.0.0.1:20102,127.0.0.1:20103,127.0.0.1:20104 + +- The actual host names are passed to the client process on the command line with the ``-Xssh`` command line flag. This + takes a list of space separated, fully qualified host names. They must be actual hostnames (not just IP addresses), and + for the look-up to work, must match between the `hostname` entry on the server and the DNS name - i.e. the result of + calling `hostname` on the server should be the same as the first part of the hostname used on the command line up to + the first dot:: + + java -jar jmeter-corda.jar -Xssh node1.mydomain.com node2.mydomain.com + +- The server side RMI server port can be configured in the ``jmeter.properties`` file as property ``server.rmi.port``, + and defaults to ``1099``. It is crucial that the same value is used on the server and client side. +- The client local RMI port (server to client callback) is also configured in the ``jmeter.properties`` file as property + ``client.rmi.localport``. Again it is crucial that this value is the same on server and client side. +- The server local RMI port (used to send requests to the server) needs to be different for each host so different + information can be send to different tunnels for different servers from the client. Therefore, the property + ``server.rmi.localport`` in the ``jmeter.properties`` file is only used as a fallback - it would only work for a single + JMeter server instance being used at a time. Instead, the mappings of port to host name come from an additinal config + file ``server-rmi.config`` that lists a mapping of unqualified hostname to port:: + + # lines starting in # are comments and ignored + node1:10101 + node2:10102 + + These values must be the same on client and server side and must match the configured hostname of the target host. + The configuration file can be passed to the process using the ``-XserverRmiMappings`` command line flag. + +Connecting to remote JMeter servers +----------------------------------- + +The default JMeter properties file in the Corda JMeter JAR assumes that 5 instances of JMeter server are running on hosts +that can be reached via SSH tunnels. When using the run remotely function in the GUI, JMeter will try to connect to the +hosts listed in its properties file. + +If you need to specify a different list of remote hosts, e.g. for a different number of hosts, to specify different port +numbers or to specify explicit namesin case SSH tunnelling is not required, you will have to take a copy of the +``jmeter.properties`` file, modify it accordingly and specify the file to use on the command line using the +``-XjmeterProperties`` argument. + +Running with SSH tunnels +++++++++++++++++++++++++ + +In order to run the client with SSH tunnels, a typical command line would be:: + + java -jar jmeter-corda.jar -Xssh [ ...] [-XsshUser ] [-XjmeterProperties ] -XserverRmiMappings [-- ] + +The SSH tunnel code assumes private/public key authentication, interactive authentication is not supported. In order to +establish the SSH tunnels, an SSH agent must be running that has exactly one private key loaded that can be +used to establish connections to all server hosts. When starting from a POSIX system (Unix/linux/Mac OS), the tool expects +a POSIX SSHagent to be started, and the environment variable ``SSH_AUTH_SOCK`` to be set. On a Windows system, the key +is expected to be served up by the Putty SSH agent `Pageant `_. + +Running without SSH tunnels ++++++++++++++++++++++++++++ + +If SSH tunneling is not required, the list of actual host names needs to be in the JMeter properties file as property +``remote_hosts``, or can be passed in with the JMeter argument ``-R``: + +``jmeter.properties`` + ``remote_hosts=node1.mydomain.com,node2.mydomain.com`` + +Command line + ``java -jar jmeter-corda.jar -- -Rnode1.mydomain.com,node2.mydomain.com`` + +None of the Corda provided SSH code options need to be configured, so the server command line should just include the +``-s`` option, and the client command line would be:: + + java -jar jmeter-corda.jar -XjmeterProperties [-- ] + +It is recommended to consult the JMeter manual on `Remote Testing `_ +when setting up remote testing without SSH tunneling. + diff --git a/docs/source/performance-testing/toc-tree.rst b/docs/source/performance-testing/toc-tree.rst new file mode 100644 index 0000000000..75b25efa97 --- /dev/null +++ b/docs/source/performance-testing/toc-tree.rst @@ -0,0 +1,15 @@ +==================================== +Corda Enterprise Performance Testing +==================================== + +This section contains the documentation on running performance test of Corda Enterprise + +.. toctree:: + :maxdepth: 1 + + introduction + installation + running-jmeter-corda + jmeter-samplers + jmeter-testplans + r3-performance-runs \ No newline at end of file