ExternalVendorCode/corda
1
0
Fork 0
mirror of https://github.com/corda/corda.git synced 2025-05-11 13:03:08 +00:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

ENT-2578 Initial documentation for performance test suite (#1489)

Browse Source 
* Initial introduction for performance test documentation

* Describing tests

* Some more doc structure

* Correct protocoll (rmi) for remote jMeter

* Remote connection/ssh tunneling

* Corda wrapper and arguments

* jMeter GUI

* rmi -> RMI

* headless jmeter

* Sampler descriptions

* SamplerClient interface and CashPaySampler

* Testplans

* Minor changes/typos

* Installation page and moved to development for now.

* Some clarifications

* fixes

* reproducing r3 perf runs

* Add todos/tickets for missing bits in the documentation.

* Information on reproducing the test cluster

* Edit TODOs

* Small fix to server installation

* Spelling of JMeter

* Small Typos

* More minor review fixes

* More minor review fixes

* Capitalisation

* CorDapp spelling

* Review fixes

* typo

* Remove blank lines

* typo

* Describe r3 test set-up

* Performance network topology

* Describe included test plans

* Some lines on creating testplans

* Fix spelling of command line flags

* Minor review fixes

* typo

* Clarify server set-up

* typo
This commit is contained in:
Christian Sailer 2018-10-30 09:14:05 +00:00 committed by GitHub
parent 412cc4622e
commit 1a9e7ab1ef
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
18 changed files with 894 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
docs/source/index.rst
View File

@ -88,6 +88,7 @@ application development please continue to refer to `the main project documentat
design/threat-model/corda-threat-model.md
design/data-model-upgrades/signature-constraints.md
design/data-model-upgrades/package-namespace-ownership.md
performance-testing/toc-tree.rst
.. conditional-toctree::
:caption: Operations
@ -95,4 +96,6 @@ application development please continue to refer to `the main project documentat
:if_tag: htmlmode
corda-nodes-index.rst
corda-networks-index.rstcertificate-revocation
corda-networks-index.rst
certificate-revocation

75
docs/source/performance-testing/installation.rst Normal file
View File

@ -0,0 +1,75 @@
===================================================
Obtaining and Installing the Performance Test Suite
===================================================
As a registered user of Corda Enterprise, you can download the performance test suite at **TODO** add location https://r3-cev.atlassian.net/browse/ENT-2643.
File Contents
=============
The performance test suite comes as a zip file containing the following files:
``jmeter-corda.jar``
The JAR file that contains the wrapped JMeter code to to drive performance tests.
``perftestcordapp.jar``
The performance test CorDapp used in for the built-in samplers and the included sample test plans. This needs to
be deployed to any node of the system under test if these test plans will be used.
Sample Testplans
A number of test plan JMX files **TODO** describe test plans https://r3-cev.atlassian.net/browse/ENT-2644
Sample ``jmeter.properties``
An example of the ``jmeter.properties`` file used to configure JMeter. If you need a custom configuration, it is
recommended to base it on this file.
Installation
============
Client Installation
-------------------
Simply create a working directory for JMeter Corda on the client machine and unzip the performance test suite to this
directory. This app requires an Oracle JRE version 1.8 build 172 or later. After unpacking,
you should immediately be able to run it from a shell by typing ``java -jar jmeter-corda.jar``. Please refer to
:doc:`running-jmeter-corda` for details and command line options.
.. _jmeter_server:
Installing JMeter server
------------------------
The same JAR file used for running the client can be run as a server as well by starting it with the ``-s`` flag as part
of the JMeter arguments. It can be installed by simply copying the JAR file to the server and having an adequate JRE
installed.
If SSH tunneling is in use, the server will also need access to the same RMI port mappings file used on the client side.
Also, if the client side uses a customised JMeter properties file that differs from the default one baked into the JMeter
JAR file by more than just the
list of remote host names from, this needs to be used on the server side as well. See :doc:`running-jmeter-corda`
for details on the command line options required. A typical command line might look like this::
java -jar <path to jmeter-corda jar> -XjmeterProperties <path to properties file> -XserverRmiMappings <path to RMI mappings file> -- -s
If you want to use JMeter Corda remotely, it is suggested to install the JMeter server as a system service on the servers
it is required to run on. Assuming a linux system with a systemd based run control, the service to install would look
something like this::
[Unit]
Description=Jmeter Corda
Requires=network.target
[Service]
Type=simple
User=<jmeter Corda user>
WorkingDirectory=<jmeter deploy directory>
ExecStart=/usr/bin/java -Xmx512m -jar <JMeter deploy directory>/jmeter-corda.jar -XjmeterProperties <path to properties file> -XserverRmiMappings <path to RMI mappings file> -- -s
Restart=on-failure
[Install]
WantedBy=multi-user.target
Sampler Client SDK
==================
Coming soon.

167
docs/source/performance-testing/introduction.rst Normal file
View File

@ -0,0 +1,167 @@
============
Introduction
============
Corda Performance Test Suite
============================
Use the Corda Enterprise performance test suite to stress/soak test a Corda installation, driving either a single
node or a small network of nodes including a notary.
It uses `Apache JMeter <https://jmeter.apache.org>`_ to start flows on nodes via RPC calls, and
capture the start/return rates and thus throughput of the system under test.
.. warning::
This guide assumes that you have a working Corda test network or
know how to set one up - the set-up and topology of the Corda network as well as the hardware used can have a sweeping
impact on performance, so there is not much point in performance testing before these points have been considered.
Test Architecture
=================
A typical test architecture consists of the following components:
- a Corda network to be tested. This should be a network of Corda nodes along with a notary that is self-contained
(i.e. does not depend on any external services). See e.g. :doc:`../corda-test-networks` for information on
setting up a network.
- a CorDapp that is to be tested and needs to be installed on the cluster
- an app to drive the test - Apache JMeter is used here
Apache JMeter
-------------
Apache JMeter runs tests that repeatedly trigger an action, wait for a response and record start/success/failure
timings and so on, and allow to view the result data interactively or rendered as reports in various formats. Run controls
like parallelising tasks, running tasks in a specific order and count and time based repetitions are already built in.
The interactions with the system under test are done via so called *samplers* (see :doc:`jmeter-samplers`) that can be
triggered by JMeter and then
run an action. JMeter has a number of built-in samplers, mostly around web technology, e.g. for HTTP requests, database
queries, starting scripts and so on. It is also possible to provide custom samplers that can run Java code when invoked.
For the Corda performance tests, a custom sampler is used that invokes one or more specific flows via remote procedure
calls (RPC), where all the required parameters for the flow and RPC call are passed to the sampler as parameters from
the test definition.
Interactive Mode
****************
By default, JMeter runs in interactive mode, i.e. it brings up a graphical user interface (GUI) that allows the user to
create, view, modify and run a test definition. Tests can either be in process (i.e. the sampler runs in the GUI
process) or can be fanned out to a set of JMeter server instances that will run under the control of a JMeter client
connected to them (see :ref:`Server Mode <JMeter-server>`)
Non-Interactive Mode
********************
Once a test definition is complete, it can be run in headless mode by providing the test definition and a report target
directory on the command line.
.. _JMeter-server:
Server Mode
***********
By adding the ``-s`` flag, JMeter can run as a server process that runs samplers controlled by a client connected to it
via Java Remote Method Invocation (RMI).
This allows a single client to e.g. run load from various servers for one test run and collate all the results in the
client.
jmeter-corda
------------
Apache JMeter can be fairly tricky to run in a specific configuration - therefore the Corda Enterprise performance test
suite provides a wrapper around JMeter that comes in a fat JAR with all required dependencies and a default configuration,
and sets up the required directories and config files that JMeter needs to start. It is also bundled with a set of default
Corda performance test samplers. On top of that, it supports opening SSH tunnels to machines running remote JMeter server
instances.
Performance Test CorDapp
------------------------
The performance test suite contains a performance test CorDapp (``perftest-cordapp.jar``) that is roughly modelled on the
finance CorDapp shipped with Corda Enterprise. It contains a number of flows that issue tokens, and pay them to other
parties. There are flows that e.g. issue and pay tokens with or without using coin selection, or create arbitrary
numbers of change output or coin input states to test the behaviour of the system when using various transaction sizes
and shapes.
Basic Performance Test Set-Up
-----------------------------
The typical set-up used for performance tests at R3 consists of a small Corda network of 2-4 nodes and a notary to
notarise transactions. These all run inside a datacenter or virtual network in the cloud with open connectivity (or at
least Corda P2P and RPC communication enabled between the nodes). On each of the node machines, an instance of JMeter
is running in server mode.
The driving app sits outside the network and connects to the JMeter servers through SSH tunnels. In the basic test
measuring the throughput of a node, the test definition instructs all JMeter servers to open RPC connections to one node,
thus saturating the RPC handler and driving the node as hard as possible. The test typically e.g. issues cash on the node
(no interaction with other nodes) or sends cash to a second node which requires sending P2P messages back and forth.
.. image:: resources/jmeter-network-overview.png
:scale: 75%
Performance Tests
=================
There are a number of different parts of the system that can be benchmarked with different performance tests, represented
by different test plans and/or samplers. In general, the closer a performance test is to real world load, the less it is
possible to isolate pinch points in the system under test. Hence a typical performance test run consists a of a number
of these tests that allow seeing where a performance drop off occurs.
If the reasons for a performance bottleneck cannot be figured out using a set of performance tests, it might be necessary
to attach a remote profile app to one of the nodes and profile a manual performance run using any of the suite of
existing JVM profiling tools available on the market.
The performance test suite contains test plans, CorDapp and sampler for the following tests:
Performance of a Single Node
----------------------------
These tests stress components in a single node, without any dependencies on other nodes in the flow.
Empty Flow
**********
This test starts a flow that does nothing - this gives us a timing for the overhead involved in starting a flow, i.e. RPC
handling, deserialization of the request, starting/winding down a flow and sending the response. Note that a flow that
requires inputs via RPC might have a larger overhead as these might need to be deserialised.
Issuance
********
A node issuing tokens to itself. In addition to the parts used above, this also loads/starts the CorDapp, creates states
in the vault and thus uses persistence to the database.
Inter-Node Performance
----------------------
These are flows that are closer to modelling real world loads to varying degrees.
Issue and Pay Flow
******************
This flow makes the node under test issue some cash to itself and then pays it to a second node. This involves initiating
a transaction with the target node, and then having the transaction notarised by a network notary, thus creating a load that
is similar to what a node will do under real world conditions. This flow has a few variations that can be controlled via
the test definition:
- Use coin selection - the flow can either just pay the issued cash or use coin selection to select the cash to pay (this
is used to isolate coin selection issues from general transaction performance)
- Anonymous identities - the flow can turn on anonymous identities. This means that a new private/public key pair will be
generated for each transaction, allowing to measure the overhead this introduces.
To test the throughput a single node can achieve, this flow is run against a single node from all JMeter servers. In order
to measure network throughput, it can also be run against all nodes from their respective JMeter server.
Advanced Flows
**************
The issue and pay flow creates a somewhat realistic load but still has a very uniform, artificial usage pattern of resources.
Therefore more advanced test flows/test plans have been developed that allow to issue a large amount of cash once and
then start to break it up in smaller payments, allowing the following settings to be tweaked:
- Number of states to be transferred in one transaction
- Number of change states created per transaction (i.e. the number of output states of the transaction)
- Number of input states to a new transaction (i.e. pay a larger sums from change shards of the previous transaction).
Advanced tests also include testing e.g. connecting to the target node via float/firewall.

157
docs/source/performance-testing/jmeter-samplers.rst Normal file
View File

@ -0,0 +1,157 @@
===============
JMeter Samplers
===============
JMeter uses samplers to interact with the system under test. It comes with a number of built-in
`samplers <https://jmeter.apache.org/usermanual/component_reference.html#samplers>`_, mostly
around testing web sites and infrastructure.
Corda Flow Sampling
===================
For performance testing Corda, the `Java Request sampler
<https://jmeter.apache.org/usermanual/component_reference.html#Java_Request>`_ is used. This sampler works by calling
a Java class implementing the ``org.apache.jmeter.protocol.java.sampler.JavaSamplerClient`` interface and passing
a set of parameters::
public interface JavaSamplerClient {
void setupTest(JavaSamplerContext context);
SampleResult runTest(JavaSamplerContext context);
void teardownTest(JavaSamplerContext context);
Arguments getDefaultParameters();
}
When JMeter runs a test using such a sampler, the ``setupTest`` method is called once at the beginning of the test for
each instance of the sampler. The actual test will be calling the``runTest`` method one ore more times and aggregating
the sample results. Finally, the ``teardownTest`` method will be called once per instance. As thread groups in a JMeter
test plan run all of their content in parallel, a new instance of the sampler will be created for each thread in the
group, and ``setupTest`` and ``teardownTest`` get called once per thread. Note that ``teardownTest`` will only be called
once all thread groups have run and the test plan is terminating.
Provided Sampler Clients
========================
JMeter Corda provides a number of sampler client implementations that can be used with Java Request sampler. They all
share some common base infrastructure that allows them to invoke flows on a Corda node via RPC. All of these samplers
are built against a special performance test CorDapp called ``perftestcordapp``. On each call to run the sampler, one
RPC flow to the respective flow used by this sampler is made, and the run function will block until the flow result is
returned.
``EmptyFlowSampler``
This sampler client has the class name ``com.r3.corda.jmeter.EmptyFlowSampler`` and starts the flow
``com.r3.corda.enterprise.perftestcordapp.flows.EmptyFlow``. As the name suggests, the ``call()`` method of this flow
is empty, it does not run any logic of its own. Invoking this flow goes through the whole overhead of invoking a flow
via RPC without adding any additional flow work, and can therefore be used to measure the pure overhead of invoking
a flow in a given set-up.
.. image:: resources/empty-flow-sampler.png
:scale: 85%
This sampler client requires the minimal set of properties to be required that it shares with all Corda sampler
clients documented here:
label
A label for reporting results on this sampler - if in doubt what to put here ``${__samplername} will fill in the
sampler client class name.
host
The host name on which the Corda node is running. The sampler client will connect to this host via RPC. This name needs
to be resolvable from where the sampler client is running (i.e. if using remote JMeter calls, this means from the
server, not the client machine).
port
The RPC port of the Corda node.
username
The RPC user name of the Corda node.
password
The RPC password of the Corda node.
``CashIssueSampler``
This sampler client has the class name ``com.r3.corda.jmeter.CashIssueSampler`` and starts the flow
``com.r3.corda.enterprise.perftestcordapp.flows.CashIssueFlow``. This flow will self-issue 1.1 billions
of cash tokens on the node it is running on and store it in the vault.
.. image:: resources/cash-issue-sampler.png
:scale: 85%
In addition to the common properties described above under ``EmptyFlowSampler``, this sampler client also requires:
notaryName
The X500 name of the notary that will be acceptable to transfer cash tokens issued via this sampler. Issuing tokens
does not need to be notarised, and therefore invoking this sampler does not create traffic to the notary. However,
the notary is stored as part of the cash state and must be valid to do anything else with the cash state, therefore
this sampler will check the notary indentity against the network parameters of the node.
``CashIssueAndPaySampler``
This sampler client has the classname ``com.r3.corda.jmeter.CashIssueAndPaySampler`` and can start either the
the flow ``com.r3.corda.enterprise.perftestcordapp.flows.CashIssueAndPaymentFlow`` or
``com.r3.corda.enterprise.perftestcordapp.flows.CashIssueAndpaymentNoSelection``, depending on its parameters.
Either way it issues 2 million dollars in tokens and then transfers the sum to a configurable other node, thus
invoking the full vault access, peer-to-peer communication and notarisation cycle.
.. image:: resources/cash-issue-and-pay-sampler.png
:scale: 85%
In addition to the parameters required for the ``CashIssueSampler``, this also requires:
otherPartyName
The X500 name of the recipient node of the payment.
useCoinSelection
Whether to use coin selection to select the tokens for paying or use the cash reference returned by the issuance
call. The value of this flag switches between the two different flows mentioned above. Coin selection adds a set
of additional problems to the processing, so it is of interest to measure its impact.
anonymousIdentities
Switches the creation of anonymised per-transactions keys on and off.
``CashPaySampler``
A sampler that issues cash once per run in its ``setupTest`` method, and then generates a transaction to pay 1 dollar "numberOfStatesPerTx" times
to a specified party per sample, thus invoking the notary and the payee via P2P.
This allows us to test performance with different numbers of states per transaction, and to eliminate issuance from
each sample (unlike CashIssueAndPaySampler).
The classname of this sampler client is ``com.r3.corda.jmeter.CashPaySampler``.
.. image:: resources/cash-pay-sampler.png
:scale: 85%
In addition to the base requirements as in the ``CashIssueSampler``, this sampler client requires the following
parameters:
otherPartyName
The Corda X500 name of the party receiving the payments
numberOfStatesPerTx
The number of $1 payments that are batched up and transferred to the recipient in one transaction, thus allowing
to observe the impact of transaction size on peer to peer throughput and notarisation.
anonymousIdentities
Switches the creation of anonymised per-transactions keys on and off.
Custom Sampler Clients
======================
The sampler clients provided with JMeter Corda all target the performance test CorDapp developed along with the
performance test tool kit. In order to drive performance tests using different CorApps, custom samplers need to be
used that can drive the respective CorDapp via RPC.
Loading a Custom Sampler Client
-------------------------------
The JAR file for the custom sampler needs to be added to the search path of JMeter in order for the Java sampler to
be able to load it. The ``-XadditionalSearchPaths`` flag can be used to do this.
If the custom sampler uses flows or states from another CorDapp that is not packaged with the
JMeter Corda package, this needs to be on the classpath for the JMeter instance running the sampler as the RPC interface
requires instantiating classes to serialize them. The JMeter property ``user.classpath`` can be used to set up additional
class paths to search for required classes.
Writing a Custom Sampler Client
-------------------------------
Coming soon

111
docs/source/performance-testing/jmeter-testplans.rst Normal file
View File

@ -0,0 +1,111 @@
====================================
Understanding and Creating Testplans
====================================
JMeter offers a very flexible and powerful tool kit to build a large variety of testplans. This document can only give
a brief overview over what testplans are typically used for Corda performance testing. To get an overview of what is
available in JMeter, and if you are looking into writing more elaborate testplans, please refer to the `JMeter user
manual <https://jmeter.apache.org/usermanual/index.html>`_.
Structure of a Simple Testplan
==============================
The testplan is a hierarchy of configuration elements. The only elements used in our simple test plans are
*Variables*, *Thread Groups*, *Samplers* and *Listeners*. Rather than creating a new testplan from scratch,
it might a good idea to take a copy of one of the provided example test plans and modify that.
.. image:: resources/jmeter-testplan.png
User Defined Variables
A variables element defines key/value pairs that can be used in all following following elements instead of string
literals. They can be referenced by using a ``$`` sign and curly braces, e.g. ``${varName}``.
.. image:: resources/variables.png
Thread Group
A thread group collects a set of actions that form one step in the test plan. All elements within a thread group
will be run in order. As the name suggest, a thread group can spin up threads to run serveral instances of its
content in parallel. Note that everything inside the group will be run in every thread (including any initialisation
or tear down steps). However, the tear down steps will only be run when all thread groups have finished at the end
of the test plan - not at the end of the thread group's execution. It cannot be used to release resources that
the next thread group is supposed to reuse.
The thread group also allows to repeat the action that has been configured. Repitition can either be count based, or
scheduler based (e.g. for 5 minutes). Counter based repition will loop the given number of times for each thread,
so 10 thread with 10 repetitions means 100 runs of group content.
If a test plan contains a list of thread groups, they will be run sequentially. If the runtime of the thread group
is controlled via a scheduler, a start-up delay is configurable that the process will wait between finishing the
previous thread group and starting the next one, e.g. to let the system under test finish any lingering requests,
flush queues and return to idle state.
Sampler
Usually, there will be a sampler inside the thread group that creates the load for the Corda network via RPC
invocations. Note that the thread group will call ``setupTest`` only before the first iteration, and ``teardownTest``
only after all tests in all thread groups have finished, but will call ``runTest`` for every loop iteration
configured in the thread group.
However, all task are run for each thread in parallel.
Listeners
Listeners collect the output of samplers and handle it by either displaying it, storing it to file or
aggregating it in some way. Note that starting and stopping does not clear data from the listeners, and some
statistics are calculated across runs and breaks, so clearing the relevant data before starting a run is vital when
running from the GUI. The description of all built-in listeners can be found in the `documentation
<https://jmeter.apache.org/usermanual/component_reference.html#listeners>`_ - here is a short list of listeners
typically used for Corda runs.
View Results in Table
This outpug handler is useful for debugging the test system - every result will be displayed as a line in a table,
including a status and stats about sent and received bytes. The table can become very long and unwieldy quickly
when running with many threads and/or loops.
Graph Results
This aggregator plots various stats about processing time and throughput over time. It can be handy to view
trends, but as the time window is fixed and it loops back to the beginning and plots over existing graphs,
the usefulness for long running tests is limited.
Aggregate Report
This aggregator collects processing time stats and throughput on a per thread group basis as a table, and keeps
a running total for the run.
Aggregate Graph
Like the Aggregate Report, but also offers to tools to plot the results.
Included Testplans
==================
Two testplans are included with the performance test suite, to be found in the ``Sample Testplans`` directory in the zip
file. Note that both testplans are examples how they can look and are not runnable out of the box - you need to supply
values for host names, party/notary names, RPC credentials and so on for your Corda installation.
``Example Flow Request.jmx``
This is a very simple testplan that has one thread group that just runs cash self issue. Everything is configured
directly on the sampler's page - just enter the details of your installed Corda node here and you should be good
to go. The structure of this test plan is a good place to start to create your own test plans.
``NightlyBenchmarkSample.jmx``
This is a copy of the test plan that is used for performance testing Corda Enterprise at R3. This plan has a lot of
different tests, each in its own thread group. All the thread groups will be run one after another. These different
tests show use of all the different sampler clients described in :doc:`jmeter-samplers`.
This testplan uses variables to avoid repetition of values that might require changing. You need to enter the
appropriate values for your Corda network installation here to get the plan working.
A useful way to try out bits of the plan is to load it in an interactive JMeter session and disable most of the
thread groups in order to only run a few tests. (Thread groups can be enabled/disabled in the right-click context
menu).
Creating Testplans
==================
The JMeter GUI can be used to create and try out new testplans. The easiest way of doing this to take a copy of an
existing testplan and modify this as the structure of all testplans is fairly similar.
- Optionally, there can be user variables and set-up steps defined at the top of the test plan
- Each plan needs one or more thread groups that contain the actual tests to run, i.e. samplers that exercise the system
under test. Each thread group can just contain a sampler, or furhter logical elements like repetition/decision elements
that run different samplers in various combinations.
- Thread groups can contains listeners that only process results from this thread group
- The testplan should have at least one global listener that captures results from all thread groups.
Detailed description of the parts of a test plan can be found in the `JMeter documentation
<https://jmeter.apache.org/usermanual/test_plan.html>`_.

155
docs/source/performance-testing/r3-performance-runs.rst Normal file
View File

@ -0,0 +1,155 @@
===========================================
Reproducing the R3 performance test numbers
===========================================
The performance test suite contains all the code and configuration to reproduce the performance tests run at R3 and
presented or published occasionally.
Test Network
============
The performance test runs in a self contained test network consisting of
- 4 node machines, each running one Corda node
- 1 simple notary running on a separate machine
- separate database servers for all the nodes
- optionally a doorman/network map server - the test network can be set up using the bootstrapper or the network map
approach - this does not impact performance however.
The performance network sits behind a firewall that allows incoming connections only via SSH, however the machines
within the network can connect to each other on a range of ports (Corda Remote Procedure Call (RPC), Corda peer to
peer connection (P2P) and JDBC connections to various database servers are all allowed).
.. image:: resources/performance-cluster.png
:scale: 75%
A Corda JMeter server instance is running on all 4 nodes and on the notary server. The JMeter client connects to the
JMeter server instances via SSH tunnels. As the JMeter servers run within the firewalled network, they are free to
open RPC connections to any of the Corda nodes - which one they connect to is configured in the JMeter test plan.
They usually connect all to one node to drive the test flows as hard as possible, but might connect to different
nodes for different tests.
The location of the node's working directory is important - as nodes use Artemis for peer to peer communication, it is
crucial that the Artemis spool directory is on a fast storage solution, e.g. a local SSD. It should not be on a network
mount or SAN drive.
There is a separate database for each node - for historical reasons, the notary and node 2 share a database in the,
set-up, therefore node 2 gets the lightest load in order to not cause a slow-down by overloading the database. It
is entirely possible to run the whole network with one database that has different users/schemas for each node and the
notary, however, this set-up might not reach full performance as database throughput becomes a bottleneck.
The default set-up uses Microsoft SQL Server instances as databases, but some testing has been done with Postgres
and Oracle 12 databases.
Machines in the Test Network
----------------------------
Microsoft Azure
+++++++++++++++
The initial performance network was set up on a cluster of Microsoft Azure nodes. The following specs have been chosen
for the network setup
========= ================== =========================================== =========================================================
Machine Type Description Notes
========= ================== =========================================== =========================================================
node 1 Standard D32s 32 cores, 128GB memory, Ubuntu 16.04 main target node for flow invocation
node 2 Standard F8s 8 cores, 16GB memory, Ubuntu 16.04 mostly used for JMeter server, Corda node lightly used
node 3 Standard D32s 32 cores, 128GB memory, Ubuntu 16.04 target node configured with float/Corda firewall
node 4 Standard D16s 16 cores, 64GB memory, Ubuntu 16.04 recipient node for payment flows
notary Standard F8s 8 cores, 16GB memory, Ubuntu 16.04 notary node
sql 1 Standard DS12 v2 4 cores, 28GB memory, Windows Server 2016 SQL Server for node 1
sql 2 Standard DS12 v2 4 cores, 28GB memory, Windows Server 2016 SQL Server for node 2 and notary
sql 3 Standard DS12 v2 4 cores, 28GB memory, Windows Server 2016 SQL Server for node 3
sql 4 Standard DS12 v2 4 cores, 28GB memory, Windows Server 2016 SQL Server for node 4
========= ================== =========================================== =========================================================
This set-up has been used to performance tune Corda until Autumn 2018. One of the lessons learned was that performance
testing in the cloud has a few drawbacks:
- The performance of virtual machines in the cloud is not always consistent. How fast they go exactly depends on what
else is going on, so fluctuations in measured performance could not always be attributed to changes to Corda.
- Cloud databases are optimised for web apps etc. Therefore, they are mostly read-optimised. Corda, however, writes
a lot of checkpoints and transactions to the database which required us to install custom database instances.
- The virtual hard drives in the cloud have very limited througput, which causes bottlenecks for Artemis queues,
log file writing and database writing. Getting more performant storage in the cloud is either very expensive or
plainly not possible.
Dedicated Hardware
++++++++++++++++++
Due to the observed drawbacks with performance testing in the cloud, the performance testing has been moved to
dedicated hardware in a rented data center. To build up the performance test network, the following machine types are
acquired. They are all placed in the same data center, with 10Gb/s network connectivity.
Node Servers
Intel 2x Xeon E5-2687Wv4, 24 cores hyperthreaded, 256GB DDR4 ECC 2133 MHz, 2x 400GB NVME
Database Servers
Intel Xeon E5-2687Wv4 - 12 cores hyperthreaded, 256GB DDR4 ECC 2133 MHz, 2 x 1GB NVME
Notary Server
Intel Xeon E5-2687Wv4 - 12 cores hyperthreaded, 256GB DDR4 ECC 2133 MHz, 2 x 1GB NVME
Node Configuration
------------------
For performance runs, the nodes run with pretty standard configuration. However, there are a few tweaks:
- The JVM gets set up with 8 gigabyte heapsize and using the G1 garbage collector.
- The database transaction isolation level is set to ``READ_COMMITTED``.
- In the tuning part of the Enterprise configuration, a generous number of flow and RPC threads are configured.
- Dev mode is turned off for all nodes including the notary.
::
custom = {
jvmArgs = [ "-Xmx8g", "-Xms8g", "-XX:+UseG1GC" ]
}
database {
transactionIsolationLevel = "READ_COMMITTED"
}
enterpriseConfiguration = {
tuning = {
rpcThreadPoolSize = 32
flowThreadPoolSize = 128
}
}
devMode = false
Depending on the type of machines used, it is recommended that the ``rpcThreadPoolSize`` does not exceed the number of
virtual cores (i.e 2 x number of cores when hyperthreading is enabled), and the ``flowThreadPoolSize`` should be around
4 times the number of virtual cores. Depending on other factors like disk I/O, networking, memory etc. tweaking these
numbers can yield better performance results.
Notary Configuration
--------------------
When using a simple, single node notary, it has the following configuration in addition to the above::
notary {
className="net.corda.notary.jpa.JPANotaryService"
extraConfig {
}
validating=false
}
Database Configuration
----------------------
Each node has its own, dedicated database server running Microsoft SQL Server 2016 Standard.
Note that the disk write latency and throughput on the database machine are critical for the node's performance as the
Corda checkpointing mechanism means that many checkpoints are written to the database, and then deleted again when the
flow finishes. Checkpoints will only be read when a flow is resumed or restarted, which might not happen a lot when
running a large number of short-lived flows. Therefore, a read-optimised database server (as e.g. the hosted database
instances offered by the big cloud providers) is not ideal for a busy Corda node and will limit throughput.
It is recommended to manage the database server manually and optimise for write throughput at least as much as for read.
Test Plan
=========
For the test, JMeter server instances on 4 node machines and the simple notary machine all connect to one node via RPC
to drive the tests. A external JMeter client uses the ``NightlyBenchmark.jmx`` test plan to run the tests.

BIN
docs/source/performance-testing/resources/batch-notarise-sampler.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 19 KiB

BIN
docs/source/performance-testing/resources/cash-issue-and-pay-sampler.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

BIN
docs/source/performance-testing/resources/cash-issue-sampler.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 14 KiB

BIN
docs/source/performance-testing/resources/cash-pay-sampler.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 18 KiB

BIN
docs/source/performance-testing/resources/empty-flow-sampler.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 13 KiB

BIN
docs/source/performance-testing/resources/jmeter-buttons.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 90 KiB

BIN
docs/source/performance-testing/resources/jmeter-network-overview.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 54 KiB

BIN
docs/source/performance-testing/resources/jmeter-testplan.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 198 KiB

BIN
docs/source/performance-testing/resources/performance-cluster.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 121 KiB

BIN
docs/source/performance-testing/resources/variables.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 2.8 KiB

210
docs/source/performance-testing/running-jmeter-corda.rst Normal file
View File

@ -0,0 +1,210 @@
====================
Running JMeter Corda
====================
Jmeter Corda is distributed as a runnable "fat" JAR containing all the dependenices required to run the application.
It comes prepacked with Corda :doc:`jmeter-samplers` and a wrapper that sets up a basic configuration and allows
configuration of SSH tunnels. The arguments for the ``jmeter-corda`` command line fall into two categories: there are
a number of arguments that are consumed by the custom wrapper, followed by a double dash ``--``. Anything after this
will be passed on to the JMeter code as JMeter arguments (though the wrapper can add more arguments to this). The typical
call will be::
java -jar jmeter-corda-jar <wrapper arguments> -- <jmeter arguments>
JMeter Corda Wrapper Arguments
==============================
The JMeter Corda wrappper understands the following optional arguments:
``-?``, ``-h``, ``--help``
This will print a help text for the JMeter Corda wrapper arguments and also trigger printing of the JMeter help
text
``-Xssh``
This tells the wrapper to set up SSH tunnels. It takes a list of host names as argument. See :ref:`ssh_tunnel` for
details.
``-XsshUser``
User name to use for remote servers if creating SSH tunnels. This defaults to the current user name.
``-XadditionalSearchPaths``
A semicolon separated list of directories containing class files or JAR files that JMeter should consider for finding
samplers or plug-ins. The wrapper sets up a default search path that contains the samplers included in the JAR and
the required plug-ins to talk to Corda. Any custom samplers to be used in tests need to be included here.
``-XjmeterProperties``
Path to a JMeter properties file. JMeter always needs configuration via a properties file - there is a ``jmeter.properties``
file included in the jmeter-corda.jar that sets sensible defaults. The wrapper will make JMeter use the pre-packed
properties if this option is not set.
``-XserverRmiMappings``
Port mappings for the remote method invocation tunnels when setting up SSH tunnels. See :ref:`ssh_tunnel` for details.
These arguments have to come first on the command line. They can be followed by a double dash and JMeter arguments
that will be fed through to JMeter. The wrapper will add some arguments to the list passed on to JMeter and does disallow
some arguments. Some JMeter arguments used/discussed on this page are:
``-p``
This is the way of telling JMeter what properties file to use. This will **always** be set by the wrapper code and
must **not** be set on the command line. Use the ``-XjmeterProperties`` argument to the wrapper instead.
``-s``
This makes JMeter run in server mode, i.e. it will run headless and wait for instructions from a client via remote
method invocation. See also :ref:`jmeter_server`.
Running the JMeter GUI
======================
By default, JMeter Corda will start up as client in GUI mode. The GUI allows to view/create test plans, and run tests
interactively either locally or remotely, and view results. The UI has a set of handy butons at the top that allow to
quickly create or load a testplan, control local or remote test runs, and clear any test results collected so far.
The latter is particularly important when looking at averages, as old data might skew the results being looked at.
.. image:: resources/jmeter-buttons.png
:scale: 75%
The *clear current view* button only clears the data in the currently viewed output collector - if a test plan has several
output listeners defined (as in the example above, we have *Aggregate Graph*, *Graph Results* and *View Results in Table*),
any collector not currently selected is not affected. *Clear all data* will clear the results from all collectors.
See :doc:`jmeter-testplans` for details on the testplan, and how to create one.
Running JMeter headless
=======================
JMeter can be run in headless mode, i.e. without starting up a GUI, by giving it a test definition on the command line.
This allows to run performance tests excluding any influences from GUI rendering or interaction issues, and allows
to run performance tests in an automated manner, e.g. as part of a nightly build. This is controlled by JMeter arguments
passed through to the JMeter instances. The relevant arguments are:
``-n``
This will run a JMeter client in headless mode. This requires to also pass a test definition via ``-t`` and an output
file via ``-l``.
``-t <testplan.jmx>``
Provide a testplan xml file to load and run.
``-l <results.jtl>``
Path to an output jtl file to be written - this is a JMeter specific CSV format.
``-r``
Run via RMI against the remote servers configured in the JMeter properties file. Without this option, JMeter
will run the testplan locally.
Example::
java -jar jmeter-corda.jar -Xssh node1.mydomain.com node2.mydomain.com -- -n -t /home/<user>/tesplan.jmx -l /home/<user>/results.jtl -r
See `JMeter documentation <https://jmeter.apache.org/usermanual/get-started.html#non_gui>`_ for details.
Using remote JMeter
===================
JMeter can connect to JMeter server instances using Java Remote Invocation (RMI). The hosts and ports a JMeter client
instance can connect to are listed in the properties file it is using. When running the GUI, the remote hosts will be
used when using the start/stop remotely functions, while the simple start/stop will still work locally. When running
headless, the remote hosts will be used when the ``-r`` option is given.
If no SSH tunneling is required, the ``-R`` option can be used to pass in a different list of remote hosts to the client.
If the SSH tunneling set-up from the wrapper is used, this option is **not** supported and **will fail**.
.. _ssh_tunnel:
Ssh Tunnel Set-Up
-----------------
The RMI traffic requires non-standard ports to be open, therefore the standard practice is to
route it via SSH tunnels. Therefore, the remote hosts in the default configuration are all specified as ``127.0.0.1``,
i.e. localhost, with different port numbers, for which SSH tunnels will be opened to the hosts specified on the command
line using the ``-Xssh`` flag.
Due to the Java RMI architecture, three tunnels need to be opened for each remote JMeter instace - one for the connection
to the RMI server, and an outbound and inbound tunnel. For this to work, it is crucial that the JMeter client and the
server instances agree on the ports they use, and the hostnames used on the command line and the server RMI mapping
do match exactly.
The Corda JMeter executable contains code to set up the tunnels and orchestrate the connections on the client and server
side. All the tunnels are set up from the client side. The information required to set up the tunnels comes from three
places:
- The number of hosts to create tunnels for and their client side RMI server ports are read from the ``jmeter.properties``
file that is used to run this instance of JMeter. For using the SSH tunnel, localhost addresses are listed here as the
actual remoting is done via SSH. By default, the five remote addresses shown below are listed in the properties file
included in the JAR. This configuration is only relevant on the client side::
#---------------------------------------------------------------------------
# Remote hosts and RMI configuration
#---------------------------------------------------------------------------
# Remote Hosts - comma delimited
remote_hosts=127.0.0.1:20100,127.0.0.1:20101,127.0.0.1:20102,127.0.0.1:20103,127.0.0.1:20104
- The actual host names are passed to the client process on the command line with the ``-Xssh`` command line flag. This
takes a list of space separated, fully qualified host names. They must be actual hostnames (not just IP addresses), and
for the look-up to work, must match between the `hostname` entry on the server and the DNS name - i.e. the result of
calling `hostname` on the server should be the same as the first part of the hostname used on the command line up to
the first dot::
java -jar jmeter-corda.jar -Xssh node1.mydomain.com node2.mydomain.com
- The server side RMI server port can be configured in the ``jmeter.properties`` file as property ``server.rmi.port``,
and defaults to ``1099``. It is crucial that the same value is used on the server and client side.
- The client local RMI port (server to client callback) is also configured in the ``jmeter.properties`` file as property
``client.rmi.localport``. Again it is crucial that this value is the same on server and client side.
- The server local RMI port (used to send requests to the server) needs to be different for each host so different
information can be send to different tunnels for different servers from the client. Therefore, the property
``server.rmi.localport`` in the ``jmeter.properties`` file is only used as a fallback - it would only work for a single
JMeter server instance being used at a time. Instead, the mappings of port to host name come from an additinal config
file ``server-rmi.config`` that lists a mapping of unqualified hostname to port::
# lines starting in # are comments and ignored
node1:10101
node2:10102
These values must be the same on client and server side and must match the configured hostname of the target host.
The configuration file can be passed to the process using the ``-XserverRmiMappings`` command line flag.
Connecting to remote JMeter servers
-----------------------------------
The default JMeter properties file in the Corda JMeter JAR assumes that 5 instances of JMeter server are running on hosts
that can be reached via SSH tunnels. When using the run remotely function in the GUI, JMeter will try to connect to the
hosts listed in its properties file.
If you need to specify a different list of remote hosts, e.g. for a different number of hosts, to specify different port
numbers or to specify explicit namesin case SSH tunnelling is not required, you will have to take a copy of the
``jmeter.properties`` file, modify it accordingly and specify the file to use on the command line using the
``-XjmeterProperties`` argument.
Running with SSH tunnels
++++++++++++++++++++++++
In order to run the client with SSH tunnels, a typical command line would be::
java -jar jmeter-corda.jar -Xssh <server1 hostname> [<server hostname> ...] [-XsshUser <remote user name>] [-XjmeterProperties <jmeter properties file>] -XserverRmiMappings <RMI mappings file> [-- <jmeter arguments>]
The SSH tunnel code assumes private/public key authentication, interactive authentication is not supported. In order to
establish the SSH tunnels, an SSH agent must be running that has exactly one private key loaded that can be
used to establish connections to all server hosts. When starting from a POSIX system (Unix/linux/Mac OS), the tool expects
a POSIX SSHagent to be started, and the environment variable ``SSH_AUTH_SOCK`` to be set. On a Windows system, the key
is expected to be served up by the Putty SSH agent `Pageant <https://www.ssh.com/ssh/putty/putty-manuals/0.68/Chapter9.html>`_.
Running without SSH tunnels
+++++++++++++++++++++++++++
If SSH tunneling is not required, the list of actual host names needs to be in the JMeter properties file as property
``remote_hosts``, or can be passed in with the JMeter argument ``-R``:
``jmeter.properties``
``remote_hosts=node1.mydomain.com,node2.mydomain.com``
Command line
``java -jar jmeter-corda.jar -- -Rnode1.mydomain.com,node2.mydomain.com``
None of the Corda provided SSH code options need to be configured, so the server command line should just include the
``-s`` option, and the client command line would be::
java -jar jmeter-corda.jar -XjmeterProperties <property file including hostnames> [-- <jmeter arguments>]
It is recommended to consult the JMeter manual on `Remote Testing <https://jmeter.apache.org/usermanual/remote-test.html>`_
when setting up remote testing without SSH tunneling.

15
docs/source/performance-testing/toc-tree.rst Normal file
View File

@ -0,0 +1,15 @@
====================================
Corda Enterprise Performance Testing
====================================
This section contains the documentation on running performance test of Corda Enterprise
.. toctree::
:maxdepth: 1
introduction
installation
running-jmeter-corda
jmeter-samplers
jmeter-testplans
r3-performance-runs