corda/testing/qa/behave

Overview

The Behave Cucumber Scenarios defined under these sub-directories exercise Corda Enterprise and OS Corda distributions to include:

• Open Source master
• Corda Enterprise master
• Released versions of Open Source from V3.0 onwards
• Released versions of Corda Enterprise from V3.0 DEV PREVIEW 3 onwards

Compatibility scenarios exercise:

• mixed networks: combinations of OS and Corda Enterprise nodes configured within an R3 Network
• mixed-versioned nodes: combinations of different versions of OS (master, V3.0, V3.1, etc) and Corda Enterprise (master, V3.0 DP, V3.0 GA) nodes within an R3 Network.
• mixed-versioned corDapps: combinations of nodes running different versions of CorDapps (both valid and invalid upgrade mixes).
• mixed-services: ability to continue transacting upon notaries, oracles, doorman upgrades & changes.

Functional scenarios exercise the key behaviours of Corda Enterprise components and node configurations to include:

• basic cash management functions (issuance, transfer, redemption)
• vault usage of different database providers (H2, sql-server, postgreSQL)
• doorman certificate issuance and usage
• compatibility zone network parameter registration and updates
• network map registration and updates
• notary registration and updates

NOTE:

• the CTS framework uses R3's internal artifactory repositories to setup/configure officially released versions of Corda:
  1. https://ci-artifactory.corda.r3cev.com/artifactory/corda-releases
  2. https://ci-artifactory.corda.r3cev.com/artifactory/r3-corda-releases
• Scenarios specifying master will use the latest code checked into the respective Open Source and Enterprise repositories:
  1. https://github.com/corda/corda
  2. https://github.com/corda/enterprise/

Further goals of this project are to:

• automatically generate and register Docker images with target Azure Test Environments (eg. Functional/QA, CTS)
• effect execution of scenarios in specified Target Environments
• use Before and After hooks to pre-configure Target Environments to allow batch running of multiple scenarios (eg. a given feature will specify the environment once for one or many scenarios)

QA setup and usage instructions

Setup

Set up the staging area which will hold all the distributions of Corda referenced in the test scenarios:

• specify the staging root directory:

  $export STAGING_ROOT=$HOME/staging

• if any of your tests are referencing a master version of Corda OS, checkout the latest version of the Open Source repo and run:

  $ cd experimental/behave
  $ ./prepare.sh 

You should now see the following artifacts in your staging area:

$ ls -lR /Users/home/staging/corda/corda-master
total 285440
drwxr-xr-x  3 user  staff        96  4 Jun 16:11 apps
-rw-r--r--  1 user  staff  55439705  4 Jun 16:11 corda.jar
-rw-r--r--  1 user  staff  90699598  4 Jun 16:11 network-bootstrapper.jar

/Users/home/staging/corda/corda-master/apps:
total 4688
-rw-------  1 user  staff  2396954  4 Jun 16:11 corda-finance-4.0-SNAPSHOT.jar   

• similarly, if any of your tests are referencing a master version of Corda Enterprise, checkout the latest version of the Enterprise repo and run:

  $ cd experimental/behave
  $ ./prepare.sh 

You should now see the following artifacts in your staging area:

$ ls -lR /Users/home/staging/corda/r3corda-master/
total 503264
drwxr-xr-x  6 user  staff        192  4 Jun 16:39 apps
-rw-r--r--  1 user  staff    6239281  4 Jun 16:39 corda-rpcProxy.jar
-rw-r--r--  1 user  staff   66441262  4 Jun 16:38 corda.jar
-rw-r--r--  1 user  staff   71987460  4 Jun 16:39 dbmigration.jar
-rw-r--r--  1 user  staff  112985324  4 Jun 16:39 network-bootstrapper.jar
-rwxr-xr-x  1 user  staff       1224  4 Jun 16:39 startRPCproxy.sh

/Users/home/staging/corda/r3corda-master/apps:
total 82248
-rw-------  1 user  staff   2418144  4 Jun 16:38 corda-finance-R3.CORDA-3.0-SNAPSHOT.jar

Notes:

• all versions of Corda published to the Artifactory release repositories are automatically downloaded and configured on the fly as part of a test scenario.
• Docker must be running for SQL Server, Oracle and PostrgeSQL tests.

Usage

Checkout the latest version of the Enterprise repo and run:

  $cd experimental/behave
  ../../gradlew behaveJar

Change to the QA testing directory containing the scenario scripts and run:

  $cd ../../test/qa
  $export BEHAVE_JAR=$(ls ../../../experimental/behave/build/libs/corda-behave-*.jar | tail -n1)

To verify the tool is ready to be used run:

$ java -DSTAGING_ROOT=${STAGING_ROOT} -jar ${BEHAVE_JAR}

You should now see the following output providing details on how to run the tool:

Missing required option(s) [path]
Usage: ScenarioRunner [options] --path <location of feature scenario definitions>

Examples:
    ScenarioRunner -path <features-dir>
    ScenarioRunner -path <features-dir>/<name>.feature
    ScenarioRunner -path <features-dir>/<name>.feature:3:9

    ScenarioRunner -path <features-dir> --plugin html --tags @qa
    ScenarioRunner -path <features-dir> --plugin html --tags @compatibility

Please refer to the Cucumber documentation https://cucumber.io/docs/reference/jvm for more info.

Option (* = required)                  Description                          
---------------------                  -----------                          
-d                                                                          
--glue [location of additional step    (default: net.corda.behave.scenarios)
  definitions, hooks and plugins]                                           
* --path <Path location of .feature                                         
  specifications>                                                           
--plugin [register additional plugins  (default: pretty)                    
  (see https://cucumber.                                                    
  io/docs/reference/jvm)]                                                   
--tags [only run scenarios marked as                                        
  @<tag-name>]  

Note: passing in a -d option will perform a dry run only (validates the syntax of the scenario but does not execute the code)

Scenario scripts

There are currently two sets of scripts:

1. Functional

# the Cucumber behave test scenario definitions themselves
$ls -l functional/resources/features/functional.feature
# Unix script to easily run these
$ls -l functional/resources/scripts/run-functional.sh

2. Interoperability & Compatibility

# the Cucumber behave test scenario definitions themselves
$ls -l compatibility/resources/features/interoperability.feature
# Unix script to easily run these
$ls -l compatibility/resources/scripts/run-interoperability.sh

Note: the complete suite of Compatibility test scenarios have note yet been fully implemented.

You can now run the above test scripts in a number of different ways:

• as a complete suite called a feature set:

# run all scenarios in the functional.feature file
$java -DSTAGING_ROOT=${STAGING_ROOT} -jar ${BEHAVE_JAR} -path functional/resources/features/functional.feature

# run all scenarios in the interoperability.feature file
$java -DSTAGING_ROOT=${STAGING_ROOT} -jar ${BEHAVE_JAR} -path compatibility/resources/features/interoperability.feature

• as a set of tagged scenarios within a feature file:

# run all scenarios tagged with qa in the interoperability.feature file
$java -DSTAGING_ROOT=${STAGING_ROOT} -jar ${BEHAVE_JAR} -path compatibility/resources/features/interoperability.feature:@qa

• as an individual scenario for all example configurations:

# run the scenario defined on line 5 of the interoperability.feature file
$java -DSTAGING_ROOT=${STAGING_ROOT} -jar ${BEHAVE_JAR} -path compatibility/resources/features/interoperability.feature:5

# by default the above will run as many times as there are parameterised variable definitions

    Examples:
      | R3-Corda-Node-Version        | Corda-Node-Version | Currency |
      | r3-master                    | corda-master       | GBP      |
      | corda-3.0                    | corda-3.1          | GBP      |
      | R3.CORDA-3.0.0-DEV-PREVIEW-3 | corda-3.0          | GBP      |
      | R3.CORDA-3.0.0-DEV-PREVIEW-3 | corda-3.1          | GBP      |

• as an individual scenario for a single example configuration you will need to comment out the other example line items:

# run the scenario defined on line 5 of the interoperability.feature file
$java -DSTAGING_ROOT=${STAGING_ROOT} -jar ${BEHAVE_JAR} -path compatibility/resources/features/interoperability.feature:5

# by default the above will the scenario using *corda-3.0* and *corda-3.1* node versions only:

    Examples:
      | R3-Corda-Node-Version        | Corda-Node-Version | Currency |
#      | r3-master                    | corda-master       | GBP      |
      | corda-3.0                    | corda-3.1          | GBP      |
#      | R3.CORDA-3.0.0-DEV-PREVIEW-3 | corda-3.0          | GBP      |
#      | R3.CORDA-3.0.0-DEV-PREVIEW-3 | corda-3.1          | GBP      |