corda/testing/qa/behave/README.md

Overview
========

The Behave Cucumber Scenarios defined under these sub-directories exercise Corda Enterprise and OS Corda distributions to include:
- Open Source master
- Corda Enterprise master
- Released versions of Open Source from V3.0 onwards
- Released versions of Corda Enterprise from V3.0 DEV PREVIEW 3 onwards

**Compatibility** scenarios exercise:
- mixed networks: combinations of OS and Corda Enterprise nodes configured within an R3 Network
- mixed-versioned nodes: combinations of different versions of OS (master, V3.0, V3.1, etc) and Corda Enterprise (master, V3.0 DP, V3.0 GA) nodes within an R3 Network.
- mixed-versioned corDapps: combinations of nodes running different versions of CorDapps (both valid and invalid upgrade mixes).
- mixed-services: ability to continue transacting upon notaries, oracles, doorman upgrades & changes.

**Functional** scenarios exercise the key behaviours of Corda Enterprise components and node configurations to include:
- basic cash management functions (issuance, transfer, redemption)
- vault usage of different database providers (H2, sql-server, postgreSQL)
- doorman certificate issuance and usage
- compatibility zone network parameter registration and updates
- network map registration and updates
- notary registration and updates

NOTE:
- the CTS framework uses R3's internal artifactory repositories to setup/configure officially released versions of Corda:
    1. https://ci-artifactory.corda.r3cev.com/artifactory/corda-releases
    2. https://ci-artifactory.corda.r3cev.com/artifactory/r3-corda-releases
- Scenarios specifying `master` will use the latest code checked into the respective Open Source and Enterprise repositories:
    1. https://github.com/corda/corda
    2. https://github.com/corda/enterprise/
    
Further goals of this project are to:
- automatically generate and register Docker images with target Azure Test Environments (eg. Functional/QA, CTS)
- effect execution of scenarios in specified Target Environments 
- use Before and After hooks to pre-configure Target Environments to allow batch running of multiple scenarios (eg. a given
  feature will specify the environment once for one or many scenarios)
  
QA setup and usage instructions
===============================

Setup
-----
Set up the staging area which will hold all the distributions of Corda referenced in the test scenarios:

- specify the staging root directory: 

```bash
  $export STAGING_ROOT=$HOME/staging
```

- if any of your tests are referencing a master version of [Corda OS](https://github.com/corda/corda), checkout the latest version of the Open Source repo and run:

```bash
  $ cd experimental/behave
  $ ./prepare.sh 
```

   You should now see the following artifacts in your staging area:
   
```bash
$ ls -lR /Users/home/staging/corda/corda-master
total 285440
drwxr-xr-x  3 user  staff        96  4 Jun 16:11 apps
-rw-r--r--  1 user  staff  55439705  4 Jun 16:11 corda.jar
-rw-r--r--  1 user  staff  90699598  4 Jun 16:11 network-bootstrapper.jar

/Users/home/staging/corda/corda-master/apps:
total 4688
-rw-------  1 user  staff  2396954  4 Jun 16:11 corda-finance-4.0-SNAPSHOT.jar   
```

- similarly, if any of your tests are referencing a master version of [Corda Enterprise](https://github.com/corda/enterprise), checkout the latest version of the Enterprise repo and run:

```bash
  $ cd experimental/behave
  $ ./prepare.sh 
```
   
   You should now see the following artifacts in your staging area:

```bash
$ ls -lR /Users/home/staging/corda/r3corda-master/
total 503264
drwxr-xr-x  6 user  staff        192  4 Jun 16:39 apps
-rw-r--r--  1 user  staff    6239281  4 Jun 16:39 corda-rpcProxy.jar
-rw-r--r--  1 user  staff   66441262  4 Jun 16:38 corda.jar
-rw-r--r--  1 user  staff   71987460  4 Jun 16:39 dbmigration.jar
-rw-r--r--  1 user  staff  112985324  4 Jun 16:39 network-bootstrapper.jar
-rwxr-xr-x  1 user  staff       1224  4 Jun 16:39 startRPCproxy.sh

/Users/home/staging/corda/r3corda-master/apps:
total 82248
-rw-------  1 user  staff   2418144  4 Jun 16:38 corda-finance-R3.CORDA-3.0-SNAPSHOT.jar
```

Notes: 

- all versions of Corda published to the Artifactory release repositories are automatically downloaded and configured
on the fly as part of a test scenario.      
- Docker must be running for SQL Server, Oracle and PostrgeSQL tests.

Usage
-----

Checkout the latest version of the Enterprise repo and run:

```bash
  $cd experimental/behave
  ../../gradlew behaveJar
````

Change to the QA testing directory containing the scenario scripts and run:
```bash
  $cd ../../test/qa
  $export BEHAVE_JAR=$(ls ../../../experimental/behave/build/libs/corda-behave-*.jar | tail -n1)
```

To verify the tool is ready to be used run:

```bash
$ java -DSTAGING_ROOT=${STAGING_ROOT} -jar ${BEHAVE_JAR}
```

You should now see the following output providing details on how to run the tool:

```bash
Missing required option(s) [path]
Usage: ScenarioRunner [options] --path <location of feature scenario definitions>

Examples:
    ScenarioRunner -path <features-dir>
    ScenarioRunner -path <features-dir>/<name>.feature
    ScenarioRunner -path <features-dir>/<name>.feature:3:9

    ScenarioRunner -path <features-dir> --plugin html --tags @qa
    ScenarioRunner -path <features-dir> --plugin html --tags @compatibility

Please refer to the Cucumber documentation https://cucumber.io/docs/reference/jvm for more info.

Option (* = required)                  Description                          
---------------------                  -----------                          
-d                                                                          
--glue [location of additional step    (default: net.corda.behave.scenarios)
  definitions, hooks and plugins]                                           
* --path <Path location of .feature                                         
  specifications>                                                           
--plugin [register additional plugins  (default: pretty)                    
  (see https://cucumber.                                                    
  io/docs/reference/jvm)]                                                   
--tags [only run scenarios marked as                                        
  @<tag-name>]  
```
Note: passing in a *-d* option will perform a dry run only (validates the syntax of the scenario but does not execute the code)

Scenario scripts
----------------  
There are currently two sets of scripts:
1. Functional

```bash
# the Cucumber behave test scenario definitions themselves
$ls -l functional/resources/features/functional.feature
# Unix script to easily run these
$ls -l functional/resources/scripts/run-functional.sh
```

2. Interoperability & Compatibility

```bash
# the Cucumber behave test scenario definitions themselves
$ls -l compatibility/resources/features/interoperability.feature
# Unix script to easily run these
$ls -l compatibility/resources/scripts/run-interoperability.sh
```

Note: the complete suite of Compatibility test scenarios have note yet been fully implemented.

You can now run the above test scripts in a number of different ways:

* as a complete suite called a *feature set*:

```bash
# run all scenarios in the functional.feature file
$java -DSTAGING_ROOT=${STAGING_ROOT} -jar ${BEHAVE_JAR} -path functional/resources/features/functional.feature
```

```bash
# run all scenarios in the interoperability.feature file
$java -DSTAGING_ROOT=${STAGING_ROOT} -jar ${BEHAVE_JAR} -path compatibility/resources/features/interoperability.feature
```

* as a set of tagged scenarios within a feature file: 

```bash
# run all scenarios tagged with qa in the interoperability.feature file
$java -DSTAGING_ROOT=${STAGING_ROOT} -jar ${BEHAVE_JAR} -path compatibility/resources/features/interoperability.feature:@qa
```

* as an individual scenario for all example configurations:

```bash
# run the scenario defined on line 5 of the interoperability.feature file
$java -DSTAGING_ROOT=${STAGING_ROOT} -jar ${BEHAVE_JAR} -path compatibility/resources/features/interoperability.feature:5

# by default the above will run as many times as there are parameterised variable definitions

    Examples:
      | R3-Corda-Node-Version        | Corda-Node-Version | Currency |
      | r3-master                    | corda-master       | GBP      |
      | corda-3.0                    | corda-3.1          | GBP      |
      | R3.CORDA-3.0.0-DEV-PREVIEW-3 | corda-3.0          | GBP      |
      | R3.CORDA-3.0.0-DEV-PREVIEW-3 | corda-3.1          | GBP      |
```

* as an individual scenario for a single example configuration you will need to comment out the other example line items:

```bash
# run the scenario defined on line 5 of the interoperability.feature file
$java -DSTAGING_ROOT=${STAGING_ROOT} -jar ${BEHAVE_JAR} -path compatibility/resources/features/interoperability.feature:5

# by default the above will the scenario using *corda-3.0* and *corda-3.1* node versions only:

    Examples:
      | R3-Corda-Node-Version        | Corda-Node-Version | Currency |
#      | r3-master                    | corda-master       | GBP      |
      | corda-3.0                    | corda-3.1          | GBP      |
#      | R3.CORDA-3.0.0-DEV-PREVIEW-3 | corda-3.0          | GBP      |
#      | R3.CORDA-3.0.0-DEV-PREVIEW-3 | corda-3.1          | GBP      |
```