Commit Graph

477 Commits

Author SHA1 Message Date
Ramzi El-Yafi
575df97c52 Mark integration test tasks with "big" node taint 2019-12-13 10:08:51 +00:00
Razvan Codreanu
d33dbb2ea9 TM-104 Switching the distributed testing plugin to the one released in artifactory (#5764)
* TM-104 switch to using the published plugin

* TM-104 switching to artifactory plugin

* TM-104 remove unused plugin

* TM-104 adding docker plugin

* TM-104 adding docker plugin take 2

* add dependencies-dev and set distributed build plugin to changing
2019-11-28 14:49:39 +00:00
Stefano Franz
16eb2fce78
Allow taints on kubernetes nodes to determine scheduling of tests (#5755)
* allow a test group to specify taints for the nodes

* specify "effect" of toleration
modify test groups to use taints
modify preallocation to use taints

* add extra command line flags for unit test run
2019-11-25 09:39:37 +00:00
Stefano Franz
138a2f114d
Refresh token by invoking KubeCTL before each Kuberenetes API invocation (#5732)
* fix merge issue with sidecar image work

* add logic to refresh token in GKE

* set streaming to true

* set streaming to true

* set streaming to false, and increment number of workers for regression test as approaching 1h limit

* set PR build to run unit and integration tests

* try method distribution to see if faster

* rename test stage to reflect the fact it runs unit tests

* remove commented out parallel unit test code
2019-11-19 15:01:42 +00:00
Razvan Codreanu
7106b0f1b9 Converting Groovy to Java in the Distributed testing plugin (#5642)
* TM-74 Converting ParallelTestGroup

* TM-74 private fields, getters, distributedTesting refactoring

* TM-74 More conversions

* TM-74 reinstating ListTests groovy class

* TM-74 extracting enum and clas from ListTests

* TM-74 switching to java ListTests

* TM-74 null check

* TM-74 new access modifiers

* TM-74 minor changes

* TM-74 scrapping the conversions of ListTests to java

* TM-74 reverting build.gradle

* TM-74 formatting main gradle

* TM-74 removing the commented out code. It has been linked in the jira

* TM-74 adding list tests back for investigation

* TM-74 collecting after every operation

* TM-74 collecting after every operation

* TM-74 more debugging

* TM-74 more debugging with lifecycle now

* TM-74 debugging on the existing groovy file

* TM-74 with stream output

* TM-74 switching back to java

* TM-74 reverting java code, fixing cast in distributed testing file?

* TM-74 nailing down the line causing the problem

* TM-74 casting

* TM-74 changing types

* TM-74 stacktrace at pod lvl

* TM-74 stacktrace at pod lvl

* TM-74 fix issue with immutable list

* TM-74 reverting changes now that fix is in place

* TM-74 switching to generic task

* TM-74 turning off debugging

* TM-74 allocating by method
2019-11-05 16:59:19 +00:00
Stefano Franz
f9890a5359
PreAllocate pod resources during image build phase (#5587)
* use zulu for jdk
add some parallel groups

* port kubesTest to Java
remove asterix from tests listed by ListTests, instead add after allocation

* attempt to setup unit test builds with correct github integrations

# Conflicts:
#	.ci/dev/unit/Jenkinsfile

* fix issue with github context

* add credentials block

* start pre-allocating pods for builds

* test

* add blocks for reporting build stages

* add logic to preallocate pods during image building

* tidy up Jenkinsfile for unit tests

* add magic command line flag to enable preallocation of pods

* make docker tag deterministic

* fix issue concatenating docker tag inputs

* add build type specific Jenkinsfile

* try new preallocation approach

* make pre-allocation prefix group specific

* force deAllocator to wait for pods to be actually deleted

* revert jenkinsfiles in .ci

* use smarter waiting logic to address review comments

* add --stacktrace to builds to help debugging

* fix issue with closed stream

* add some logging around preallocation

* tidy up by refactoring (de)allocate task generation into method

* change default from 20 pods to 5 pods

* fix issue where docker tag was unstable between building and running tests

* more documentation

* add some infrastructure around setting the log level for a given build

* change preallocation pod duration to 5min

* see if fast enough if using combined unit and integration tests

* disable unit tests

* print out test summaries

* try and make the kubes client a per-use object, rather than a long lived object. This is step one of making GKE use possible

* add log line about what command is executed in the pod
2019-10-28 11:48:04 +00:00
Chris Rankin
971eb56a98 CORDA-3346: Remove the JitPack repository from Corda. 2019-10-21 13:15:03 +01:00
Razvan Codreanu
ee09cd8762 TM-45 Make detektBaseline pass (#5561)
* TM-45 make the baseline generating task show a successful build regardless of the existing detekt violations

* TM-45 address PR feedback
2019-10-15 15:49:31 +01:00
Stefano Franz
6e98adb085
use zulu for jdk in testing image (#5583)
* use zulu for jdk
add some parallel groups

* port kubesTest to Java
remove asterix from tests listed by ListTests, instead add after allocation
2019-10-14 13:35:22 +01:00
Stefano Franz
298c91ce82
Webook test branch (#5528)
* add ability to index by class OR method

* disable unit tests again

* pending commit

* stream output of build for debugging

* remove git from the base image

* re-enable NodeRPCTests

* add ability to distribute tests based on TeamCity CSV output for test times

* try enabling unit tests again after test distribution has been fixed

* refactor BucketingAllocator to be a class, which bucketingAllocatorTask delegates to.

* finishing touches for improved test distribution

* create new pipelines for builds
2019-10-08 16:33:24 +02:00
josecoll
246ec8766e
CORDA-3224 JDK11 built and published artifacts to include classifier. (#5488)
* JDK11 built and published artifacts to include classifier.

* Added additional non-intuitive gradle dust sprinkled recommended by CR.

* Added additional non-intuitive gradle dust sprinkled recommended by CR.

* Bump corda gradle plugins version.

* Incorporating PR review feedback.

* jdk11Classifier
2019-10-08 10:45:10 +01:00
josecoll
42e104ca4f
Publish checkpoint agent jar and allow for inclusion of version id in jar upon run-time execution. (#5553) 2019-10-04 14:28:42 +01:00
Tudor Malene
298d8ba69c CORDA-3024 Rename the webserver (#5489) 2019-09-26 10:20:49 +01:00
josecoll
6666ee1a46
Set JFX 3rd party library dependency (fontawesomefx) according to Java version. (#5513) 2019-09-24 12:09:01 +01:00
Jonathan Locke
d5d313148d
NOTIK Downgrade Dokka back to 0.9.17 due to failing docs_builder
NOTIK Downgrade Dokka back to 0.9.17 due to failing docs_builder
2019-09-19 20:34:29 +01:00
Stefano Franz
974c45bb3c
add ability to group test types together (#5459)
* add ability to group test types together

* add ability to specify podCount for use in parallel testing

* remove compiler xml

* add Jenkinsfile to enable scanning

* trigger build

* add ability to specify what docker tag to use from outside of the build

* fix docker work dir

* fix pipeline syntax issues

* use environment rather than `def`

* move agent restrictor outside of stages block

* use steps block

* more pipeline syntax fixes

* even more pipeline syntax fixes

* even more pipeline syntax fixes

* add kubenetize as property to image build

* move clear of docker image to end of build rather than start to prevent colocated builds

* escape dollar on docker image remove command

* attempt to kill all existing jobs

* fix compile issue due to killall_jobs

* fix compile issue due to killall_jobs pt2

* fix spelling

* make all variables environment variables

* add logic to delete images locally after pushing

* wrap testing phase with try / finally so that junit reports are always evaluated

* change the behaviour around post build actions

* break implicit link between testing phase and image building phase, allowing testing to occur without a rebuild and push of image

* prepend registry name to provided tag

* allow tasks to specify whether they wish to stream output from containers

* add timestamps directive to Jenkinsfile to have timing info on output

* make KubesTest resilient against transient pod failures in k8s

* increase CPU request

* add logic to allow specifying container resource requests

* attempt to run unit and integration tests in parallel

* change unit tests to use 3 cores to allow co-location on 8c machines

* join grouped tests together to give pod meaningful name

* add step to renew token with GKE

* change renew step to use pods instead of nodes

* fix bug where memory request is not correctly passed to pod

* disable unit tests for now
2019-09-19 17:41:06 +00:00
josecoll
94a1d357ec Downgrade Dokka back to 0.9.17 due to failing docs_builder. 2019-09-19 17:34:53 +01:00
Zoltan Kiss
7bbbc71fdf TM-23 compileAll task to compile all code (#5490)
* Add simple compileAll task to be used by warning check

* lazy configure compileAll
2019-09-19 10:50:29 +01:00
josecoll
3fafbe551c
CORDA-2050 Upgrade Corda to Java 11 (compatibility mode) (#5356)
Upgrade Corda to run with Java 11 (compatibility mode) - see https://github.com/corda/corda/pull/5356
2019-09-18 10:26:26 +01:00
Schife
3b1a49a2cf TM-20 rebasing the detekt changes to be able to merge into OS 4.3. The changes include: detekt integration, rule configurations, baseline of the current issues that exist in 4.3 and a MaxLineLength rule violation fix to ANSIProgress test since it was causing the baseline to fail to load due to the special characters in the test (#5464) 2019-09-12 14:08:50 +01:00
Zoltan Kiss
f171de7b69 TM-23 Fail build on compiler warnings (#5453)
* java compile respects compilation.allWarningsAsErrors

* suppress or cleanup warnings

* suppress warning

* use non-deprecated kotlin dependency

* rename property

* handle property existence check

* Deal with warnings
2019-09-11 16:34:51 +01:00
Stefano Franz
a842740c9e
WIP Kubenetes parallel build (#5396)
* Split integration tests

* add simple example of printing all methods annotated with @Test

* add docker plugin to root project
remove docker plugin from child projects
add Dockerfile for image to use when testing
add task to build testing image to root project

* add comment describing proposed testing workflow

* simple attempt at running tests in docker container

* add my first k8s interaction script

* add fabric8 as dependnency to buildSrc

* before adding classpath

* collect reports from containers and run through testReports

* re-enable kubes backed testing

* for each project
1. add a list tests task
2. use this list tests task to modify the included tests
3. add a parallel version of the test task

* tweak logic for downloading test report XML files

* use output of parallel testing tasks in report tasks to determine build resultCode

* prepare for jenkins test

* prepare for jenkins test

* make docker reg password system property

* add logging to print out docker reg creds

* enable docker build

* fix gradle build file

* gather xml files into root project

* change log level for gradle modification

* stop printing gradle docker push passwd

* tidy up report generation

* fix compilation errors

* split signature constraints test into two

* change Sig constraint tests type hierarchy

* tidy up build.gradle

* try method based test includes

* add unit test for test listing

* fix  bug with test slicing

* stop filtering ignored tests to make the numbers match existing runs

* change log level to ensure print out

* move all plugin logic to buildSrc files

* tidy up test modification
add comments to explain what DistributedTesting plugin does

* move new plugins into properly named packages

* tidy up runConfigs

* fix compile errors due to merge with slow-integration-test work

* add system parameter to enable / disable build modification

* add -Dkubenetise to build command

* address review comments

* type safe declaration of parameters in KubesTest
2019-09-03 15:40:08 +00:00
James Brown
b8e278680d CORDA-3120 Update owasp scanner (#5335) 2019-08-08 16:05:54 +01:00
Stefano Franz
88894bc592
add a shared memory port allocator to allow multiple processes to sha… (#5223)
* add a shared memory port allocator to allow multiple processes to share a single allocation pool

* remove dangerous reset function on port allocator

* set forkCount = 2 in node integration test

* only allow one build of a cordapp at any given time for Driver tests

* make all portallocation requests use same starting point

* globally set forks to 6

* tweak forking parameters to allow parallel builds

* tweak unit test parallelism

* 2 workers for integrationTest

* some more tweaks for parallel builds

* some more tweaks for parallel builds

* seems that 49K is not the start of ephemeral ports on all kernels

* tweak parallel settings

* try fix RPC shutdown test in parallel env

* add some logging for RPC shutdown test

* added some logging around PortAllocation tests - try figure out where they are getting stuck

* added some logging around PortAllocation tests - try figure out where they are getting stuck

* fix api-scanner tests

* minimize api changes

* revert to complying with existing API

* add the AtomicInteger for api compatibility reasons

* make sizing script executable

* address review comments pt1

* address review comments pt2

* fix compile errors after review comments

* return to using home dir as temp dir seemed to interact badly with gradle
2019-07-02 18:38:33 +00:00
szymonsztuka
f89008c070 CORDA-2998 fix network builder (#5265)
CORDA-2998 fix network builder
2019-07-02 11:32:48 +01:00
Dimos Raptis
5db70192ec [CORDA-3042] - Revert upgrade of dokka (#5259) 2019-06-27 18:28:12 +01:00
Dan Newton
5e48ad0f52 Upgrade jackson_version to 2.9.7 (#5261)
`jackson-module-kotlin` version `2.9.5` was causing runtime errors.
Updating to `2.9.7` resolved these errors. The dependency was not
updated to a more recent version as they require kotlin 1.3.
2019-06-27 16:48:35 +01:00
Shams Asari
4e835d5483
CORDA-2333: Updated the majority of the dependencies that were out of date (#5235)
Some where not touched, including the crypto ones, libraries related to messaging and Kotlin.
2019-06-20 16:19:16 +01:00
Dominic Fox
f01f8a129e ENT-3444 create test-db module (#5093)
* ENT-3444 define RequiresDB annotation and junit5 extension

* Move to internal

* info to trace

* Limit exposure of gradle imports

* Enable annotation inheritance, and multiple SQL scripts per class or method

* Get the test context class globally for all groups

* usingRemoteDatabase flag
2019-06-18 10:57:20 +01:00
Shams Asari
df19b444dd
CORDA-2333: Reverting jersey and mockito as it currently causes issues with ENT (#5228) 2019-06-17 19:27:35 +01:00
Shams Asari
bb3997ed5b
CORDA-2333: Reverting ClassGraph version back to 4.6.12 (#5221)
4.8.38 not working with the changes from 2152961d
2019-06-13 17:22:09 +01:00
Shams Asari
2152961d46
CORDA-2333: Dependency update pass for tests and demos (#5211) 2019-06-13 10:15:13 +01:00
Shams Asari
8480d49e7e
CORDA-2333: Bumped ClassGraph version to latest (#5214) 2019-06-12 13:27:51 +01:00
josecoll
8678bad88d
CORDA-2970 Revert usage of Gradle JUnit 5 Platform Runner (#5173)
* Revert usage of Gradle 5 useJUnitPlatform() which causes significant test execution performance degradation.

* Remove completely top-level Java CordaCaplet unit tests causing consistent TC failure on Azure VMs:

 java.security.NoSuchAlgorithmException: no such algorithm: NONEwithEdDSA for provider BC
2019-06-05 10:17:02 +01:00
James Brown
24c8a5a2dd ENT-3540 Reverting jackson - kotlin runtime issue (#5133)
Jackson was updated to 2.9.8, however this version has a transitive dependency on kotlin 1.3.10 which is incompatible with corda (which uses 1.2 currently).
2019-06-05 09:04:13 +01:00
josecoll
06db5973d5
Reduce test execution times by explicitly configure quasar package exclusions (#5161)
* Reduce test execution times by explicitly configure quasar package exclusions (using new quasar util extension introduced in Corda gradle plugins 5.0.1-SNAPSHOT).

* Remove SNAPSHOT from Corda gradle plugins version identifier.

* Incorporating PR review feedback from CR.

* Minor consolidation clean-up.

* Minor consolidation clean-up.

* Minor consolidation clean-up.
2019-05-30 10:09:11 +01:00
Anthony Keenan
b06fa62860
Update OWASP dependency checker to v4.0.2 to fix clash with Gradle 5 upgrade (#5177) 2019-05-29 10:43:32 +00:00
josecoll
0263650a5f
Revert "Merge pull request #5145 from corda/jdk11-migration-gradle5-upgrade" (#5146)
This reverts commit f46377556c, reversing
changes made to ba675e73b9.
2019-05-20 16:16:49 +01:00
josecoll
84ca6405c2 Rebase from OS master to incorporate upgrade to Gradle 5.2.1 2019-05-20 15:09:36 +01:00
josecoll
d2f2f4aaca Display JAVA_HOME 2019-05-20 15:09:36 +01:00
josecoll
b03b5133f6 Added configurable flag to continue on test failure so TC can perform complete test execution sweep. 2019-05-20 15:09:34 +01:00
josecoll
9183e401c3 Upgraded Mockito and targetCompatibility to 11 (REVISIT). 2019-05-20 15:08:27 +01:00
josecoll
8d4816f203 Allow corda gradle plugin snapshot version resolution from artifactory 'corda-dev' 2019-05-20 15:08:27 +01:00
josecoll
8c921cfe71 Upgrade jacoco to JDK11 supported version 2019-05-20 15:08:27 +01:00
josecoll
a56ef580b2 Temp remove usage of java modularity plugin. 2019-05-20 15:08:27 +01:00
josecoll
57fb63940f Include Gradle JPMS plugin (v1.5.0) 2019-05-20 15:08:27 +01:00
josecoll
ba675e73b9
CORDA-2893 Upgrade Corda to use Gradle 5.4.1 (Take 2) (#5137)
Upgrade Corda to use Gradle 5.4.1
2019-05-20 14:26:42 +01:00
Chris Rankin
b9f7c1a08a CORDA-2893, ENT-3422: Tweak JUnit 5 configurations to keep vintage engine off compile classpaths. (#5138)
* Tweak JUnit 5 configurations to keep vintage engine off compile classpaths.

* Configure Gradle tests to use JUnit 5.
2019-05-20 11:57:56 +01:00
josecoll
0eab42b63e
Revert "CORDA-2893 Upgrade Corda to use Gradle 5.x (#5063)" (#5135)
This reverts commit 88babefd89.
2019-05-17 11:11:05 +01:00
josecoll
88babefd89
CORDA-2893 Upgrade Corda to use Gradle 5.x (#5063)
* Upgrade gradle wrapper to Gradle 5.0

* Upgrade to use locally deployed version of Capsule plugin (using Gradle 5.0)

* Upgrade to use Corda Gradle Plugins 5.x (inclusive of Gradle 5.0 compatibility fixes)

* Fix compile-time problems resolving log4j packages.

* Update to use Artifactory deployed version of Capsule plugin (using Gradle 5.0)

* Upgrade to use Gradle 4.2.1 (excluding default memory setting change: command line client now starts with 64MB of heap instead of 1GB.)

* Added explicit compile-time dependency on 'de.jensd:fontawesomefx-commons:8.13' (compile-time error in tools:explorer without)

* Update "wrapper" gradleVersion to 5.4.1

* Update Capsule plugin version label to reference R3 forked build.

* Commit all gradle upgrade changes following: ./gradlew wrapper --gradle-version 5.4.1

* Increase maximum heap memory for Test tasks to 1GB, and upgrade build-scan plugin to 2.2.1.

* Increase Test tasks' maximum memory to 1.5GB - what could go wrong?

* Up maxHeapSize to 2g for gradle test runner (global setting).

* Added explicit compile-time dependency on 'de.jensd:fontawesomefx-commons:8.13' (compile-time error in tools:demobench without)

* Added forkEvery for node Unit tests.

* Up :node test task 'forkEvery' to 10.

* TC test execution tuning (:core, :node => forkEvery 10, default JVM heap size)

* TC test execution tuning (bump JVM heap size up to 1g for :node)

* TC test execution tuning (bump JVM heap size up to 1.5g for :node)

* TC test execution tuning (re-instate global JVM heap size of 1Gb)

* TC test execution tuning (re-instate JVM heap size of 2Gb for :node)

* Update Corda Gradle plugins to 5.0.0

* Updated plugin resolution order + renamed artifactory URL to use "software.r3.com"

* Reorder plugin resolution such that mavenlocal() is always queried first.
2019-05-17 10:36:21 +01:00