Commit Graph

8002 Commits

Author SHA1 Message Date
Stefano Franz
e09cd84339
add test time publishing to regression test build (#5664) 2019-11-04 13:05:13 +00:00
Stefano Franz
e4e920eee9
multiprocess port allocator is no longer used, so we can remove the tests as they add a significant amount of time to run (2-3 min) (#5663) 2019-11-04 11:56:38 +00:00
Barry
91e6c9783f TM-51 Read and write test results to artifactory. (#5597)
* TM-51  Prep for reading and writing test results to artifactory.

* TM-51  Tests from target branch if no tests for current branch

* TM-51  Placeholder for test averaging over runs.

* TM-51  Replace slashes in branch names used as tags.

* TM-51  More placeholder work for the mean duration work.

* TM-51  Write out average tests results as as csv.

The csv file should grow and be updated on each run.  This includes whether or not we are running unit tests, integration tests and so on.

* TM-51  Comment out old junit test archiving, add more comments.

* TM-51  Zip task needs to depend on a csv creation task.

If there isn't a csv file present, then the zip task doesn't run due to 'NO-SOURCE'

* TM-51  Zip task should ignore empty dirs

* TM-51  Fix up loading of test results.

We were looking for the wrong artifact name.
Add a bit more logging.

* TM-51  Fix up possible problem with allocating by class distribution.

If we encounter a class we haven't seen before, there won't be any tests.
This means we should give it some weight.  '1' is far too small.

* TM-51  Test that we are definitely increment the run count.

Tracking down whether the zipped csv file should have incremented.

* TM-51  Better default value for missing test/class names.

Begin by using mean unit test duration, but we have the option to bump
that to the mean class unit tests duration.

* TM-51  More debug information around csv writing.

We should be incrementing the tests.

* TM-51  Reload the csv before updating it.

* TM-51  Reduce verbosity of logging.

* TM-51  Reinstate unit tests.  Remove logging verbosity.

* TM-51  Load tests from artifactory in memory and avoid interim file.

* TM-51  Better handling of zero duration tests.

Ensure we return zero times from junit artifacts which may either be zero or have no recorded time.  Before writing the tests duration csv file, store those with a known time, and then store those with zero using the average time.

* TM-51  Log whether we have recorded a test.

Tracking down the curious case where we seem to not be rerunning the
same set of tests on the second run.

* TM-51  Capture junit files as well.

Trying to track down whether some tests are intermittently run.

* TM-51  Change task dependencies to ensure ziptask is triggered.

* TM-51  Remove test assertion, and trigger build

* TM-51  Add corda/enterprise to artifactory tag name.

Moved properties to own file.

* TM-51  Remove unnecessary mean class-based duration.

* TM-51  Add more BucketingAllocator tests.

We need these to nail down its behaviour some more.

* TM-51  Further log information.

We don't seem to be finding the tests in the 'production' runs which is odd.

* TM-51  corda type double set?

* TM-51  do not set the project type in the properties.

SRP and all that.

* TM-51  better plan reporting

* TM-51  duration may be zero

Another runtime problem that doesn't show in tests.

* TM-51  better plan reporting

* fix missing space after image id

* fix merge issue in DistributedTesting

* TM-51  remove unused code when GET/PUT-ting to Artifactory.

* TM-51  put tasks in gradle group and tidy up zip task creation

* TM-51 Fix the junit XML path.

* TM-51 Fix the task graph

* TM-51 Less logging
2019-11-02 09:07:53 +00:00
Stefano Franz
cf849fbdbd
Merge pull request #5662 from corda/setup_regression_builds
Setup regression builds in jenkins
2019-11-01 16:51:07 +00:00
stefano
f8b4b334e3 truncate pod name from start rather than end 2019-11-01 16:10:35 +00:00
stefano
d54f2ddd87 fix Library import 2019-11-01 15:57:08 +00:00
stefano
e7e7de1d05 add regression test jenkins file 2019-11-01 15:53:05 +00:00
Stefano Franz
359bb64d69
Merge pull request #5655 from corda/my_merge_branch
Use Kubernetes Jobs rather than Pods to preallocate nodes
2019-11-01 13:11:46 +00:00
snedamle
93ff072812 adding one point to whitelist contract constraints migration - signed… (#5568)
* adding one point to whitelist contract constraints migration - signed CorDapp JAR must be registered with the CZ network operator

* 1. Removing later releases section
2. Changing 4.0 to 4.3

* Changing 4.3 to |corda_version|
2019-11-01 10:49:04 +00:00
Andrius Dagys
d033fceeef CORDA-3365: Reintroduce dependency to fix BFT-Smart notary (#5640)
The commons-codec:1.10 library was removed due to a security vulnerability,
but in commons-codec:1.13 it appears to have been fixed.
2019-11-01 09:05:12 +00:00
James Higgs
e0701231ac [DOCS] Add rows for new database tables in node-database.rst (#5654) 2019-10-31 17:57:52 +00:00
stefano
7a9ee89ded modify watcher to watch jobs rather than pods 2019-10-31 16:51:33 +00:00
stefano
bd9d8dbdbd promote failure to delete to error log level 2019-10-31 15:58:05 +00:00
stefano
547e6d9edd use jobs to preallocate nodes instead of Pods as they support auto delete 2019-10-31 15:57:03 +00:00
Chris Rankin
2895283500 CORDA-3388: Restore mapping of 'java.lang.Void -> void' (#5650) 2019-10-30 17:47:13 +00:00
Stefan Iliev
03ab258fc2 Revert "CORDA-3307 - add support for environment variables in linux (#5523)" (#5643)
This reverts commit c882b221a5.
2019-10-29 17:55:58 +00:00
Razvan Codreanu
d5462a2afe Re enabling persistent volume claims (#5628)
* TM-68 reenabling persistent volume claims using azure files

* TM-68 jenkins stackstracee

* TM-68 removing duplicate volume

* TM-68 pushing storage class yaml file

* TM-68 writing all results to the new persistent volume

* TM-68 fix wrong directory

* TM-68 fix wrong directory

* reapply lost merge commit

* investigate missing POD from test results

* more investigations around pods not executing their tests

* make Pod command line more strict with regards to sub command failure

* make logs an artifact within jenkins

* tidy up command line
2019-10-29 16:23:22 +00:00
Stefano Franz
f9890a5359
PreAllocate pod resources during image build phase (#5587)
* use zulu for jdk
add some parallel groups

* port kubesTest to Java
remove asterix from tests listed by ListTests, instead add after allocation

* attempt to setup unit test builds with correct github integrations

# Conflicts:
#	.ci/dev/unit/Jenkinsfile

* fix issue with github context

* add credentials block

* start pre-allocating pods for builds

* test

* add blocks for reporting build stages

* add logic to preallocate pods during image building

* tidy up Jenkinsfile for unit tests

* add magic command line flag to enable preallocation of pods

* make docker tag deterministic

* fix issue concatenating docker tag inputs

* add build type specific Jenkinsfile

* try new preallocation approach

* make pre-allocation prefix group specific

* force deAllocator to wait for pods to be actually deleted

* revert jenkinsfiles in .ci

* use smarter waiting logic to address review comments

* add --stacktrace to builds to help debugging

* fix issue with closed stream

* add some logging around preallocation

* tidy up by refactoring (de)allocate task generation into method

* change default from 20 pods to 5 pods

* fix issue where docker tag was unstable between building and running tests

* more documentation

* add some infrastructure around setting the log level for a given build

* change preallocation pod duration to 5min

* see if fast enough if using combined unit and integration tests

* disable unit tests

* print out test summaries

* try and make the kubes client a per-use object, rather than a long lived object. This is step one of making GKE use possible

* add log line about what command is executed in the pod
2019-10-28 11:48:04 +00:00
kyriathar
e2836b1106 CORDA-3279 Change single quotes to double quotes fixes node's shutdown (#5611)
A ConfigException$Parse would be thrown at CordaCaplet#parseConfigFile.

com.typesafe.config.ConfigFactory needs ':' to be included in a double quoted and not in a single quoted string.
2019-10-23 10:01:24 +01:00
Christian Sailer
df8cc7282f CORDA-3360 Add nodeInfo permissions to web user 2019-10-23 09:58:59 +01:00
szymonsztuka
b524c6368b CORDA-3335 Corda Shell flow kill - better warning for misformatted flow ID (#5601)
* CORDA-3081 warn that flow ID passed to flow kill is malformed as due to JDK8 doesn't fully validate it (JDK8 bug https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8159339)

* CORDA-3335 Corda Shell flow kill - better warning for misformatted flow ID - exit earlier and don't RPC to node, refactoring for detekt
2019-10-22 11:32:17 +01:00
Ryan Fowler
5da114caa3 CORDA-3281: Drop some errors to warnings and clean up logic around (#5605)
shell "gracefulShutdown" command.
2019-10-22 11:02:04 +01:00
Dimos Raptis
61cdfa5b26 [CORDA-3341] - Add missing liquibase script for hibernate test (#5623) 2019-10-22 09:52:37 +01:00
Stefano Franz
d693a9c1ce
TM-65 re-add local port availability check (#5618)
* re-add local port availability check

* attempt to fix issue with port allocator

* ensure the serversocket is closed
2019-10-21 16:17:17 +01:00
Jonathan Locke
2ea7e6ccae
CORDA-3346: Remove the JitPack repository from Corda (#5620)
CORDA-3346: Remove the JitPack repository from Corda (#5620)
2019-10-21 15:13:42 +01:00
Anthony Keenan
b9a3b3a871
CORDA-3336: Remove log4j plugins to stop errors in config generator (#5617) 2019-10-21 13:31:09 +01:00
Dimos Raptis
bb7c06fa45 [CORDA-3342] - Show proper error message and adjust indentation in shell (#5612) 2019-10-21 13:21:12 +01:00
Chris Rankin
971eb56a98 CORDA-3346: Remove the JitPack repository from Corda. 2019-10-21 13:15:03 +01:00
Stefan Iliev
c882b221a5 CORDA-3307 - add support for environment variables in linux (#5523)
* Added a new way for environment variables to be loaded, which allows for underscore based separation.

* Moved test to its own kotlin file.

* Added case insensitivity support.

* The corda. prefix is now case insensitive too.

* Removed unused variable.

* Added env variables support for driverDSL. Shadowing corda. properties raises an exception.

* Driver api stability fix.

* Changed type of cordapps param to reflect the real one, rather than what IntelliJ auto completed.

* Some detekt issue fixes. Spread operator removed, baselined api stability constructors and buggy line.

* Fixed misspelled variable.

* Reverted unintentional changes.

* Added suppress instead of changing baseline.

* Reworked logic to handle previously defined CORDA_ starting properties and handle accordingly. Fixed a bug where wrong class was used for reflection walking.

* Fix for detekt issues.

* Changed message to a more understandable one.

* Changelog + doc note, console error grammar.

* Changes according to PR review.
2019-10-21 12:01:14 +01:00
Jonathan Locke
292f83ba1d
CORDA-3304: Add test for max attempts on reconnecting rpc (#5608)
CORDA-3304: Add test for max attempts on reconnecting rpc (#5608)
2019-10-21 09:57:44 +01:00
Jonathan Locke
fc9343628b
CORDA-3332: Add rpc reconnection to node explorer (#5613)
CORDA-3332: Add rpc reconnection to node explorer (#5613)
2019-10-18 19:24:31 +01:00
Razvan Codreanu
c5c80033d3 TM-67 converting ImageBuilding from groovy to java (#5609)
* TM-67 converting ImageBuilding from groovy to java

* TM-67 wrong way to set properties

* TM-67 do first does not seem to work

* TM-67 setting credentials first

* TM-67 small logic fix

* TM-67 adding stacktrace

* TM-67 addressing PR comments

* TM-67 fixed normal builds that don't require docker

* TM-67 adding guard rails to the code

* TM-67 removing stacktrace

* TM-67 addressing pr comments
2019-10-18 16:54:09 +01:00
Ryan Fowler
b0cc5f5ca3 CORDA-3332: Add rpc reconnection to node explorer. 2019-10-18 16:25:49 +01:00
Anthony Keenan
49e904b3a2
Fix corda docker image names in docs (#5606) 2019-10-18 10:46:59 +01:00
Jonathan Locke
7d90e305ea
ENT-4119: Make welcome message more accurate (#5596)
[ENT-4119] Make welcome message more accurate
2019-10-18 08:52:16 +01:00
Dimos Raptis
a3adb4816a [CORDA-3304] - Add test for max attempts on reconnecting rpc 2019-10-17 16:27:50 +01:00
Roger Willis
8978512784
Merge pull request #5599 from corda/willh-db-docs
CORDA-3313 Update docs in line with DB changes.
2019-10-17 15:35:36 +01:00
Stefano Franz
22490ecb51
disable pvc creation during k8s build (#5604)
* disable pvc creation

* make testruns available without pvc
2019-10-17 14:40:55 +01:00
Stefano Franz
5bfdf4ce20 CORDA-3257 Docker image: do not post json to filter generated zip for testnet generation (#5598) 2019-10-17 14:31:31 +01:00
Jonathan Locke
18fbd93268
Merge pull request #5585 from corda/CORDA-3304-rpc-max-retries
[CORDA-3304] Introduce max number of retries per invocation for recon…
2019-10-17 11:54:41 +01:00
Jonathan Locke
e9b85a35c6
CORDA-3317 correct docs typo
CORDA-3317 correct docs typo
2019-10-17 08:35:57 +01:00
Will Hester
f85448072a CORDA-3313 formatting 2019-10-16 16:23:32 +01:00
davidrapacchiale
bcb1eb2fe1 Corda-3317 correct docs typo
Removed ??? from "Note: this information is not currently supposed to be
used in production."
2019-10-16 16:15:42 +01:00
Will Hester
312c72d3fb CORDA-3313 Update docs in line with DB changes. 2019-10-16 16:04:57 +01:00
Dimos Raptis
608fdb82f7 [ENT-4119] Make welcome message generic 2019-10-16 15:08:34 +01:00
Jonathan Locke
1dec07f4d1
CORDA-3152: Register custom serializers for jackson as well as amqp
CORDA-3152: Register custom serializers for jackson as well as amqp
2019-10-16 13:08:01 +01:00
Dimos Raptis
f37638c93d [CORDA-3122] - Cleanup non-finalised, errored flows (#5594)
* [CORDA-3122] - Cleanup non-finalised, errored flows

* detekt
2019-10-16 09:37:28 +01:00
Ryan Fowler
bfa460bc07 CORDA-3152: Register custom serializers for jackson as well as amqp 2019-10-15 15:52:31 +01:00
Razvan Codreanu
ee09cd8762 TM-45 Make detektBaseline pass (#5561)
* TM-45 make the baseline generating task show a successful build regardless of the existing detekt violations

* TM-45 address PR feedback
2019-10-15 15:49:31 +01:00
Razvan Codreanu
45172515ac TM-41 Ability to resume test runs (#5573)
* TM-41 writing test completions to file to keep track of what was finished. to be used in the case that a pod terminates abruptly

* TM-41 addressing PR comments

* TM-41 addressing PR comments

* TM-41 adding exclusion list to guard against tests being passes as a group

* TM-41 trying to find the jenkins breakpoint

* TM-41 debugging jenkins

* TM-41 revert debugging change

* TM-41 revert debugging changes

* TM-41 revert debugging changes

* TM-41 fixing merge conflicts

* TM-41 now that TM-40 is merged static needs to be updated

* TM-41 refactor constant

* TM-41 fixing jenkins failure

* TM-41 trying new path

* TM-41 moving the file reading to the task that will be executed by the workers as the master does not have a persistent volume

* TM-41 moving the after test as well
2019-10-15 15:14:41 +01:00