Commit Graph

37 Commits

Author SHA1 Message Date
Ronan Browne
5e3ee980b3 INFRA-2040: Merging forward updates from 4.7 to 4.8 (2023-07-05) 2023-07-05 20:54:02 +01:00
Ronan Browne
01ead5376b
ES-758: add corda remotes env vars to Jenkins logic ()
* ES-758: add corda remotes

* ES-758: add corda remotes
2023-06-23 20:46:36 +01:00
Ronan Browne
3271978361 INFRA-1697: remove authentication details from gradlew wrapper download and add helper to add them in when ran on CI 2022-03-16 16:41:16 +00:00
Waldemar Zurowski
813c8a72d4 NOTICK: Merging forward updates from OS 4.4 to OS 4.5
Conflicts:
	.ci/dev/nightly-regression/Jenkinsfile
	Jenkinsfile
2021-06-03 12:10:34 +02:00
Waldemar Zurowski
8d994283c6 INFRA-1299: switching from AKS to linear builds on two Jenkins agents
* removed `.ci/dev/unit/Jenkinsfile` and
`.ci/dev/integration/Jenkinsfile` - they don't seem to be used anywhere
* switched mandatory tests to linear build - modeled after Corda
  Enterprise
* switched nightly regression tests to linear build - modeled after
  Corda Enterprise
2021-06-01 16:21:22 +02:00
Waldemar Żurowski
a81e871356
NOTICK: Reverted use of Artifactory cache mechanism () 2020-07-30 05:25:01 +01:00
Waldemar Zurowski
5ee2626530 Empty JUnit results are not allowed 2020-07-21 15:55:56 +01:00
Waldemar Żurowski
309d2ec69f
INFRA-330 Use Artifactory as cache for all dependencies ()
* Use a virtual repo (corda-remotes) containing all Corda repositories with dependencies
* activated when CORDA_USE_CACHE environment variable is set
* Update Jenkins configuration to use new functionality
* it does *not* affect local builds as long as environment variable is not set!
2020-07-17 09:39:45 +01:00
Waldemar Zurowski
f3d8d0d57d Merge branch 'release/os/4.4' into wz/merge-44-45-2020-07-15 2020-07-15 10:24:45 +01:00
Waldemar Żurowski
ef9934ed4e
Added build discarder settings back ()
builds are expired only after 14 days
2020-07-15 10:17:58 +01:00
Waldemar Zurowski
68771cf430 Merge branch 'release/os/4.4' into merge-44-45 2020-06-19 11:02:34 +01:00
Razvan Codreanu
813e9dd242
INFRA-394 enable full junit log to avoid truncation () 2020-06-18 12:52:33 +01:00
Schife
b39aaacd3a Merge branch 'release/os/4.4' of https://github.com/corda/corda into raz/os44to45 2020-06-12 15:35:33 +01:00
Razvan Codreanu
2b7c220522
INFRA-387 preallocation and deployNodes migration to Code Checks ()
* preallocate for slow integration tests as well
2020-06-12 15:30:15 +01:00
Ramzi El-Yafi
40a29869c2 Merge remote-tracking branch 'origin/release/os/4.4' into ramzi/44-45-merge 2020-06-05 15:27:34 +01:00
Ramzi El-Yafi
6ab6099075
[INFRA-368] Add deployNodes and install gradle tasks () 2020-06-03 14:05:12 +01:00
Razvan Codreanu
f4c1119727
INFRA-284 switching from local k8s label () 2020-04-17 15:53:10 +01:00
Razvan Codreanu
4210c0d81f
INFRA-284 switching from local k8s label () 2020-04-17 15:52:47 +01:00
Razvan Codreanu
2ad996c0bd
INFRA-284 Testing out new aks template ()
* INFRA-284 Testing out new aks template
* INFRA-284 switching back to k8s on all files
2020-04-16 07:56:39 +01:00
Dimitris Gounaris
67e3e09a4e
TM-209: Dependx introduction for test running optimization ()
* Add dependx step to build, in monitor state

* Bump plugin version

* Update plugin version

Co-authored-by: Dimitris Gounaris <dimitrisgounaris@20LDN-MAC257.local>
2020-03-11 10:51:38 +00:00
Razvan Codreanu
c6ab07d92c
TM-197 Fix failing test caused by regression and ignoring flaky DistributedServiceTests ()
* TM-197 Setting bouncy castle provider in order for the test to pass

* TM-197 setting timeout for all builds at 3 hours

* TM-197 ignoring unstable tests
2020-02-12 14:26:06 +00:00
Razvan Codreanu
42eca48a02
TM-197 Build stability changes ()
* TM-197 switching 4.3 to use local k8s instances and also make the maximum duration of builds 3 hours, fix 1 test and ignore 2 flaky ones

* update to use local-k8s version of the plugin

Co-authored-by: Stefano Franz <roastario@gmail.com>
2020-02-12 13:58:30 +00:00
Stefano Franz
d95c3e2b8b Use a local instance of k8s instead of remote cluster for parallel testing ()
* switch to new version of plugin

* change plugin version to force refresh

* try using pre-populated cache

* do not preallocate

* no daemon

* reduce cores per fork to increase parallelism

* even more aggresive cpu allocation
2020-01-27 14:58:21 +00:00
Ramzi El-Yafi
ac8a4fc08c TM-170 Generic on-demand-test configuration ()
* Generic on-demand test configuration

* Rename library "magic" string from existing-build-control to
corda-shared-build-pipeline-steps
2020-01-24 11:26:52 +00:00
Ramzi El-Yafi
ffe75db464 TM-170 Generic on-demand-test configuration ()
* Generic on-demand test configuration

* Rename library "magic" string from existing-build-control to
corda-shared-build-pipeline-steps
2020-01-24 10:39:23 +00:00
Stefano Franz
16eb2fce78
Allow taints on kubernetes nodes to determine scheduling of tests ()
* allow a test group to specify taints for the nodes

* specify "effect" of toleration
modify test groups to use taints
modify preallocation to use taints

* add extra command line flags for unit test run
2019-11-25 09:39:37 +00:00
Stefano Franz
138a2f114d
Refresh token by invoking KubeCTL before each Kuberenetes API invocation ()
* fix merge issue with sidecar image work

* add logic to refresh token in GKE

* set streaming to true

* set streaming to true

* set streaming to false, and increment number of workers for regression test as approaching 1h limit

* set PR build to run unit and integration tests

* try method distribution to see if faster

* rename test stage to reflect the fact it runs unit tests

* remove commented out parallel unit test code
2019-11-19 15:01:42 +00:00
Stefano Franz
e09cd84339
add test time publishing to regression test build () 2019-11-04 13:05:13 +00:00
Barry
91e6c9783f TM-51 Read and write test results to artifactory. ()
* TM-51  Prep for reading and writing test results to artifactory.

* TM-51  Tests from target branch if no tests for current branch

* TM-51  Placeholder for test averaging over runs.

* TM-51  Replace slashes in branch names used as tags.

* TM-51  More placeholder work for the mean duration work.

* TM-51  Write out average tests results as as csv.

The csv file should grow and be updated on each run.  This includes whether or not we are running unit tests, integration tests and so on.

* TM-51  Comment out old junit test archiving, add more comments.

* TM-51  Zip task needs to depend on a csv creation task.

If there isn't a csv file present, then the zip task doesn't run due to 'NO-SOURCE'

* TM-51  Zip task should ignore empty dirs

* TM-51  Fix up loading of test results.

We were looking for the wrong artifact name.
Add a bit more logging.

* TM-51  Fix up possible problem with allocating by class distribution.

If we encounter a class we haven't seen before, there won't be any tests.
This means we should give it some weight.  '1' is far too small.

* TM-51  Test that we are definitely increment the run count.

Tracking down whether the zipped csv file should have incremented.

* TM-51  Better default value for missing test/class names.

Begin by using mean unit test duration, but we have the option to bump
that to the mean class unit tests duration.

* TM-51  More debug information around csv writing.

We should be incrementing the tests.

* TM-51  Reload the csv before updating it.

* TM-51  Reduce verbosity of logging.

* TM-51  Reinstate unit tests.  Remove logging verbosity.

* TM-51  Load tests from artifactory in memory and avoid interim file.

* TM-51  Better handling of zero duration tests.

Ensure we return zero times from junit artifacts which may either be zero or have no recorded time.  Before writing the tests duration csv file, store those with a known time, and then store those with zero using the average time.

* TM-51  Log whether we have recorded a test.

Tracking down the curious case where we seem to not be rerunning the
same set of tests on the second run.

* TM-51  Capture junit files as well.

Trying to track down whether some tests are intermittently run.

* TM-51  Change task dependencies to ensure ziptask is triggered.

* TM-51  Remove test assertion, and trigger build

* TM-51  Add corda/enterprise to artifactory tag name.

Moved properties to own file.

* TM-51  Remove unnecessary mean class-based duration.

* TM-51  Add more BucketingAllocator tests.

We need these to nail down its behaviour some more.

* TM-51  Further log information.

We don't seem to be finding the tests in the 'production' runs which is odd.

* TM-51  corda type double set?

* TM-51  do not set the project type in the properties.

SRP and all that.

* TM-51  better plan reporting

* TM-51  duration may be zero

Another runtime problem that doesn't show in tests.

* TM-51  better plan reporting

* fix missing space after image id

* fix merge issue in DistributedTesting

* TM-51  remove unused code when GET/PUT-ting to Artifactory.

* TM-51  put tasks in gradle group and tidy up zip task creation

* TM-51 Fix the junit XML path.

* TM-51 Fix the task graph

* TM-51 Less logging
2019-11-02 09:07:53 +00:00
Razvan Codreanu
d5462a2afe Re enabling persistent volume claims ()
* TM-68 reenabling persistent volume claims using azure files

* TM-68 jenkins stackstracee

* TM-68 removing duplicate volume

* TM-68 pushing storage class yaml file

* TM-68 writing all results to the new persistent volume

* TM-68 fix wrong directory

* TM-68 fix wrong directory

* reapply lost merge commit

* investigate missing POD from test results

* more investigations around pods not executing their tests

* make Pod command line more strict with regards to sub command failure

* make logs an artifact within jenkins

* tidy up command line
2019-10-29 16:23:22 +00:00
Stefano Franz
f9890a5359
PreAllocate pod resources during image build phase ()
* use zulu for jdk
add some parallel groups

* port kubesTest to Java
remove asterix from tests listed by ListTests, instead add after allocation

* attempt to setup unit test builds with correct github integrations

# Conflicts:
#	.ci/dev/unit/Jenkinsfile

* fix issue with github context

* add credentials block

* start pre-allocating pods for builds

* test

* add blocks for reporting build stages

* add logic to preallocate pods during image building

* tidy up Jenkinsfile for unit tests

* add magic command line flag to enable preallocation of pods

* make docker tag deterministic

* fix issue concatenating docker tag inputs

* add build type specific Jenkinsfile

* try new preallocation approach

* make pre-allocation prefix group specific

* force deAllocator to wait for pods to be actually deleted

* revert jenkinsfiles in .ci

* use smarter waiting logic to address review comments

* add --stacktrace to builds to help debugging

* fix issue with closed stream

* add some logging around preallocation

* tidy up by refactoring (de)allocate task generation into method

* change default from 20 pods to 5 pods

* fix issue where docker tag was unstable between building and running tests

* more documentation

* add some infrastructure around setting the log level for a given build

* change preallocation pod duration to 5min

* see if fast enough if using combined unit and integration tests

* disable unit tests

* print out test summaries

* try and make the kubes client a per-use object, rather than a long lived object. This is step one of making GKE use possible

* add log line about what command is executed in the pod
2019-10-28 11:48:04 +00:00
Zoltan Kiss
e6edfe31d9 Short git sha as docker img tag ()
* short git sha as docker img tag

* move subSequence
2019-10-14 15:51:18 +01:00
Stefano Franz
970f60c625 NOTICK: Fix Initiate Flow with Anonymous party ()
* delete buildSrc block configuring multiple plugins

* remove outer stage block

* fix issues around initiateFlow with anonymous party

* code checks

* disable unit tests

* fix flowframeworktest

* undo some extraneous changes
2019-10-14 11:32:22 +01:00
Stefano Franz
c18c3aed95
disable unit tests () 2019-10-11 18:08:36 +01:00
Stefano Franz
298c91ce82
Webook test branch ()
* add ability to index by class OR method

* disable unit tests again

* pending commit

* stream output of build for debugging

* remove git from the base image

* re-enable NodeRPCTests

* add ability to distribute tests based on TeamCity CSV output for test times

* try enabling unit tests again after test distribution has been fixed

* refactor BucketingAllocator to be a class, which bucketingAllocatorTask delegates to.

* finishing touches for improved test distribution

* create new pipelines for builds
2019-10-08 16:33:24 +02:00
Stefano Franz
974c45bb3c
add ability to group test types together ()
* add ability to group test types together

* add ability to specify podCount for use in parallel testing

* remove compiler xml

* add Jenkinsfile to enable scanning

* trigger build

* add ability to specify what docker tag to use from outside of the build

* fix docker work dir

* fix pipeline syntax issues

* use environment rather than `def`

* move agent restrictor outside of stages block

* use steps block

* more pipeline syntax fixes

* even more pipeline syntax fixes

* even more pipeline syntax fixes

* add kubenetize as property to image build

* move clear of docker image to end of build rather than start to prevent colocated builds

* escape dollar on docker image remove command

* attempt to kill all existing jobs

* fix compile issue due to killall_jobs

* fix compile issue due to killall_jobs pt2

* fix spelling

* make all variables environment variables

* add logic to delete images locally after pushing

* wrap testing phase with try / finally so that junit reports are always evaluated

* change the behaviour around post build actions

* break implicit link between testing phase and image building phase, allowing testing to occur without a rebuild and push of image

* prepend registry name to provided tag

* allow tasks to specify whether they wish to stream output from containers

* add timestamps directive to Jenkinsfile to have timing info on output

* make KubesTest resilient against transient pod failures in k8s

* increase CPU request

* add logic to allow specifying container resource requests

* attempt to run unit and integration tests in parallel

* change unit tests to use 3 cores to allow co-location on 8c machines

* join grouped tests together to give pod meaningful name

* add step to renew token with GKE

* change renew step to use pods instead of nodes

* fix bug where memory request is not correctly passed to pod

* disable unit tests for now
2019-09-19 17:41:06 +00:00
Stefano Franz
4ef032071d
Add Jenkinsfile for integration into CI ()
* remove compiler xml

* add Jenkinsfile to enable scanning

* trigger build

* add ability to specify what docker tag to use from outside of the build

* fix docker work dir

* fix pipeline syntax issues

* use environment rather than `def`

* move agent restrictor outside of stages block

* use steps block

* more pipeline syntax fixes

* even more pipeline syntax fixes

* even more pipeline syntax fixes

* add kubenetize as property to image build

* move clear of docker image to end of build rather than start to prevent colocated builds

* escape dollar on docker image remove command

* attempt to kill all existing jobs

* fix compile issue due to killall_jobs

* fix compile issue due to killall_jobs pt2

* fix spelling

* make all variables environment variables
2019-09-16 10:29:58 +00:00