Commit Graph

24 Commits

Author SHA1 Message Date
Razvan Codreanu
d5462a2afe Re enabling persistent volume claims (#5628)
* TM-68 reenabling persistent volume claims using azure files

* TM-68 jenkins stackstracee

* TM-68 removing duplicate volume

* TM-68 pushing storage class yaml file

* TM-68 writing all results to the new persistent volume

* TM-68 fix wrong directory

* TM-68 fix wrong directory

* reapply lost merge commit

* investigate missing POD from test results

* more investigations around pods not executing their tests

* make Pod command line more strict with regards to sub command failure

* make logs an artifact within jenkins

* tidy up command line
2019-10-29 16:23:22 +00:00
Stefano Franz
f9890a5359
PreAllocate pod resources during image build phase (#5587)
* use zulu for jdk
add some parallel groups

* port kubesTest to Java
remove asterix from tests listed by ListTests, instead add after allocation

* attempt to setup unit test builds with correct github integrations

# Conflicts:
#	.ci/dev/unit/Jenkinsfile

* fix issue with github context

* add credentials block

* start pre-allocating pods for builds

* test

* add blocks for reporting build stages

* add logic to preallocate pods during image building

* tidy up Jenkinsfile for unit tests

* add magic command line flag to enable preallocation of pods

* make docker tag deterministic

* fix issue concatenating docker tag inputs

* add build type specific Jenkinsfile

* try new preallocation approach

* make pre-allocation prefix group specific

* force deAllocator to wait for pods to be actually deleted

* revert jenkinsfiles in .ci

* use smarter waiting logic to address review comments

* add --stacktrace to builds to help debugging

* fix issue with closed stream

* add some logging around preallocation

* tidy up by refactoring (de)allocate task generation into method

* change default from 20 pods to 5 pods

* fix issue where docker tag was unstable between building and running tests

* more documentation

* add some infrastructure around setting the log level for a given build

* change preallocation pod duration to 5min

* see if fast enough if using combined unit and integration tests

* disable unit tests

* print out test summaries

* try and make the kubes client a per-use object, rather than a long lived object. This is step one of making GKE use possible

* add log line about what command is executed in the pod
2019-10-28 11:48:04 +00:00
Razvan Codreanu
c5c80033d3 TM-67 converting ImageBuilding from groovy to java (#5609)
* TM-67 converting ImageBuilding from groovy to java

* TM-67 wrong way to set properties

* TM-67 do first does not seem to work

* TM-67 setting credentials first

* TM-67 small logic fix

* TM-67 adding stacktrace

* TM-67 addressing PR comments

* TM-67 fixed normal builds that don't require docker

* TM-67 adding guard rails to the code

* TM-67 removing stacktrace

* TM-67 addressing pr comments
2019-10-18 16:54:09 +01:00
Stefano Franz
22490ecb51
disable pvc creation during k8s build (#5604)
* disable pvc creation

* make testruns available without pvc
2019-10-17 14:40:55 +01:00
Razvan Codreanu
45172515ac TM-41 Ability to resume test runs (#5573)
* TM-41 writing test completions to file to keep track of what was finished. to be used in the case that a pod terminates abruptly

* TM-41 addressing PR comments

* TM-41 addressing PR comments

* TM-41 adding exclusion list to guard against tests being passes as a group

* TM-41 trying to find the jenkins breakpoint

* TM-41 debugging jenkins

* TM-41 revert debugging change

* TM-41 revert debugging changes

* TM-41 revert debugging changes

* TM-41 fixing merge conflicts

* TM-41 now that TM-40 is merged static needs to be updated

* TM-41 refactor constant

* TM-41 fixing jenkins failure

* TM-41 trying new path

* TM-41 moving the file reading to the task that will be executed by the workers as the master does not have a persistent volume

* TM-41 moving the after test as well
2019-10-15 15:14:41 +01:00
Zoltan Kiss
a1dd6abe17 TM-40 Ephemeral workspace for k8s workers that survives restarts (#5567)
* Simplify

* Mount shared dir to worker

* format

* podnames with separators

* refactor parameters

* Use PVC for storage

* pvc in namespace

* KubesTest simplify

* no tolowercase

* no private

* lowercase

* RetryStrategy

* minor changes

* wait forever

* undo .idea

* elvis

* add comment

* regcred

* use correct ConfigBuilder

* delete java, will migrate later

* Revert "delete java, will migrate later"

This reverts commit e3bab1f3

* Merging changes in groovy to new java file

* format

* rename variable

* fix log

* private

* remove bak

* move java files

* Revert "move java files"

This reverts commit 89aa4c35
2019-10-15 10:52:44 +01:00
Stefano Franz
6e98adb085
use zulu for jdk in testing image (#5583)
* use zulu for jdk
add some parallel groups

* port kubesTest to Java
remove asterix from tests listed by ListTests, instead add after allocation
2019-10-14 13:35:22 +01:00
Stefano Franz
298c91ce82
Webook test branch (#5528)
* add ability to index by class OR method

* disable unit tests again

* pending commit

* stream output of build for debugging

* remove git from the base image

* re-enable NodeRPCTests

* add ability to distribute tests based on TeamCity CSV output for test times

* try enabling unit tests again after test distribution has been fixed

* refactor BucketingAllocator to be a class, which bucketingAllocatorTask delegates to.

* finishing touches for improved test distribution

* create new pipelines for builds
2019-10-08 16:33:24 +02:00
Stefano Franz
974c45bb3c
add ability to group test types together (#5459)
* add ability to group test types together

* add ability to specify podCount for use in parallel testing

* remove compiler xml

* add Jenkinsfile to enable scanning

* trigger build

* add ability to specify what docker tag to use from outside of the build

* fix docker work dir

* fix pipeline syntax issues

* use environment rather than `def`

* move agent restrictor outside of stages block

* use steps block

* more pipeline syntax fixes

* even more pipeline syntax fixes

* even more pipeline syntax fixes

* add kubenetize as property to image build

* move clear of docker image to end of build rather than start to prevent colocated builds

* escape dollar on docker image remove command

* attempt to kill all existing jobs

* fix compile issue due to killall_jobs

* fix compile issue due to killall_jobs pt2

* fix spelling

* make all variables environment variables

* add logic to delete images locally after pushing

* wrap testing phase with try / finally so that junit reports are always evaluated

* change the behaviour around post build actions

* break implicit link between testing phase and image building phase, allowing testing to occur without a rebuild and push of image

* prepend registry name to provided tag

* allow tasks to specify whether they wish to stream output from containers

* add timestamps directive to Jenkinsfile to have timing info on output

* make KubesTest resilient against transient pod failures in k8s

* increase CPU request

* add logic to allow specifying container resource requests

* attempt to run unit and integration tests in parallel

* change unit tests to use 3 cores to allow co-location on 8c machines

* join grouped tests together to give pod meaningful name

* add step to renew token with GKE

* change renew step to use pods instead of nodes

* fix bug where memory request is not correctly passed to pod

* disable unit tests for now
2019-09-19 17:41:06 +00:00
Stefano Franz
4ef032071d
Add Jenkinsfile for integration into CI (#5468)
* remove compiler xml

* add Jenkinsfile to enable scanning

* trigger build

* add ability to specify what docker tag to use from outside of the build

* fix docker work dir

* fix pipeline syntax issues

* use environment rather than `def`

* move agent restrictor outside of stages block

* use steps block

* more pipeline syntax fixes

* even more pipeline syntax fixes

* even more pipeline syntax fixes

* add kubenetize as property to image build

* move clear of docker image to end of build rather than start to prevent colocated builds

* escape dollar on docker image remove command

* attempt to kill all existing jobs

* fix compile issue due to killall_jobs

* fix compile issue due to killall_jobs pt2

* fix spelling

* make all variables environment variables
2019-09-16 10:29:58 +00:00
Stefano Franz
d5400e514e
add exception handling to handle situation where builds are tidying up same pods (#5440)
* add asterix to test name

* fix usages of System.hasProperty

* add exception handling to handle condition when Build2 is deleting pods that are being tidied up by Build1

* attempt to fix namespace issues

* add pull task to pull latest base image

* add logic to download dependencies in parallel before running tests
2019-09-06 16:48:49 +00:00
Stefano Franz
14b2882bd7
Improve docker image building to use a stable working directory (#5433)
* add docker working directory parameters to image building
remove gitSha from "stable" buildId, to allow updates to a PR to kill pods

* split up chained null checks  onto separate lines

* fail build if docker image fails to build
force reporting task to run regardless of previous inputs

* add asterix to test name

* fix usages of System.hasProperty
2019-09-05 13:23:41 +00:00
Stefano Franz
a842740c9e
WIP Kubenetes parallel build (#5396)
* Split integration tests

* add simple example of printing all methods annotated with @Test

* add docker plugin to root project
remove docker plugin from child projects
add Dockerfile for image to use when testing
add task to build testing image to root project

* add comment describing proposed testing workflow

* simple attempt at running tests in docker container

* add my first k8s interaction script

* add fabric8 as dependnency to buildSrc

* before adding classpath

* collect reports from containers and run through testReports

* re-enable kubes backed testing

* for each project
1. add a list tests task
2. use this list tests task to modify the included tests
3. add a parallel version of the test task

* tweak logic for downloading test report XML files

* use output of parallel testing tasks in report tasks to determine build resultCode

* prepare for jenkins test

* prepare for jenkins test

* make docker reg password system property

* add logging to print out docker reg creds

* enable docker build

* fix gradle build file

* gather xml files into root project

* change log level for gradle modification

* stop printing gradle docker push passwd

* tidy up report generation

* fix compilation errors

* split signature constraints test into two

* change Sig constraint tests type hierarchy

* tidy up build.gradle

* try method based test includes

* add unit test for test listing

* fix  bug with test slicing

* stop filtering ignored tests to make the numbers match existing runs

* change log level to ensure print out

* move all plugin logic to buildSrc files

* tidy up test modification
add comments to explain what DistributedTesting plugin does

* move new plugins into properly named packages

* tidy up runConfigs

* fix compile errors due to merge with slow-integration-test work

* add system parameter to enable / disable build modification

* add -Dkubenetise to build command

* address review comments

* type safe declaration of parameters in KubesTest
2019-09-03 15:40:08 +00:00
Chris Rankin
ef703c50be
Configure buildSrc to be multi-module. (#2658)
* Configure buildSrc to be multi-module.
* Declare canonicalizer plugin's descriptor via build.gradle.
2018-03-05 12:07:36 +00:00
Tommy Lillehagen
08c4fe2e9e Reformat files in buildSrc 2017-10-09 15:26:39 +01:00
Clinton Alexander
6c79ea675a Removed plugins now published in main repo. 2016-11-18 13:13:14 +00:00
Clinton Alexander
3ab56fa223 Added back gradle config needed by quasar and capsule. 2016-11-04 17:53:37 +00:00
Clinton Alexander
a4a48d06f3 Corda JAR now contains required files. 2016-11-04 17:15:06 +00:00
Clinton Alexander
9a4cc04eb5 Moved source and javadoc publish tasks into a default publish tasks plugin. 2016-09-30 10:57:20 +01:00
Andras Slemmer
1f14fe0705 build: Fix CanonicalizerPlugin dependencies 2016-08-26 12:31:09 +01:00
Andras Slemmer
f783d8466d build: Move definition of CanonicalizerPlugin into buildSrc 2016-08-26 12:11:23 +01:00
Andras Slemmer
8b2211ee0f build: compile config depends on quasar, add comment on local .jar dep 2016-06-17 10:33:13 +01:00
Andras Slemmer
6f70affc6f build: Remove no-op line 2016-06-17 10:32:09 +01:00
Andras Slemmer
ded9cdbcb0 build: add buildSrc/, QuasarPlugin 2016-06-17 10:15:30 +01:00