corda/Jenkinsfile

67 lines
2.5 KiB
Plaintext
Raw Normal View History

import static com.r3.build.BuildControl.killAllExistingBuildsForJob
@Library('existing-build-control')
import static com.r3.build.BuildControl.killAllExistingBuildsForJob
killAllExistingBuildsForJob(env.JOB_NAME, env.BUILD_NUMBER.toInteger())
pipeline {
agent { label 'k8s' }
add ability to group test types together (#5459) * add ability to group test types together * add ability to specify podCount for use in parallel testing * remove compiler xml * add Jenkinsfile to enable scanning * trigger build * add ability to specify what docker tag to use from outside of the build * fix docker work dir * fix pipeline syntax issues * use environment rather than `def` * move agent restrictor outside of stages block * use steps block * more pipeline syntax fixes * even more pipeline syntax fixes * even more pipeline syntax fixes * add kubenetize as property to image build * move clear of docker image to end of build rather than start to prevent colocated builds * escape dollar on docker image remove command * attempt to kill all existing jobs * fix compile issue due to killall_jobs * fix compile issue due to killall_jobs pt2 * fix spelling * make all variables environment variables * add logic to delete images locally after pushing * wrap testing phase with try / finally so that junit reports are always evaluated * change the behaviour around post build actions * break implicit link between testing phase and image building phase, allowing testing to occur without a rebuild and push of image * prepend registry name to provided tag * allow tasks to specify whether they wish to stream output from containers * add timestamps directive to Jenkinsfile to have timing info on output * make KubesTest resilient against transient pod failures in k8s * increase CPU request * add logic to allow specifying container resource requests * attempt to run unit and integration tests in parallel * change unit tests to use 3 cores to allow co-location on 8c machines * join grouped tests together to give pod meaningful name * add step to renew token with GKE * change renew step to use pods instead of nodes * fix bug where memory request is not correctly passed to pod * disable unit tests for now
2019-09-19 17:41:06 +00:00
options { timestamps() }
environment {
DOCKER_TAG_TO_USE = "${env.GIT_COMMIT.subSequence(0, 8)}"
EXECUTOR_NUMBER = "${env.EXECUTOR_NUMBER}"
BUILD_ID = "${env.BUILD_ID}-${env.JOB_NAME}"
}
stages {
add ability to group test types together (#5459) * add ability to group test types together * add ability to specify podCount for use in parallel testing * remove compiler xml * add Jenkinsfile to enable scanning * trigger build * add ability to specify what docker tag to use from outside of the build * fix docker work dir * fix pipeline syntax issues * use environment rather than `def` * move agent restrictor outside of stages block * use steps block * more pipeline syntax fixes * even more pipeline syntax fixes * even more pipeline syntax fixes * add kubenetize as property to image build * move clear of docker image to end of build rather than start to prevent colocated builds * escape dollar on docker image remove command * attempt to kill all existing jobs * fix compile issue due to killall_jobs * fix compile issue due to killall_jobs pt2 * fix spelling * make all variables environment variables * add logic to delete images locally after pushing * wrap testing phase with try / finally so that junit reports are always evaluated * change the behaviour around post build actions * break implicit link between testing phase and image building phase, allowing testing to occur without a rebuild and push of image * prepend registry name to provided tag * allow tasks to specify whether they wish to stream output from containers * add timestamps directive to Jenkinsfile to have timing info on output * make KubesTest resilient against transient pod failures in k8s * increase CPU request * add logic to allow specifying container resource requests * attempt to run unit and integration tests in parallel * change unit tests to use 3 cores to allow co-location on 8c machines * join grouped tests together to give pod meaningful name * add step to renew token with GKE * change renew step to use pods instead of nodes * fix bug where memory request is not correctly passed to pod * disable unit tests for now
2019-09-19 17:41:06 +00:00
stage('Corda Pull Request - Generate Build Image') {
steps {
withCredentials([string(credentialsId: 'container_reg_passwd', variable: 'DOCKER_PUSH_PWD')]) {
sh "./gradlew " +
"-Dkubenetize=true " +
"-Ddocker.push.password=\"\${DOCKER_PUSH_PWD}\" " +
"-Ddocker.work.dir=\"/tmp/\${EXECUTOR_NUMBER}\" " +
PreAllocate pod resources during image build phase (#5587) * use zulu for jdk add some parallel groups * port kubesTest to Java remove asterix from tests listed by ListTests, instead add after allocation * attempt to setup unit test builds with correct github integrations # Conflicts: # .ci/dev/unit/Jenkinsfile * fix issue with github context * add credentials block * start pre-allocating pods for builds * test * add blocks for reporting build stages * add logic to preallocate pods during image building * tidy up Jenkinsfile for unit tests * add magic command line flag to enable preallocation of pods * make docker tag deterministic * fix issue concatenating docker tag inputs * add build type specific Jenkinsfile * try new preallocation approach * make pre-allocation prefix group specific * force deAllocator to wait for pods to be actually deleted * revert jenkinsfiles in .ci * use smarter waiting logic to address review comments * add --stacktrace to builds to help debugging * fix issue with closed stream * add some logging around preallocation * tidy up by refactoring (de)allocate task generation into method * change default from 20 pods to 5 pods * fix issue where docker tag was unstable between building and running tests * more documentation * add some infrastructure around setting the log level for a given build * change preallocation pod duration to 5min * see if fast enough if using combined unit and integration tests * disable unit tests * print out test summaries * try and make the kubes client a per-use object, rather than a long lived object. This is step one of making GKE use possible * add log line about what command is executed in the pod
2019-10-28 11:48:04 +00:00
"-Ddocker.build.tag=\"\${DOCKER_TAG_TO_USE}\"" +
" clean pushBuildImage preAllocateForAllParallelIntegrationTest --stacktrace"
}
add ability to group test types together (#5459) * add ability to group test types together * add ability to specify podCount for use in parallel testing * remove compiler xml * add Jenkinsfile to enable scanning * trigger build * add ability to specify what docker tag to use from outside of the build * fix docker work dir * fix pipeline syntax issues * use environment rather than `def` * move agent restrictor outside of stages block * use steps block * more pipeline syntax fixes * even more pipeline syntax fixes * even more pipeline syntax fixes * add kubenetize as property to image build * move clear of docker image to end of build rather than start to prevent colocated builds * escape dollar on docker image remove command * attempt to kill all existing jobs * fix compile issue due to killall_jobs * fix compile issue due to killall_jobs pt2 * fix spelling * make all variables environment variables * add logic to delete images locally after pushing * wrap testing phase with try / finally so that junit reports are always evaluated * change the behaviour around post build actions * break implicit link between testing phase and image building phase, allowing testing to occur without a rebuild and push of image * prepend registry name to provided tag * allow tasks to specify whether they wish to stream output from containers * add timestamps directive to Jenkinsfile to have timing info on output * make KubesTest resilient against transient pod failures in k8s * increase CPU request * add logic to allow specifying container resource requests * attempt to run unit and integration tests in parallel * change unit tests to use 3 cores to allow co-location on 8c machines * join grouped tests together to give pod meaningful name * add step to renew token with GKE * change renew step to use pods instead of nodes * fix bug where memory request is not correctly passed to pod * disable unit tests for now
2019-09-19 17:41:06 +00:00
sh "kubectl auth can-i get pods"
}
}
add ability to group test types together (#5459) * add ability to group test types together * add ability to specify podCount for use in parallel testing * remove compiler xml * add Jenkinsfile to enable scanning * trigger build * add ability to specify what docker tag to use from outside of the build * fix docker work dir * fix pipeline syntax issues * use environment rather than `def` * move agent restrictor outside of stages block * use steps block * more pipeline syntax fixes * even more pipeline syntax fixes * even more pipeline syntax fixes * add kubenetize as property to image build * move clear of docker image to end of build rather than start to prevent colocated builds * escape dollar on docker image remove command * attempt to kill all existing jobs * fix compile issue due to killall_jobs * fix compile issue due to killall_jobs pt2 * fix spelling * make all variables environment variables * add logic to delete images locally after pushing * wrap testing phase with try / finally so that junit reports are always evaluated * change the behaviour around post build actions * break implicit link between testing phase and image building phase, allowing testing to occur without a rebuild and push of image * prepend registry name to provided tag * allow tasks to specify whether they wish to stream output from containers * add timestamps directive to Jenkinsfile to have timing info on output * make KubesTest resilient against transient pod failures in k8s * increase CPU request * add logic to allow specifying container resource requests * attempt to run unit and integration tests in parallel * change unit tests to use 3 cores to allow co-location on 8c machines * join grouped tests together to give pod meaningful name * add step to renew token with GKE * change renew step to use pods instead of nodes * fix bug where memory request is not correctly passed to pod * disable unit tests for now
2019-09-19 17:41:06 +00:00
stage('Corda Pull Request - Run Tests') {
parallel {
stage('Integration Tests') {
steps {
sh "./gradlew " +
"-DbuildId=\"\${BUILD_ID}\" " +
"-Dkubenetize=true " +
PreAllocate pod resources during image build phase (#5587) * use zulu for jdk add some parallel groups * port kubesTest to Java remove asterix from tests listed by ListTests, instead add after allocation * attempt to setup unit test builds with correct github integrations # Conflicts: # .ci/dev/unit/Jenkinsfile * fix issue with github context * add credentials block * start pre-allocating pods for builds * test * add blocks for reporting build stages * add logic to preallocate pods during image building * tidy up Jenkinsfile for unit tests * add magic command line flag to enable preallocation of pods * make docker tag deterministic * fix issue concatenating docker tag inputs * add build type specific Jenkinsfile * try new preallocation approach * make pre-allocation prefix group specific * force deAllocator to wait for pods to be actually deleted * revert jenkinsfiles in .ci * use smarter waiting logic to address review comments * add --stacktrace to builds to help debugging * fix issue with closed stream * add some logging around preallocation * tidy up by refactoring (de)allocate task generation into method * change default from 20 pods to 5 pods * fix issue where docker tag was unstable between building and running tests * more documentation * add some infrastructure around setting the log level for a given build * change preallocation pod duration to 5min * see if fast enough if using combined unit and integration tests * disable unit tests * print out test summaries * try and make the kubes client a per-use object, rather than a long lived object. This is step one of making GKE use possible * add log line about what command is executed in the pod
2019-10-28 11:48:04 +00:00
"-Ddocker.run.tag=\"\${DOCKER_TAG_TO_USE}\"" +
" deAllocateForAllParallelIntegrationTest allParallelIntegrationTest --stacktrace"
add ability to group test types together (#5459) * add ability to group test types together * add ability to specify podCount for use in parallel testing * remove compiler xml * add Jenkinsfile to enable scanning * trigger build * add ability to specify what docker tag to use from outside of the build * fix docker work dir * fix pipeline syntax issues * use environment rather than `def` * move agent restrictor outside of stages block * use steps block * more pipeline syntax fixes * even more pipeline syntax fixes * even more pipeline syntax fixes * add kubenetize as property to image build * move clear of docker image to end of build rather than start to prevent colocated builds * escape dollar on docker image remove command * attempt to kill all existing jobs * fix compile issue due to killall_jobs * fix compile issue due to killall_jobs pt2 * fix spelling * make all variables environment variables * add logic to delete images locally after pushing * wrap testing phase with try / finally so that junit reports are always evaluated * change the behaviour around post build actions * break implicit link between testing phase and image building phase, allowing testing to occur without a rebuild and push of image * prepend registry name to provided tag * allow tasks to specify whether they wish to stream output from containers * add timestamps directive to Jenkinsfile to have timing info on output * make KubesTest resilient against transient pod failures in k8s * increase CPU request * add logic to allow specifying container resource requests * attempt to run unit and integration tests in parallel * change unit tests to use 3 cores to allow co-location on 8c machines * join grouped tests together to give pod meaningful name * add step to renew token with GKE * change renew step to use pods instead of nodes * fix bug where memory request is not correctly passed to pod * disable unit tests for now
2019-09-19 17:41:06 +00:00
}
}
PreAllocate pod resources during image build phase (#5587) * use zulu for jdk add some parallel groups * port kubesTest to Java remove asterix from tests listed by ListTests, instead add after allocation * attempt to setup unit test builds with correct github integrations # Conflicts: # .ci/dev/unit/Jenkinsfile * fix issue with github context * add credentials block * start pre-allocating pods for builds * test * add blocks for reporting build stages * add logic to preallocate pods during image building * tidy up Jenkinsfile for unit tests * add magic command line flag to enable preallocation of pods * make docker tag deterministic * fix issue concatenating docker tag inputs * add build type specific Jenkinsfile * try new preallocation approach * make pre-allocation prefix group specific * force deAllocator to wait for pods to be actually deleted * revert jenkinsfiles in .ci * use smarter waiting logic to address review comments * add --stacktrace to builds to help debugging * fix issue with closed stream * add some logging around preallocation * tidy up by refactoring (de)allocate task generation into method * change default from 20 pods to 5 pods * fix issue where docker tag was unstable between building and running tests * more documentation * add some infrastructure around setting the log level for a given build * change preallocation pod duration to 5min * see if fast enough if using combined unit and integration tests * disable unit tests * print out test summaries * try and make the kubes client a per-use object, rather than a long lived object. This is step one of making GKE use possible * add log line about what command is executed in the pod
2019-10-28 11:48:04 +00:00
// stage('Unit Tests') {
// steps {
// sh "./gradlew " +
// "-DbuildId=\"\${BUILD_ID}\" " +
// "-Dkubenetize=true " +
// "-Ddocker.run.tag=\"\${DOCKER_TAG_TO_USE}\"" +
// " deAllocateForAllParallelUnitTest allParallelUnitTest --stacktrace"
// }
// }
}
}
}
post {
always {
archiveArtifacts artifacts: '**/pod-logs/**/*.log', fingerprint: false
junit '**/build/test-results-xml/**/*.xml'
}
cleanup {
deleteDir() /* clean up our workspace */
}
}
}