corda/Jenkinsfile

175 lines
6.2 KiB
Plaintext
Raw Normal View History

#!groovy
/**
* Jenkins pipeline to build Corda Opensource Pull Requests.
*/
@Library('corda-shared-build-pipeline-steps')
import static com.r3.build.BuildControl.killAllExistingBuildsForJob
killAllExistingBuildsForJob(env.JOB_NAME, env.BUILD_NUMBER.toInteger())
/**
* Common Gradle arguments for all Gradle executions
*/
String COMMON_GRADLE_PARAMS = [
'--no-daemon',
'--stacktrace',
'--info',
/*
** revert default behavour for `ignoreFailures` and
** do not ignore test failures in PR builds
*/
'-Ptests.ignoreFailures=false',
'-Pcompilation.warningsAsErrors=false',
'-Ptests.failFast=true',
'-Ddependx.branch.origin="${GIT_COMMIT}"', // DON'T change quotation - GIT_COMMIT variable is substituted by SHELL!!!!
'-Ddependx.branch.target="${CHANGE_TARGET}"', // DON'T change quotation - CHANGE_TARGET variable is substituted by SHELL!!!!
'--build-cache',
].join(' ')
pipeline {
agent { label 'standard' }
/*
* List options in alphabetical order
*/
options {
ansiColor('xterm')
buildDiscarder(logRotator(daysToKeepStr: '14', artifactDaysToKeepStr: '14'))
parallelsAlwaysFailFast()
timeout(time: 6, unit: 'HOURS')
timestamps()
}
add ability to group test types together (#5459) * add ability to group test types together * add ability to specify podCount for use in parallel testing * remove compiler xml * add Jenkinsfile to enable scanning * trigger build * add ability to specify what docker tag to use from outside of the build * fix docker work dir * fix pipeline syntax issues * use environment rather than `def` * move agent restrictor outside of stages block * use steps block * more pipeline syntax fixes * even more pipeline syntax fixes * even more pipeline syntax fixes * add kubenetize as property to image build * move clear of docker image to end of build rather than start to prevent colocated builds * escape dollar on docker image remove command * attempt to kill all existing jobs * fix compile issue due to killall_jobs * fix compile issue due to killall_jobs pt2 * fix spelling * make all variables environment variables * add logic to delete images locally after pushing * wrap testing phase with try / finally so that junit reports are always evaluated * change the behaviour around post build actions * break implicit link between testing phase and image building phase, allowing testing to occur without a rebuild and push of image * prepend registry name to provided tag * allow tasks to specify whether they wish to stream output from containers * add timestamps directive to Jenkinsfile to have timing info on output * make KubesTest resilient against transient pod failures in k8s * increase CPU request * add logic to allow specifying container resource requests * attempt to run unit and integration tests in parallel * change unit tests to use 3 cores to allow co-location on 8c machines * join grouped tests together to give pod meaningful name * add step to renew token with GKE * change renew step to use pods instead of nodes * fix bug where memory request is not correctly passed to pod * disable unit tests for now
2019-09-19 17:41:06 +00:00
/*
* List environment variables in alphabetical order
*/
environment {
TM-51 Read and write test results to artifactory. (#5597) * TM-51 Prep for reading and writing test results to artifactory. * TM-51 Tests from target branch if no tests for current branch * TM-51 Placeholder for test averaging over runs. * TM-51 Replace slashes in branch names used as tags. * TM-51 More placeholder work for the mean duration work. * TM-51 Write out average tests results as as csv. The csv file should grow and be updated on each run. This includes whether or not we are running unit tests, integration tests and so on. * TM-51 Comment out old junit test archiving, add more comments. * TM-51 Zip task needs to depend on a csv creation task. If there isn't a csv file present, then the zip task doesn't run due to 'NO-SOURCE' * TM-51 Zip task should ignore empty dirs * TM-51 Fix up loading of test results. We were looking for the wrong artifact name. Add a bit more logging. * TM-51 Fix up possible problem with allocating by class distribution. If we encounter a class we haven't seen before, there won't be any tests. This means we should give it some weight. '1' is far too small. * TM-51 Test that we are definitely increment the run count. Tracking down whether the zipped csv file should have incremented. * TM-51 Better default value for missing test/class names. Begin by using mean unit test duration, but we have the option to bump that to the mean class unit tests duration. * TM-51 More debug information around csv writing. We should be incrementing the tests. * TM-51 Reload the csv before updating it. * TM-51 Reduce verbosity of logging. * TM-51 Reinstate unit tests. Remove logging verbosity. * TM-51 Load tests from artifactory in memory and avoid interim file. * TM-51 Better handling of zero duration tests. Ensure we return zero times from junit artifacts which may either be zero or have no recorded time. Before writing the tests duration csv file, store those with a known time, and then store those with zero using the average time. * TM-51 Log whether we have recorded a test. Tracking down the curious case where we seem to not be rerunning the same set of tests on the second run. * TM-51 Capture junit files as well. Trying to track down whether some tests are intermittently run. * TM-51 Change task dependencies to ensure ziptask is triggered. * TM-51 Remove test assertion, and trigger build * TM-51 Add corda/enterprise to artifactory tag name. Moved properties to own file. * TM-51 Remove unnecessary mean class-based duration. * TM-51 Add more BucketingAllocator tests. We need these to nail down its behaviour some more. * TM-51 Further log information. We don't seem to be finding the tests in the 'production' runs which is odd. * TM-51 corda type double set? * TM-51 do not set the project type in the properties. SRP and all that. * TM-51 better plan reporting * TM-51 duration may be zero Another runtime problem that doesn't show in tests. * TM-51 better plan reporting * fix missing space after image id * fix merge issue in DistributedTesting * TM-51 remove unused code when GET/PUT-ting to Artifactory. * TM-51 put tasks in gradle group and tidy up zip task creation * TM-51 Fix the junit XML path. * TM-51 Fix the task graph * TM-51 Less logging
2019-11-02 09:07:53 +00:00
ARTIFACTORY_CREDENTIALS = credentials('artifactory-credentials')
BUILD_CACHE_CREDENTIALS = credentials('gradle-ent-cache-credentials')
BUILD_CACHE_PASSWORD = "${env.BUILD_CACHE_CREDENTIALS_PSW}"
BUILD_CACHE_USERNAME = "${env.BUILD_CACHE_CREDENTIALS_USR}"
CORDA_ARTIFACTORY_PASSWORD = "${env.ARTIFACTORY_CREDENTIALS_PSW}"
CORDA_ARTIFACTORY_USERNAME = "${env.ARTIFACTORY_CREDENTIALS_USR}"
CORDA_GRADLE_SCAN_KEY = credentials('gradle-build-scans-key')
CORDA_USE_CACHE = "corda-remotes"
Upgrade to gradle 7.6, kotlin 1.8 and jdk 17 Major changes due to JDK 17: 1. JDK17 JCE Provider now has built-in support for eddsas, corda uses the bouncycastle (i2p) implementation. This PR removes the conflicting algorithms from the built-in JCE provider. 2. JavaScript scripting has been removed from the JDK, the corda log4j config was using scripting to conditionally output additional diagnostic info if the MDC was populated. This PR has removed the scripting. 3. The artifactory plug-ins used are now deprecated, this PR has removed them and uses the same code as Corda 5 for publishing to artifactory. 4. Javadoc generation has been modified to use the latest dokka plug-ins. 5. Gradle 7.6 has implemented an incredibly annoying change where transitive dependencies are not put on the compile classpath, so that they have to be explicitly added as dependencies to projects. 6. Mockito has been updated, which sadly meant that quite a few source files have to changes to use the new (org.mockito.kotlin) package name. This makes this PR appear much larger than it is. 7. A number of tests have been marked as ignored to get a green, broadly they fall into 3 classes. The first is related to crypto keypair tests, it appears some logic in the JDK prefers to use the SunJCE implementation and we prefer to use bouncycastle. I believe this issue can be fixed with better test setup. The second group is related to our use of a method called "uncheckedCast(..)", the purpose of this method was to get rid of the annoying unchecked cast compiler warning that would otherwise exist. It looks like the Kotlin 1.9 compiler type inference differs and at runtime sometimes the type it infers is "Void" which causes an exception at runtime. The simplest solution is to use an explicit cast instead of unchecked cast, Corda 5 have removed unchecked cast from their codebase. The third class are a number of ActiveMQ tests which appear to have a memory leak somewhere.
2023-03-08 12:19:05 +00:00
JAVA_HOME="/usr/lib/jvm/java-17-amazon-corretto"
ENT-11255: Scan attachments to determine if they are Kotlin 1.2 or later The node now sends a transaction to the verifier if any of its attachments were compiled with Kotlin 1.2 (the net.corda.node.verification.external system property has been removed). It uses kotlinx-metadata to read the Kotlin metadata in the attachment to determine this. For now this scanning is done each time the attachment is loaded from the database. The existing external verification integration tests were converted into smoke tests so that 4.11 nodes could be involved. This required various improvements to NodeProcess.Factory. A new JAVA_8_HOME environment variable, pointing to JDK 8, is required to run these tests. There is still some follow-up work that needs to be done: Sending transactions from a 4.11 node to a 4.12 node works, but not the other way round. A new WireTransaction component group needs to be introduced for storing 4.12 attachments so that they can be safely ignored by 4.11 nodes, and the 4.12 node needs to be able to load both 4.11 and 4.12 versions of the same contracts CorDapp so that they can be both attached to the transaction. Even though attachments are cached when retrieved from the database, the Kotlin metadata version should be stored in the attachments db table, rather than being scanned each time. Finally, VerificationService was refactored into NodeVerificationSupport and can be passed into SignedTransaction.verifyInternal, instead of needing the much heavier VerifyingServiceHub. This makes it easier for internal tools to verify transactions and spawn the verifier if necessary.
2024-01-22 11:31:51 +00:00
JAVA_8_HOME = "/usr/lib/jvm/java-1.8.0-amazon-corretto"
}
stages {
stage('Compile') {
steps {
authenticateGradleWrapper()
sh script: [
'./gradlew',
COMMON_GRADLE_PARAMS,
'clean',
'jar',
'--parallel'
].join(' ')
}
}
add ability to group test types together (#5459) * add ability to group test types together * add ability to specify podCount for use in parallel testing * remove compiler xml * add Jenkinsfile to enable scanning * trigger build * add ability to specify what docker tag to use from outside of the build * fix docker work dir * fix pipeline syntax issues * use environment rather than `def` * move agent restrictor outside of stages block * use steps block * more pipeline syntax fixes * even more pipeline syntax fixes * even more pipeline syntax fixes * add kubenetize as property to image build * move clear of docker image to end of build rather than start to prevent colocated builds * escape dollar on docker image remove command * attempt to kill all existing jobs * fix compile issue due to killall_jobs * fix compile issue due to killall_jobs pt2 * fix spelling * make all variables environment variables * add logic to delete images locally after pushing * wrap testing phase with try / finally so that junit reports are always evaluated * change the behaviour around post build actions * break implicit link between testing phase and image building phase, allowing testing to occur without a rebuild and push of image * prepend registry name to provided tag * allow tasks to specify whether they wish to stream output from containers * add timestamps directive to Jenkinsfile to have timing info on output * make KubesTest resilient against transient pod failures in k8s * increase CPU request * add logic to allow specifying container resource requests * attempt to run unit and integration tests in parallel * change unit tests to use 3 cores to allow co-location on 8c machines * join grouped tests together to give pod meaningful name * add step to renew token with GKE * change renew step to use pods instead of nodes * fix bug where memory request is not correctly passed to pod * disable unit tests for now
2019-09-19 17:41:06 +00:00
stage('Stash') {
steps {
stash name: 'compiled', useDefaultExcludes: false
}
}
add ability to group test types together (#5459) * add ability to group test types together * add ability to specify podCount for use in parallel testing * remove compiler xml * add Jenkinsfile to enable scanning * trigger build * add ability to specify what docker tag to use from outside of the build * fix docker work dir * fix pipeline syntax issues * use environment rather than `def` * move agent restrictor outside of stages block * use steps block * more pipeline syntax fixes * even more pipeline syntax fixes * even more pipeline syntax fixes * add kubenetize as property to image build * move clear of docker image to end of build rather than start to prevent colocated builds * escape dollar on docker image remove command * attempt to kill all existing jobs * fix compile issue due to killall_jobs * fix compile issue due to killall_jobs pt2 * fix spelling * make all variables environment variables * add logic to delete images locally after pushing * wrap testing phase with try / finally so that junit reports are always evaluated * change the behaviour around post build actions * break implicit link between testing phase and image building phase, allowing testing to occur without a rebuild and push of image * prepend registry name to provided tag * allow tasks to specify whether they wish to stream output from containers * add timestamps directive to Jenkinsfile to have timing info on output * make KubesTest resilient against transient pod failures in k8s * increase CPU request * add logic to allow specifying container resource requests * attempt to run unit and integration tests in parallel * change unit tests to use 3 cores to allow co-location on 8c machines * join grouped tests together to give pod meaningful name * add step to renew token with GKE * change renew step to use pods instead of nodes * fix bug where memory request is not correctly passed to pod * disable unit tests for now
2019-09-19 17:41:06 +00:00
stage('All Tests') {
add ability to group test types together (#5459) * add ability to group test types together * add ability to specify podCount for use in parallel testing * remove compiler xml * add Jenkinsfile to enable scanning * trigger build * add ability to specify what docker tag to use from outside of the build * fix docker work dir * fix pipeline syntax issues * use environment rather than `def` * move agent restrictor outside of stages block * use steps block * more pipeline syntax fixes * even more pipeline syntax fixes * even more pipeline syntax fixes * add kubenetize as property to image build * move clear of docker image to end of build rather than start to prevent colocated builds * escape dollar on docker image remove command * attempt to kill all existing jobs * fix compile issue due to killall_jobs * fix compile issue due to killall_jobs pt2 * fix spelling * make all variables environment variables * add logic to delete images locally after pushing * wrap testing phase with try / finally so that junit reports are always evaluated * change the behaviour around post build actions * break implicit link between testing phase and image building phase, allowing testing to occur without a rebuild and push of image * prepend registry name to provided tag * allow tasks to specify whether they wish to stream output from containers * add timestamps directive to Jenkinsfile to have timing info on output * make KubesTest resilient against transient pod failures in k8s * increase CPU request * add logic to allow specifying container resource requests * attempt to run unit and integration tests in parallel * change unit tests to use 3 cores to allow co-location on 8c machines * join grouped tests together to give pod meaningful name * add step to renew token with GKE * change renew step to use pods instead of nodes * fix bug where memory request is not correctly passed to pod * disable unit tests for now
2019-09-19 17:41:06 +00:00
parallel {
stage('Another agent') {
agent {
label 'standard'
}
options {
skipDefaultCheckout true
}
post {
always {
archiveArtifacts artifacts: '**/*.log', allowEmptyArchive: true, fingerprint: true
junit testResults: '**/build/test-results/**/*.xml', keepLongStdio: true,allowEmptyResults: true
}
cleanup {
deleteDir() /* clean up our workspace */
}
}
stages {
stage('Unstash') {
steps {
unstash 'compiled'
}
}
stage('Recompile') {
steps {
authenticateGradleWrapper()
sh script: [
'./gradlew',
COMMON_GRADLE_PARAMS,
'jar',
'--parallel'
].join(' ')
}
}
stage('Unit Test') {
steps {
sh script: [
'./gradlew',
COMMON_GRADLE_PARAMS,
'test'
].join(' ')
}
}
Upgrade to gradle 7.6, kotlin 1.8 and jdk 17 Major changes due to JDK 17: 1. JDK17 JCE Provider now has built-in support for eddsas, corda uses the bouncycastle (i2p) implementation. This PR removes the conflicting algorithms from the built-in JCE provider. 2. JavaScript scripting has been removed from the JDK, the corda log4j config was using scripting to conditionally output additional diagnostic info if the MDC was populated. This PR has removed the scripting. 3. The artifactory plug-ins used are now deprecated, this PR has removed them and uses the same code as Corda 5 for publishing to artifactory. 4. Javadoc generation has been modified to use the latest dokka plug-ins. 5. Gradle 7.6 has implemented an incredibly annoying change where transitive dependencies are not put on the compile classpath, so that they have to be explicitly added as dependencies to projects. 6. Mockito has been updated, which sadly meant that quite a few source files have to changes to use the new (org.mockito.kotlin) package name. This makes this PR appear much larger than it is. 7. A number of tests have been marked as ignored to get a green, broadly they fall into 3 classes. The first is related to crypto keypair tests, it appears some logic in the JDK prefers to use the SunJCE implementation and we prefer to use bouncycastle. I believe this issue can be fixed with better test setup. The second group is related to our use of a method called "uncheckedCast(..)", the purpose of this method was to get rid of the annoying unchecked cast compiler warning that would otherwise exist. It looks like the Kotlin 1.9 compiler type inference differs and at runtime sometimes the type it infers is "Void" which causes an exception at runtime. The simplest solution is to use an explicit cast instead of unchecked cast, Corda 5 have removed unchecked cast from their codebase. The third class are a number of ActiveMQ tests which appear to have a memory leak somewhere.
2023-03-08 12:19:05 +00:00
stage('Smoke Test') {
steps {
sh script: [
'./gradlew',
COMMON_GRADLE_PARAMS,
'smokeTest'
].join(' ')
}
}
stage('Slow Integration Test') {
steps {
sh script: [
'./gradlew',
COMMON_GRADLE_PARAMS,
'slowIntegrationTest'
].join(' ')
}
}
}
}
stage('Same agent') {
post {
always {
archiveArtifacts artifacts: '**/*.log', allowEmptyArchive: true, fingerprint: true
junit testResults: '**/build/test-results/**/*.xml', keepLongStdio: true, allowEmptyResults: true
}
}
stages {
stage('Integration Test') {
steps {
sh script: [
'./gradlew',
COMMON_GRADLE_PARAMS,
'integrationTest'
].join(' ')
}
}
add ability to group test types together (#5459) * add ability to group test types together * add ability to specify podCount for use in parallel testing * remove compiler xml * add Jenkinsfile to enable scanning * trigger build * add ability to specify what docker tag to use from outside of the build * fix docker work dir * fix pipeline syntax issues * use environment rather than `def` * move agent restrictor outside of stages block * use steps block * more pipeline syntax fixes * even more pipeline syntax fixes * even more pipeline syntax fixes * add kubenetize as property to image build * move clear of docker image to end of build rather than start to prevent colocated builds * escape dollar on docker image remove command * attempt to kill all existing jobs * fix compile issue due to killall_jobs * fix compile issue due to killall_jobs pt2 * fix spelling * make all variables environment variables * add logic to delete images locally after pushing * wrap testing phase with try / finally so that junit reports are always evaluated * change the behaviour around post build actions * break implicit link between testing phase and image building phase, allowing testing to occur without a rebuild and push of image * prepend registry name to provided tag * allow tasks to specify whether they wish to stream output from containers * add timestamps directive to Jenkinsfile to have timing info on output * make KubesTest resilient against transient pod failures in k8s * increase CPU request * add logic to allow specifying container resource requests * attempt to run unit and integration tests in parallel * change unit tests to use 3 cores to allow co-location on 8c machines * join grouped tests together to give pod meaningful name * add step to renew token with GKE * change renew step to use pods instead of nodes * fix bug where memory request is not correctly passed to pod * disable unit tests for now
2019-09-19 17:41:06 +00:00
}
}
}
}
}
post {
always {
findBuildScans()
}
cleanup {
deleteDir() /* clean up our workspace */
}
}
}