Commit Graph

8050 Commits

Author SHA1 Message Date
Chris Rankin
f226ddc4f2 CORDA-2876: Integrate the DJVM with the Corda Node. (#5633)
* * CORDA-2876: Migrate DJVM serialization modules into Corda.
* Pre-generate Corda classes for DJVM sandbox when node boots in production mode.
* Ensure that all DJVM test contract CorDapps are signed.
* Test examining attachments within DJVM sandbox.
* Test Contract.verify() using cryptographic verify function.
* Add test cases for more non-determinism in Contract.verify().
* Update node-driver to support testing nodes with DJVM support.
* Modify Node to allow alternative DJVM configurations for testing.
* Refactor DeterministicVerifierFactoryService for default use-case.
* Small whitespace and code-style refactors.
* Create and activate a DJVM execution profile for the Node.
* Revert making Verifier implement AutoCloseable.
* Allow the node to cache sandboxed Corda byte-code for reuse.
* Use updated Quasar agent that knows not to touch DJVM classloaders.
* Fix Quasar's package exclusions globs for DJVM.
* Deserialise LedgerTransaction into the sandbox for Contract.verify().
* Add the DJVM's serialisation modules to the Corda node.
* Update the node for the latest DJVM API, and preserve the ConstructorForDeserialization annotation on user contract classes.
* Add corda-dev to repositories while DJVM is SNAPSHOT.
* Migrate DJVM specialisation into AbstractNode's ServiceHubInternalImpl.
* Exclude sandbox.** and shaded djvm.** classes from Quasar agent.
* Add the corda-dev repository to :node for the deterministic runtime.
* Turn Verifier into an abstract base class that is specialised by BasicVerifier and DeterministicVerifier.
* Add the Corda deterministic libraries to the Node, and split the DJVM sandbox across two SandboxClassLoader instances.
* Add DJVM to contract verification path inside Corda Node.
* Minor lambda simplifications and removing unused import.
* CORDA-2871: Remove @CordaSerializable from LedgerTransaction.
* CORDA-2871: Add a callback to ServicesForResolution to allow the Node to modify a LedgerTransaction object.
* CORDA-2871: Refactor the contract verification code into a separate class,
 and allow LedgerTransaction to choose different Verifier objects.
* Update DJVM to use Corda 4.4-SNAPSHOT. (#95)
* CORDA-3330: Allow DJVM to preload / pregenerate classes from selected jars. (#92)
* Add support for SourceClassLoader.getResources() to DJVM.
* Allow a SandboxConfiguration to preload sandbox byte-code for all classes inside jars containing META-INF/DJVM-preload.
* CORDA-3309: Remove explicit try-catch in favour of UncaughtExceptionHandler. (#91)
* CORDA-3309: Install UncaughtExceptionHandler for DJVM tasks. (#88)
* Fix tests broken by Windows line endings. (#82)
* CORDA-3292: Reimplement ExecutionProfile as a data class. (#80)
* CORDA-2877: Refactor how we create child SandboxConfiguration objects. (#76)
* CORDA-2877: Load bytecode from a persistent cache to prevent repeated rewriting. (#75)
* Refactor byte-code cache to SandboxConfiguration instead of AnalysisConfiguration. We cannot "mix and match" byte-code generated by different sets of rules.
* CORDA-3137: Enhance annotation handling so that we can allow some annotations to be mapped into the sandbox without also needing to be stitched. (#72)
* CORDA-2871: Minor cosmetic fixes. (#69)
* CORDA-3218: Align DJVM with internal Corda Serialisation API. (#68)
* Ensure we get the latest SNAPSHOT of the serialisation code.
* CORDA-2871: Refactor SourceClassLoader to define source classes. (#66)
* Rewrite SourceClassLoader to support parent/child relationships.
* Revert catching TypNotPresebtException - it was a symptom of a bigger problem.
* Remove AutoCloseable from AnalysisConfiguration and SourceClassLoader.
* SourceClassLoader.getResource() must delegate to its parent first.
* CORDA-2871: Ensure ClassLoader.loadClass() throws ClassNotFoundException for all cases where the class cannot be found. (#64)
* CORDA-2871: Modify sandbox tasks to implement both java.Function and sandbox.Function (#62)
* Make TaskExecutors implement BiFunction to make them composable.
* Create ImportTask to wrap a java.Function inside a sandbox.Function.
* Add createExecutor() and createRawExecutor() APIs to SandboxClassLoader.
* Update serialization to use SandboxClassLoader.toSandboxClass().
* Remove a layer of lambdas from the serialisation code.
* Update SandboxExecutor and SandboxRawExecutor.
* Rename Executor to TaskFactory.
* Rename dangling executor -> taskFactory.
* CORDA-2871: Sanity fixes! (#63)
* Improve message for SandboxClassLoadingException.
* Fix serialisation API for using sandboxed environment.
* CORDA-3174: Extend serialisation to include InputStream and OpaqueBytesSubSequence. (#60)
* Update DJVM Example project for serialisation.
* Add serializers for InputStream and OpaqueBytesSubSequence.
* Support ZIP Inflater and CRC32 inside the sandbox.
* Allow the DJVM to wrap java.io.InputStream as sandbox.java.io.InputStream.
* Configure tests also to preserve @DeprecatedConstructorForDeserialization.
* CORDA-3174: Implement Corda serialization modules. (#59)
* Create DJVM serialization modules.
* Create test cases for Array<T>, List<T> and List<Array<T>>.
* Refactor SandboxPrimiveSerializer for all primitive types.
* Implement SandboxCollectionSerializer to support Collection types.
* Implement SandboxMapSerializer to support Map types.
* Attempt to fix infinite loop when computing Collection and Map fingerprints.
* Apply special handling when deserialising sandbox.java.lang.Character.
* Remap Java primitive types to sandbox Java object types to deter evolution.
* Use Class.getPackage().getName() to determine sandbox package name.
* Implement SandboxEnumSerializer to support Enum types.
* Implement SandboxPublicKeySerializer to support Java security keys.
* Add serialization projects to the composite example project.
* Implement serializers for BigInteger, BigDecimal, Currency and StringBuffer.
* Test that deserialising does not instantiate the untrusted user classes.
* Implement serializers for java.time.* types.
* Add serialiser for BitSet - currently disabled until BitSet itself is supported.
* Add serialisers for EnumSet and Class.
* Include support for EnumMap in the SandboxMapSerializer.
* Ensure the DJVM Example project's tests preserve @CordaSerializable.
* Add support for UUID as a primitive type.
* Use common abortReadOnly() method for declaring serialization as unsupported.
* Streamline the API for deserialising into the sandbox.
* Add preliminary support for deserialising X.509 certificates.
* Implement serializer for java.util.Optional.
* Refactor configuration of the sandbox serialization scheme.
* Add tests for deserialising arrays of basic types.
* Include method annotations in annotation stitching. This ensures that `@ConstructorForDeserialization` is not dropped.
* Enable test for SandboxBitSetSerializer.
* Enable tests for X.509 serializers.
* Implement serializers for ProtonJ primitive types.
* Serialize java.util.Date as a primitive type.
* Add the bintray Gradle plugin to the serialisation modules.
* Do not publish serialisation modules - they will become part of Corda itself.

* CORDA-2876: Only apply DJVM sources to Node Driver when devMode=true.

* Resolve DeteKT warnings.

* Require Node's JVM to set -Dnet.corda.djvm=true in order to enable DJVM.

* Enable DJVM for DemoBench nodes.

* Disable Quasar instrumentation verification for DemoBench nodes.

* Upgrade to DJVM 1.0-RC01.

* Try to modify DriverParameters in a more "ABI friendly" way.

* Refactor and simplify sandbox deserialisation of primitive objects.

* Review fixes.

* Update EvolutionSerializerFactory to handle sandboxed primitive boxed types.
2019-11-05 13:44:18 +00:00
Stefano Franz
f9d8084d44
Merge pull request #5670 from corda/my_merge_branch
Merge OS 4.3 -> OS 4.4
2019-11-05 10:55:39 +00:00
stefano
5ac1e50135 Merge remote-tracking branch 'origin/release/os/4.3' into my_merge_branch 2019-11-04 17:48:15 +00:00
Barry
4a7a9a56be TM-80 Do not publish the junit zip file to Artifactory automatically. (#5667)
This functionality was only in place for debugging purposes, to switch
it back on, set -Dpublish.junit=true in the Jenkinsfile.
2019-11-04 17:37:43 +00:00
Barry
6bf49bc4b7 TM-81 Do not write out the callstack when we cannot find tests.csv (#5669)
The first run of any new branch will not find a corresponding tests.csv
and will return 404 not found which is fine.  We do not need to display
the callstack at warning level.
2019-11-04 17:36:33 +00:00
Chris Rankin
b2c944bb0e CORDA-3390: Allow EvolutionSerializer to handle primitive types becoming nullable. (#5661) 2019-11-04 16:19:46 +00:00
Stefano Franz
f7e0ce6f0b
convert to nanos from seconds rather than milliseconds when parsing JUnit xml (#5666)
* convert to nanos from seconds rather than milliseconds

* fix tests
2019-11-04 15:17:57 +00:00
Stefano Franz
5a0b8c7992
add extra whitespace in jenkins regression build command (#5665) 2019-11-04 13:26:43 +00:00
Stefano Franz
e09cd84339
add test time publishing to regression test build (#5664) 2019-11-04 13:05:13 +00:00
Stefano Franz
e4e920eee9
multiprocess port allocator is no longer used, so we can remove the tests as they add a significant amount of time to run (2-3 min) (#5663) 2019-11-04 11:56:38 +00:00
Barry
91e6c9783f TM-51 Read and write test results to artifactory. (#5597)
* TM-51  Prep for reading and writing test results to artifactory.

* TM-51  Tests from target branch if no tests for current branch

* TM-51  Placeholder for test averaging over runs.

* TM-51  Replace slashes in branch names used as tags.

* TM-51  More placeholder work for the mean duration work.

* TM-51  Write out average tests results as as csv.

The csv file should grow and be updated on each run.  This includes whether or not we are running unit tests, integration tests and so on.

* TM-51  Comment out old junit test archiving, add more comments.

* TM-51  Zip task needs to depend on a csv creation task.

If there isn't a csv file present, then the zip task doesn't run due to 'NO-SOURCE'

* TM-51  Zip task should ignore empty dirs

* TM-51  Fix up loading of test results.

We were looking for the wrong artifact name.
Add a bit more logging.

* TM-51  Fix up possible problem with allocating by class distribution.

If we encounter a class we haven't seen before, there won't be any tests.
This means we should give it some weight.  '1' is far too small.

* TM-51  Test that we are definitely increment the run count.

Tracking down whether the zipped csv file should have incremented.

* TM-51  Better default value for missing test/class names.

Begin by using mean unit test duration, but we have the option to bump
that to the mean class unit tests duration.

* TM-51  More debug information around csv writing.

We should be incrementing the tests.

* TM-51  Reload the csv before updating it.

* TM-51  Reduce verbosity of logging.

* TM-51  Reinstate unit tests.  Remove logging verbosity.

* TM-51  Load tests from artifactory in memory and avoid interim file.

* TM-51  Better handling of zero duration tests.

Ensure we return zero times from junit artifacts which may either be zero or have no recorded time.  Before writing the tests duration csv file, store those with a known time, and then store those with zero using the average time.

* TM-51  Log whether we have recorded a test.

Tracking down the curious case where we seem to not be rerunning the
same set of tests on the second run.

* TM-51  Capture junit files as well.

Trying to track down whether some tests are intermittently run.

* TM-51  Change task dependencies to ensure ziptask is triggered.

* TM-51  Remove test assertion, and trigger build

* TM-51  Add corda/enterprise to artifactory tag name.

Moved properties to own file.

* TM-51  Remove unnecessary mean class-based duration.

* TM-51  Add more BucketingAllocator tests.

We need these to nail down its behaviour some more.

* TM-51  Further log information.

We don't seem to be finding the tests in the 'production' runs which is odd.

* TM-51  corda type double set?

* TM-51  do not set the project type in the properties.

SRP and all that.

* TM-51  better plan reporting

* TM-51  duration may be zero

Another runtime problem that doesn't show in tests.

* TM-51  better plan reporting

* fix missing space after image id

* fix merge issue in DistributedTesting

* TM-51  remove unused code when GET/PUT-ting to Artifactory.

* TM-51  put tasks in gradle group and tidy up zip task creation

* TM-51 Fix the junit XML path.

* TM-51 Fix the task graph

* TM-51 Less logging
2019-11-02 09:07:53 +00:00
Stefano Franz
cf849fbdbd
Merge pull request #5662 from corda/setup_regression_builds
Setup regression builds in jenkins
2019-11-01 16:51:07 +00:00
stefano
f8b4b334e3 truncate pod name from start rather than end 2019-11-01 16:10:35 +00:00
stefano
d54f2ddd87 fix Library import 2019-11-01 15:57:08 +00:00
stefano
e7e7de1d05 add regression test jenkins file 2019-11-01 15:53:05 +00:00
Chris Rankin
43856319bd CORDA-3392: Fix caching of local AMQPSerializer<*> instances. (#5657) 2019-11-01 14:08:27 +00:00
Stefano Franz
359bb64d69
Merge pull request #5655 from corda/my_merge_branch
Use Kubernetes Jobs rather than Pods to preallocate nodes
2019-11-01 13:11:46 +00:00
snedamle
93ff072812 adding one point to whitelist contract constraints migration - signed… (#5568)
* adding one point to whitelist contract constraints migration - signed CorDapp JAR must be registered with the CZ network operator

* 1. Removing later releases section
2. Changing 4.0 to 4.3

* Changing 4.3 to |corda_version|
2019-11-01 10:49:04 +00:00
Andrius Dagys
d033fceeef CORDA-3365: Reintroduce dependency to fix BFT-Smart notary (#5640)
The commons-codec:1.10 library was removed due to a security vulnerability,
but in commons-codec:1.13 it appears to have been fixed.
2019-11-01 09:05:12 +00:00
James Higgs
e0701231ac [DOCS] Add rows for new database tables in node-database.rst (#5654) 2019-10-31 17:57:52 +00:00
stefano
7a9ee89ded modify watcher to watch jobs rather than pods 2019-10-31 16:51:33 +00:00
stefano
bd9d8dbdbd promote failure to delete to error log level 2019-10-31 15:58:05 +00:00
stefano
547e6d9edd use jobs to preallocate nodes instead of Pods as they support auto delete 2019-10-31 15:57:03 +00:00
thorgilman
99a571bd9e Fix reference to deprecated UAT on-boarding site (#5647)
Updated the UAT on-boarding doc to so that it no longer references depreciated UAT on-boarding site and instead links to the Corda Network participation page.
2019-10-31 15:50:47 +00:00
Dimos Raptis
79c22bd77a Constrain the cordapps loaded from driver tests to the minimum required (#5653) 2019-10-31 11:26:05 +00:00
Chris Rankin
2895283500 CORDA-3388: Restore mapping of 'java.lang.Void -> void' (#5650) 2019-10-30 17:47:13 +00:00
Chris Rankin
346410a2e1 CORDA-3327: Fix API scanning for Java 11. (#5645) 2019-10-30 17:24:34 +00:00
dependabot[bot]
b7273f931e Bump pillow from 5.1.0 to 6.2.0 in /docs (#5632)
Bumps [pillow](https://github.com/python-pillow/Pillow) from 5.1.0 to 6.2.0.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/master/CHANGES.rst)
- [Commits](https://github.com/python-pillow/Pillow/compare/5.1.0...6.2.0)

Signed-off-by: dependabot[bot] <support@github.com>
2019-10-30 16:37:59 +00:00
Stefano Franz
da61b9ed2a
Merge pull request #5646 from corda/my_merge_branch
Merge OS 4.3 -> OS 4.4
2019-10-30 15:46:33 +00:00
Stefan Iliev
03ab258fc2 Revert "CORDA-3307 - add support for environment variables in linux (#5523)" (#5643)
This reverts commit c882b221a5.
2019-10-29 17:55:58 +00:00
stefano
8fd3e867f3 Merge remote-tracking branch 'origin/release/os/4.3' into my_merge_branch 2019-10-29 16:24:08 +00:00
Razvan Codreanu
d5462a2afe Re enabling persistent volume claims (#5628)
* TM-68 reenabling persistent volume claims using azure files

* TM-68 jenkins stackstracee

* TM-68 removing duplicate volume

* TM-68 pushing storage class yaml file

* TM-68 writing all results to the new persistent volume

* TM-68 fix wrong directory

* TM-68 fix wrong directory

* reapply lost merge commit

* investigate missing POD from test results

* more investigations around pods not executing their tests

* make Pod command line more strict with regards to sub command failure

* make logs an artifact within jenkins

* tidy up command line
2019-10-29 16:23:22 +00:00
Anthony Keenan
8a12224b29
NOTICK: Switch on ignore enums rule in detekt (#5641) 2019-10-29 14:46:25 +00:00
szymonsztuka
ac633bef2f CORDA-3348 Migration from Corda 3.x to 4.x for PostgreSQL require a manual workaround - fix by probing for lowercase table as well to detect if this is an existing database (#5625) 2019-10-29 11:22:52 +00:00
Nick Dunstone
4c7fbbc31b SecureRandomHash for internal shell password (#5638) 2019-10-28 15:56:11 +00:00
Stefano Franz
f9890a5359
PreAllocate pod resources during image build phase (#5587)
* use zulu for jdk
add some parallel groups

* port kubesTest to Java
remove asterix from tests listed by ListTests, instead add after allocation

* attempt to setup unit test builds with correct github integrations

# Conflicts:
#	.ci/dev/unit/Jenkinsfile

* fix issue with github context

* add credentials block

* start pre-allocating pods for builds

* test

* add blocks for reporting build stages

* add logic to preallocate pods during image building

* tidy up Jenkinsfile for unit tests

* add magic command line flag to enable preallocation of pods

* make docker tag deterministic

* fix issue concatenating docker tag inputs

* add build type specific Jenkinsfile

* try new preallocation approach

* make pre-allocation prefix group specific

* force deAllocator to wait for pods to be actually deleted

* revert jenkinsfiles in .ci

* use smarter waiting logic to address review comments

* add --stacktrace to builds to help debugging

* fix issue with closed stream

* add some logging around preallocation

* tidy up by refactoring (de)allocate task generation into method

* change default from 20 pods to 5 pods

* fix issue where docker tag was unstable between building and running tests

* more documentation

* add some infrastructure around setting the log level for a given build

* change preallocation pod duration to 5min

* see if fast enough if using combined unit and integration tests

* disable unit tests

* print out test summaries

* try and make the kubes client a per-use object, rather than a long lived object. This is step one of making GKE use possible

* add log line about what command is executed in the pod
2019-10-28 11:48:04 +00:00
Chris Rankin
bcba66f95d CORDA-2963: Update the Deterministic Java API artifacts. (#5637)
* CORDA-2963: Restore FileChannel to deterministic-rt but still remove it from the API artifact.

* CORDA-2963: Restore stub ObjectStream classes to deterministic-rt but still remove them from the API artifact.

* CORDA-2963: Add currency.data to deterministic-rt.jar as a resource.

* CORDA-2963: Add tzdb.dat to deterministic-rt.jar as a resource.

* CORDA-3326: Restore supported java.util.concurrent.* classes to the API artifact.
2019-10-25 10:39:56 +01:00
Anthony Keenan
6e43bc1db6
CORDA-3302: Align OS + Ent SchemaMigration parameters (#5622)
* Rename SchemaMigrationTest and make ourName parameter nullable to remove enterprise conflicts

* Fix detekt baseline

* Further alignment

* Update detekt
2019-10-24 18:54:35 +01:00
kyriathar
e2836b1106 CORDA-3279 Change single quotes to double quotes fixes node's shutdown (#5611)
A ConfigException$Parse would be thrown at CordaCaplet#parseConfigFile.

com.typesafe.config.ConfigFactory needs ':' to be included in a double quoted and not in a single quoted string.
2019-10-23 10:01:24 +01:00
Christian Sailer
df8cc7282f CORDA-3360 Add nodeInfo permissions to web user 2019-10-23 09:58:59 +01:00
Andrius Dagys
6493ffefef DOCS: Update notary data visibility page 2019-10-22 16:32:50 +01:00
Anthony Keenan
444881d536
Merge pull request #5624 from corda/anthony/merge-20191021
OS 4.3->OS 4.4 Merge 21-10-2019
2019-10-22 11:32:38 +01:00
szymonsztuka
b524c6368b CORDA-3335 Corda Shell flow kill - better warning for misformatted flow ID (#5601)
* CORDA-3081 warn that flow ID passed to flow kill is malformed as due to JDK8 doesn't fully validate it (JDK8 bug https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8159339)

* CORDA-3335 Corda Shell flow kill - better warning for misformatted flow ID - exit earlier and don't RPC to node, refactoring for detekt
2019-10-22 11:32:17 +01:00
Ryan Fowler
5da114caa3 CORDA-3281: Drop some errors to warnings and clean up logic around (#5605)
shell "gracefulShutdown" command.
2019-10-22 11:02:04 +01:00
Dimos Raptis
61cdfa5b26 [CORDA-3341] - Add missing liquibase script for hibernate test (#5623) 2019-10-22 09:52:37 +01:00
Stefano Franz
d693a9c1ce
TM-65 re-add local port availability check (#5618)
* re-add local port availability check

* attempt to fix issue with port allocator

* ensure the serversocket is closed
2019-10-21 16:17:17 +01:00
Chris Rankin
51ffd3d032 CORDA-3346: Replace JitPack dependency on JediTerm with 2.22 from Maven Central. (#5619) 2019-10-21 15:16:36 +01:00
Jonathan Locke
2ea7e6ccae
CORDA-3346: Remove the JitPack repository from Corda (#5620)
CORDA-3346: Remove the JitPack repository from Corda (#5620)
2019-10-21 15:13:42 +01:00
Anthony Keenan
0a194cfb51 Merge branch 'release/os/4.3' into release/os/4.4
# Conflicts:
#	docs/source/docker-image.rst
2019-10-21 13:44:32 +01:00
Anthony Keenan
b9a3b3a871
CORDA-3336: Remove log4j plugins to stop errors in config generator (#5617) 2019-10-21 13:31:09 +01:00