Commit Graph

699 Commits

Author SHA1 Message Date
Kyriakos Tharrouniatis
2c9c2985c0
CORDA-3381: Errors in vault updates publisher are unsubscribing stopping observers from working (#5912)
* Throw SQLException or PersistenceException plain, that may come out of an unsafe subscriber

* Add explanatory comment about why we changed Observer.tee to use unsafe subscribe

* Introducing not unsubscribing version of Rx.Subscriber

* Wrap PublishSubjects with FlowSafeSubjects in all tests that test Observer.tee

* Minor code formatting

* Make rawUpdates Rx.Observers not unsubscribe when accessed from CordaServices - Do not allow rawUpdates subscribing from flows

* Warning fix: Add else block to when statement

* Revert "Wrap PublishSubjects with FlowSafeSubjects in all tests that test Observer.tee"

This reverts commit e419af86

* Correcting log message

* Improve log message

* Add fiber's id to log message and exception message

* Added test, asserting FlowSafeSubscriber is alive and re-accessed upon flow retry

* Logging flow name instead of flow id at VaultService.rawUpdates subscribing error

* Add kdoc to OnNextFailedException

* Minor text correction

* Update kdocs of FlowSafeSubject/ PreventSubscriptionsSubject

* Moved FlowSafeSubject under package node.internal as it is only used by NodeVaultService

* Add comment and update kdoc explaining how to subscribe with SafeSubscriber to FlowSafeSubject

* Change PreventSubscriptionsSubject#errorAction to be more specific; to return an Exception

* Minor text update

* Update messy comment

* Replace assertThat with assertEquals

* Splitting heartBeat to heartBeat1 and hearBeat2 for more clear asserting

* Correcting comment

* Update messy comment

* Splitting heartBeat into heartBeatOnNext and heartBeatOnError

* Update test name

* Add explanatory comment to test

* Update test name

* Update test and add test comment

* Moving NotarisedTxs from SendStateFlow to VaultObserverExceptionTest inside NodeHandle.getNotarisedTransactionIds

* Moving SubscribingRawUpdatesFlow from ErrorHandling to VaultObserverExceptionTest

* Update kdoc of FlowSafeSubscriber and FlowSafeSubscriber.onNext

* Make kdoc more clear

* Throw exception upon accessing VaultService.rawUpdates from within a flow

* Changing exception thrown when accessing VaultService.rawUpdates from within a flow to a CordaRuntimeException

* Minor kdoc update

* Update test comment

* Update kdoc of FlowSafeSubscriber

* Introducing Observable.flowSafeSubscribe public API method to subscribe with -non unsubscribing- Rx.Subscribers to Observables. It also replaced FlowSafeSubject

* Move CustomSafeSubscriber outside test methods

* Minor text update

* Add timeout to tests

* Update kdoc of flowSafeSubscribe

* Update kdoc of flowSafeSubscribe

* Update kdoc of flowSafeSubscribe

* Move FlowSafeSubscriber and flowSafeSubscribe under their own package

* Fix detekt issue

* Update Detekt baseline

* Revert "Update Detekt baseline"

This reverts commit 793a8ed9

* Fix Detekt issue

* Moved strictMode flag from flowSafeSubscribe to OnFlowSafeSubscribe

Moved OnFlowSafeSubscribe into internal package

Integration tested flowSafeLooseSubscribe

* Suppress Rx Deprecation

* Rename flowSafeSubscribe to flowSafeObservable

* Renaming flowSafeObservable to continueOnError and FlowSafeSubscriber to ResilientSubscriber
2020-02-21 10:02:34 +00:00
Chris Rankin
d89ce6608a Create a unit test for serialization whitelists via driver. 2020-02-19 10:42:34 +00:00
James Higgs
3547b629c3
[NOTICK] Add a detekt rule to catch tests with no timeout (#5959)
* [NOTICK] Add a custom detekt rule for tests with no timeout, and fix remaining missing timeouts

* [NOTICK] Add a test for custom detekt rules and tidying

* add timeout annotation to new test

Co-authored-by: Stefano Franz <roastario@gmail.com>
2020-02-14 17:31:02 +00:00
Stefano Franz
b23af5f0d2
add timeouts to all tests (#5875) 2020-02-11 10:14:05 +00:00
Florian Friemel
cb6ed6042c
Move jimfs dependency from node to test-utils. (#5911) 2020-01-31 10:55:05 +00:00
Ryan Fowler
d0faf4d7cb NOTICK - allow resolveJar errors to be quieter (#5898) 2020-01-27 14:08:08 +00:00
Kyriakos Tharrouniatis
7f62046c2f CORDA-3329 Exceptions thrown in raw vault observers can cause critical issues (#5816)
Observers registered on NodeVaultService#rawUpdates, if they throw an exception when called from serviceHub#recordTransactions and if this exception is not handled by the flow hospital, then this leads to the transaction not being recorded in the local vault. This could get the ledger in an out of sync state.

In the specific case this happens within FinalityFlow#notariseAndRecord this leads to the transaction being notarized but not recorded in the local vault nor broadcasted in any counter party. The -failed to be recorded locally- transaction and its output states are not visible to any vault, and its input states not able to consumed by a new transaction, since they are recorded as consumed within the Notary. In this specific case we need not loose, by any means, the current transaction.

We will handle all cases by catching all exceptions thrown from serviceHub#recordTransactions, wrapping them with a HospitalizeFlowException and throwing it instead. The flow will get to the hospital for observation to be retried from previous checkpoint on next node restart.
2020-01-24 18:21:07 +00:00
Dan Newton
4bae045a58 ENT-4601 Public API to run external operations from a flow (#5833)
Deprecate FlowAsyncOperation and reimplement public versions FlowExternalOperation and FlowExternalAsyncOperation.

await added to FlowLogic to allow easy calling from both Java and Kotlin. There are two overrides of await (one for FlowExternalOperation and FlowExternalAsyncOperation).

Implementations of FlowExternalOperation return a result (written as blocking code) from their execute function. This operation will then be executed using a thread provided by the externalOperationExecutor.

Implementations of FlowExternalAsyncOperation return a future from their execute function. This operation must be executed on a newly spawned thread or one provided by a thread pool. It is up to developers to handle threading in this scenario.

The default thread pool (externalOperationExecutor) can be configured through the flowExternalOperationThreadPoolSize node config.

The current implementation leaves FlowAsyncOperation alone, meaning that any developers that have used it (even though it is internal) won't need to change their apps. If this was not concern I would delete it completely and replumb the state machine code. Instead, it has been marked with @DoNotImplement and executeAsync is annotated with @Deprecated
2020-01-22 09:27:17 +00:00
Viktor Kolomeyko
0978500a9a CORDA-2942: Node lifecycle events (#5846)
* CORDA-2942: Port minimal set of changes to make lifecycle events work

... and make codebase compile.

* CORDA-2942: Undo some changes which are not strictly speaking necessary

* CORDA-2942: Make `NodeServicesContext` leaner and delete `extensions-api` module

* CORDA-2942: Reduce even more number of files affected

* CORDA-2942: Integration test fix

* CORDA-2942: Make events `AfterStart` and `BeforeStop` generic w.r.t. `NodeServicesContext`

* CORDA-2942: `NodeLifecycleObserverService` and a set of integration tests.

Public API violations are expected as well as integration tests failing.

* CORDA-2942: Re-work to introduce `ServiceLifecycleObserver`

* CORDA-2942: Explicitly mention a type of exception that may be thrown for some events.

* CORDA-2942: Register `ServiceLifecycleObserver` through `AppServiceHub`

* CORDA-2942: Fix integration test + KDocs update

* CORDA-2942: Detekt and `api-current` update

* CORDA-2942: Improvement to `CordaServiceLifecycleFatalTests`

... or else it has side effects on other tests.

* CORDA-2942: Add an integration test for new API use in Java

Driver test is written in Kotlin, but services definition is written in Java.

Also KDocs improvements.

* CORDA-2942: Documentation and release notes update

* CORDA-2942: First set of changes following review by @mnesbit

* CORDA-2942: Second set of changes following review by @mnesbit

* CORDA-2942: Added multi-threaded test

* CORDA-2942: Fixes

* CORDA-2942: Undo changes to `api-current.txt`

* CORDA-2942: Bare mimimum change to `api-current.txt` for CI gate to pass.

* CORDA-2942: Address review feedback from @rick-r3

* CORDA-2942: Detekt update

* CORDA-2942: Delete `ServiceLifecycleObserverPriority` and replace it with `Int` after discussion with @mnesbit

* CORDA-2942: Introduce more `NodeLifecycleEvent` and switch services to listen for those events

* CORDA-2942: Few more changes after input from @rick-r3

* First stub on integration test
Unfinished - hang on issue and pay

* CORDA-2942: Switch to use out-of-process nodes for the inetgration test

Currently Alice and Notary stuck waiting to hear from each other.

* CORDA-2942: Extra log lines during event distribution

* CORDA-2942: Asynchronously distribute lifecycle events

* CORDA-2942: Await for complete P2P client start-up

Next step: Add vault query to integration test

* CORDA-2942: Asynchronously distribute lifecycle events

Next step: Improve integration test

* CORDA-2942: Fix test broken by recent changes and improve logging

* CORDA-2942: Improvement of the test to be able to monitor actions performed by @CordaService in the remote process

* CORDA-2942: Add node re-start step to the integration test

* CORDA-2942: Remove `CORDAPP_STOPPED` event for now

* CORDA-2942: s/CORDAPP_STARTED/STATE_MACHINE_STARTED/

* CORDA-2942: Inverse the meaning of `priority` as requested by @rick-r3

* CORDA-2942: Register `AppServiceHubImpl` for lifecycle events and put a warning when SMM is not ready.
2020-01-21 13:38:02 +00:00
Chris Rankin
a7147c1ffd ENT-4652: Provide an "attachment fixup" mechanism to repair broken transactions. (#5825)
* Do not register cordapp custom serialisers when using attachment classloader.

* Record the URLs of CorDapp JARs that contain custom serialisers. Include these JARs as extra attachments if we discover that we're missing a custom serialiser during transaction verification.

* Check for disabled serializer when explicitly requesting a custom serializer.
Refactor test case to force use of a custom serializer.

* Tidy up basic custom serializer test.

* Also test that TransactionBuilder rejects missing custom serializers.

* Remove test whitelists, which should not be needed with custom serialisers.

* Add changelog entry. Also align TestCordappImpl.findRoots() with OS backports.

* Second approach based around CorDapps inside AttachmentStorage - report missing type descriptor or any non-composable types.

* Initial implementation of Corda-Fixup rules inside a CorDapp jar.

* Replace original "automatic attachment fixing" mechanism completely.

* First review comments: restore "missing class" logic to TransactionBuilder.

* Restore "missing class" mechanism as fallback for SignedTransaction too.
2020-01-14 15:18:51 +00:00
James Higgs
d0543d7270 [CORDA-3436] Allow CorDapps access to node diagnostic information (#5834)
* [CORDA-3436] Allow CorDapps access to node diagnostic information

* [CORDA-3436] Fix API breakages

* [CORDA-3436] Improve documentation around diagnostics service

* [CORDA-3436] Remove CorDapps from the diagnostics information

* [CORDA-3436] Silence detekt warning
2020-01-09 11:18:32 +00:00
Denis Rekalov
bc96bea24a ENT-4628: Harmonize CryptoService and BCCryptoService between OS and ENT (#5822) 2019-12-18 16:54:39 +00:00
Viktor Kolomeyko
43205e1f1a CORDA-3471: Create CordaTransactionSupport and make it accessible through AppServiceHub (#5768)
* CORDA-3471: Create `CordaTransactionSupport` and use wherever possible instead of `CordaPersistence`

* CORDA-3471: Address comments by @mnesbit
- Relocate `CordaTransactionSupport` to `core`
- Create a lighter version of transaction - `VaultTransaction` that gives access to `session` object only.

* CORDA-3471: More changes after discussion with  @mnesbit
- Rename `VaultTransaction` into `SessionScope`.

* CORDA-3471: Revert changes to most of the files after conversation with @mnesbit and @rick-r3

* CORDA-3471: Introduce `CordaTransactionSupportImpl` and make it accessible via `AppServiceHub`.

* CORDA-3471: Minor change (comment).

* CORDA-3471: Address input from @mnesbit

* CORDA-3471: Address input from @rick-r3

* CORDA-3471: Make Detekt happier

* CORDA-3471: Add a new test that proves transactions can be started from client threads

As requested by @mnesbit

* CORDA-3471: Change log and documentation update.

As requested by @mnesbit
2019-12-04 17:18:40 +00:00
Chris Rankin
5a41ec9b82 CORDA-3464: Also scan attachment:// URLs for custom serializers. (#5769)
* CORDA-3464: Also scan attachment:// URLs for custom serializers.

* Only scan the given classloader - ignore this classloader's parents.

* Upgrade to ClassGraph 4.8.58 - for "robustness fixes".

* Register the attachment:// URL scheme using AttachmentsClassLoader.

* Add integration test for custom serializer in contract state.

* Rename Currancy -> Currantsy, just to make the point.
2019-12-03 15:50:21 +00:00
Stefan Iliev
926429647d CORDA-3307: Fix for underscore variables (#5682)
* Revert "Revert "CORDA-3307 - add support for environment variables in linux (#5523)" (#5643)"

This reverts commit 03ab258fc2.

* Env variables with underscore are now validated using schema validation and checking for unknown key errors.

* Resolving comments from PR review.

* Fix for deprecated import.

* Reworked logic according to PR review.

* Resolved bad string parsing problems where the json structure could be broken if some symbols were included in the key or value.
2019-11-19 17:51:52 +00:00
stefano
b11fd02f6e Merge remote-tracking branch 'origin/release/os/4.3' into my_merge_branch 2019-11-15 15:44:45 +00:00
Razvan Codreanu
deed4e9763 Enable test retry in distributed testing (#5702)
* TM-88 only write the test to the file if it passes otherwise retry

* TM-88 1 failing test for testing purposes

* TM-88 fail an integration test instead of unit test

* TM-88 include failed test for retry

* TM-88 new logic for retrying failed tests

* TM-88 refactored to use retry class when dealing with test failures + copy results even after pod failure

* TM-88 remaining pods as set to prevent duplicates, limit retries to prevent crashed build and a bit more logging

* TM-88 atomic int instead of int array

* TM-88 moving atomic int outside retry loops

* TM-88 removing second retry as it is not needed

* TM-88 small fix to final copy

* TM-88 while loop to prevent destruction of results on test retry

* TM-88 removing shutdown hook

* TM-88 reverting to previous implementation

* TM-88 copying xml files before retry

* TM-88 removing fail
2019-11-15 15:07:23 +00:00
Jonathan Locke
612fd83893 Fixed node driver issue 2019-11-14 11:25:56 +00:00
Jonathan Locke
c31cddaffc Fixed merge conflicts 2019-11-14 10:40:12 +00:00
Dan Newton
b15db200e3 NOTICK: Quick and dirty change to stop "Unable to start notaries." error message (#5686)
* Quick and dirty change to stop "Unable to start notaries." error message (#5686)

"Unable to start notaries. A required port might be bound already" is
returned whenever a startup error occurs while starting the notary nodes
in driver tests. This hides the real error.

This change prints the actual error to std_err and read from file
at a later point. This means the real error is not lost and will be
shown in failed builds.

* Suppress detekt warnings
2019-11-07 09:43:29 +00:00
stefano
6a3a9eded3 Merge remote-tracking branch 'origin/release/os/4.3' into my_merge_branch
# Conflicts:
#	core/src/main/kotlin/net/corda/core/internal/TransactionVerifierServiceInternal.kt
#	node/src/integration-test/kotlin/net/corda/node/services/rpc/RpcExceptionHandlingTest.kt
#	testing/node-driver/src/main/kotlin/net/corda/testing/node/internal/DriverDSLImpl.kt
2019-11-05 17:19:18 +00:00
Chris Rankin
02ed8e91b3 CORDA-2876: Ensure NodeConfig does not have devModeOptions outside of devMode. (#5676) 2019-11-05 16:59:40 +00:00
Chris Rankin
f226ddc4f2 CORDA-2876: Integrate the DJVM with the Corda Node. (#5633)
* * CORDA-2876: Migrate DJVM serialization modules into Corda.
* Pre-generate Corda classes for DJVM sandbox when node boots in production mode.
* Ensure that all DJVM test contract CorDapps are signed.
* Test examining attachments within DJVM sandbox.
* Test Contract.verify() using cryptographic verify function.
* Add test cases for more non-determinism in Contract.verify().
* Update node-driver to support testing nodes with DJVM support.
* Modify Node to allow alternative DJVM configurations for testing.
* Refactor DeterministicVerifierFactoryService for default use-case.
* Small whitespace and code-style refactors.
* Create and activate a DJVM execution profile for the Node.
* Revert making Verifier implement AutoCloseable.
* Allow the node to cache sandboxed Corda byte-code for reuse.
* Use updated Quasar agent that knows not to touch DJVM classloaders.
* Fix Quasar's package exclusions globs for DJVM.
* Deserialise LedgerTransaction into the sandbox for Contract.verify().
* Add the DJVM's serialisation modules to the Corda node.
* Update the node for the latest DJVM API, and preserve the ConstructorForDeserialization annotation on user contract classes.
* Add corda-dev to repositories while DJVM is SNAPSHOT.
* Migrate DJVM specialisation into AbstractNode's ServiceHubInternalImpl.
* Exclude sandbox.** and shaded djvm.** classes from Quasar agent.
* Add the corda-dev repository to :node for the deterministic runtime.
* Turn Verifier into an abstract base class that is specialised by BasicVerifier and DeterministicVerifier.
* Add the Corda deterministic libraries to the Node, and split the DJVM sandbox across two SandboxClassLoader instances.
* Add DJVM to contract verification path inside Corda Node.
* Minor lambda simplifications and removing unused import.
* CORDA-2871: Remove @CordaSerializable from LedgerTransaction.
* CORDA-2871: Add a callback to ServicesForResolution to allow the Node to modify a LedgerTransaction object.
* CORDA-2871: Refactor the contract verification code into a separate class,
 and allow LedgerTransaction to choose different Verifier objects.
* Update DJVM to use Corda 4.4-SNAPSHOT. (#95)
* CORDA-3330: Allow DJVM to preload / pregenerate classes from selected jars. (#92)
* Add support for SourceClassLoader.getResources() to DJVM.
* Allow a SandboxConfiguration to preload sandbox byte-code for all classes inside jars containing META-INF/DJVM-preload.
* CORDA-3309: Remove explicit try-catch in favour of UncaughtExceptionHandler. (#91)
* CORDA-3309: Install UncaughtExceptionHandler for DJVM tasks. (#88)
* Fix tests broken by Windows line endings. (#82)
* CORDA-3292: Reimplement ExecutionProfile as a data class. (#80)
* CORDA-2877: Refactor how we create child SandboxConfiguration objects. (#76)
* CORDA-2877: Load bytecode from a persistent cache to prevent repeated rewriting. (#75)
* Refactor byte-code cache to SandboxConfiguration instead of AnalysisConfiguration. We cannot "mix and match" byte-code generated by different sets of rules.
* CORDA-3137: Enhance annotation handling so that we can allow some annotations to be mapped into the sandbox without also needing to be stitched. (#72)
* CORDA-2871: Minor cosmetic fixes. (#69)
* CORDA-3218: Align DJVM with internal Corda Serialisation API. (#68)
* Ensure we get the latest SNAPSHOT of the serialisation code.
* CORDA-2871: Refactor SourceClassLoader to define source classes. (#66)
* Rewrite SourceClassLoader to support parent/child relationships.
* Revert catching TypNotPresebtException - it was a symptom of a bigger problem.
* Remove AutoCloseable from AnalysisConfiguration and SourceClassLoader.
* SourceClassLoader.getResource() must delegate to its parent first.
* CORDA-2871: Ensure ClassLoader.loadClass() throws ClassNotFoundException for all cases where the class cannot be found. (#64)
* CORDA-2871: Modify sandbox tasks to implement both java.Function and sandbox.Function (#62)
* Make TaskExecutors implement BiFunction to make them composable.
* Create ImportTask to wrap a java.Function inside a sandbox.Function.
* Add createExecutor() and createRawExecutor() APIs to SandboxClassLoader.
* Update serialization to use SandboxClassLoader.toSandboxClass().
* Remove a layer of lambdas from the serialisation code.
* Update SandboxExecutor and SandboxRawExecutor.
* Rename Executor to TaskFactory.
* Rename dangling executor -> taskFactory.
* CORDA-2871: Sanity fixes! (#63)
* Improve message for SandboxClassLoadingException.
* Fix serialisation API for using sandboxed environment.
* CORDA-3174: Extend serialisation to include InputStream and OpaqueBytesSubSequence. (#60)
* Update DJVM Example project for serialisation.
* Add serializers for InputStream and OpaqueBytesSubSequence.
* Support ZIP Inflater and CRC32 inside the sandbox.
* Allow the DJVM to wrap java.io.InputStream as sandbox.java.io.InputStream.
* Configure tests also to preserve @DeprecatedConstructorForDeserialization.
* CORDA-3174: Implement Corda serialization modules. (#59)
* Create DJVM serialization modules.
* Create test cases for Array<T>, List<T> and List<Array<T>>.
* Refactor SandboxPrimiveSerializer for all primitive types.
* Implement SandboxCollectionSerializer to support Collection types.
* Implement SandboxMapSerializer to support Map types.
* Attempt to fix infinite loop when computing Collection and Map fingerprints.
* Apply special handling when deserialising sandbox.java.lang.Character.
* Remap Java primitive types to sandbox Java object types to deter evolution.
* Use Class.getPackage().getName() to determine sandbox package name.
* Implement SandboxEnumSerializer to support Enum types.
* Implement SandboxPublicKeySerializer to support Java security keys.
* Add serialization projects to the composite example project.
* Implement serializers for BigInteger, BigDecimal, Currency and StringBuffer.
* Test that deserialising does not instantiate the untrusted user classes.
* Implement serializers for java.time.* types.
* Add serialiser for BitSet - currently disabled until BitSet itself is supported.
* Add serialisers for EnumSet and Class.
* Include support for EnumMap in the SandboxMapSerializer.
* Ensure the DJVM Example project's tests preserve @CordaSerializable.
* Add support for UUID as a primitive type.
* Use common abortReadOnly() method for declaring serialization as unsupported.
* Streamline the API for deserialising into the sandbox.
* Add preliminary support for deserialising X.509 certificates.
* Implement serializer for java.util.Optional.
* Refactor configuration of the sandbox serialization scheme.
* Add tests for deserialising arrays of basic types.
* Include method annotations in annotation stitching. This ensures that `@ConstructorForDeserialization` is not dropped.
* Enable test for SandboxBitSetSerializer.
* Enable tests for X.509 serializers.
* Implement serializers for ProtonJ primitive types.
* Serialize java.util.Date as a primitive type.
* Add the bintray Gradle plugin to the serialisation modules.
* Do not publish serialisation modules - they will become part of Corda itself.

* CORDA-2876: Only apply DJVM sources to Node Driver when devMode=true.

* Resolve DeteKT warnings.

* Require Node's JVM to set -Dnet.corda.djvm=true in order to enable DJVM.

* Enable DJVM for DemoBench nodes.

* Disable Quasar instrumentation verification for DemoBench nodes.

* Upgrade to DJVM 1.0-RC01.

* Try to modify DriverParameters in a more "ABI friendly" way.

* Refactor and simplify sandbox deserialisation of primitive objects.

* Review fixes.

* Update EvolutionSerializerFactory to handle sandboxed primitive boxed types.
2019-11-05 13:44:18 +00:00
Dimos Raptis
485feb2d6c CORDA-3350: Increase size of constraints column (#5639)
* CORDA-3350: Increase size of constraints column (#5639)

* Detekt

* Update api file with new threshold

* Add check in transaction builder

* Revert "Add check in transaction builder"

This reverts commit ca3128f44c.

* Add check for max number of keys

* Update api file

* Address Tudor's comments

* Remove check for pre-5 and add test for EC keys

* fix typo and rename liquibase script

* updated docs with measurement numbers for composite keys

* Make detekt happy again
2019-11-05 11:00:26 +00:00
Jonathan Locke
c193aa46f0
CID-1154: reliable finality merge to OS (#5658)
CID-1154: reliable finality merge to OS (#5658)
2019-11-05 10:48:00 +00:00
Stefano Franz
e4e920eee9
multiprocess port allocator is no longer used, so we can remove the tests as they add a significant amount of time to run (2-3 min) (#5663) 2019-11-04 11:56:38 +00:00
LankyDan
3c0631a26a Fix VaultObserverExceptionTest due to differences between ENT and OS 2019-11-04 09:42:41 +00:00
Barry
91e6c9783f TM-51 Read and write test results to artifactory. (#5597)
* TM-51  Prep for reading and writing test results to artifactory.

* TM-51  Tests from target branch if no tests for current branch

* TM-51  Placeholder for test averaging over runs.

* TM-51  Replace slashes in branch names used as tags.

* TM-51  More placeholder work for the mean duration work.

* TM-51  Write out average tests results as as csv.

The csv file should grow and be updated on each run.  This includes whether or not we are running unit tests, integration tests and so on.

* TM-51  Comment out old junit test archiving, add more comments.

* TM-51  Zip task needs to depend on a csv creation task.

If there isn't a csv file present, then the zip task doesn't run due to 'NO-SOURCE'

* TM-51  Zip task should ignore empty dirs

* TM-51  Fix up loading of test results.

We were looking for the wrong artifact name.
Add a bit more logging.

* TM-51  Fix up possible problem with allocating by class distribution.

If we encounter a class we haven't seen before, there won't be any tests.
This means we should give it some weight.  '1' is far too small.

* TM-51  Test that we are definitely increment the run count.

Tracking down whether the zipped csv file should have incremented.

* TM-51  Better default value for missing test/class names.

Begin by using mean unit test duration, but we have the option to bump
that to the mean class unit tests duration.

* TM-51  More debug information around csv writing.

We should be incrementing the tests.

* TM-51  Reload the csv before updating it.

* TM-51  Reduce verbosity of logging.

* TM-51  Reinstate unit tests.  Remove logging verbosity.

* TM-51  Load tests from artifactory in memory and avoid interim file.

* TM-51  Better handling of zero duration tests.

Ensure we return zero times from junit artifacts which may either be zero or have no recorded time.  Before writing the tests duration csv file, store those with a known time, and then store those with zero using the average time.

* TM-51  Log whether we have recorded a test.

Tracking down the curious case where we seem to not be rerunning the
same set of tests on the second run.

* TM-51  Capture junit files as well.

Trying to track down whether some tests are intermittently run.

* TM-51  Change task dependencies to ensure ziptask is triggered.

* TM-51  Remove test assertion, and trigger build

* TM-51  Add corda/enterprise to artifactory tag name.

Moved properties to own file.

* TM-51  Remove unnecessary mean class-based duration.

* TM-51  Add more BucketingAllocator tests.

We need these to nail down its behaviour some more.

* TM-51  Further log information.

We don't seem to be finding the tests in the 'production' runs which is odd.

* TM-51  corda type double set?

* TM-51  do not set the project type in the properties.

SRP and all that.

* TM-51  better plan reporting

* TM-51  duration may be zero

Another runtime problem that doesn't show in tests.

* TM-51  better plan reporting

* fix missing space after image id

* fix merge issue in DistributedTesting

* TM-51  remove unused code when GET/PUT-ting to Artifactory.

* TM-51  put tasks in gradle group and tidy up zip task creation

* TM-51 Fix the junit XML path.

* TM-51 Fix the task graph

* TM-51 Less logging
2019-11-02 09:07:53 +00:00
LankyDan
3aaddb47ea Attempt to fix detekt issues in DriverDSLImpl again 2019-11-01 14:36:47 +00:00
LankyDan
dfb86f5d9c Attempt to fix detekt issues in DriverDSLImpl 2019-11-01 14:21:17 +00:00
LankyDan
4aa9add8c8 CORDA-2050 Upgrade Corda to Java 11 (compatibility mode) (#5356) Upgrade Corda to run with Java 11 (compatibility mode) - see https://github.com/corda/corda/pull/5356 - 3fafbe55
Reapply change that was lost during merge - Adjust resolution of byteman jar to use java 11 compatible mechanism. - a1077092

Manual cherry pick of these changes (a1077092 + 3fafbe55)
2019-11-01 11:50:16 +00:00
Viktor Kolomeyko
c53ea9dde5 Eliminate extensive printout when Byteman not found on the classpath. (#1277)
Byteman is absent for most of the integration tests and long stacktrace is seen in the log
presently that un-necessarily attracts attention and consumes logging space.

(cherry picked from commit 2b6e59e7bd)
2019-11-01 11:50:16 +00:00
LankyDan
bedfba8c3d ENT-1967: Illustration for Byteman library can be used in Node integration test. (#1204) - c396b80a
(Only took `DriverDSLImpl` changes)

Simplifying internal startNode with bytemanPort parameter - d88b02f7

Manual cherry pick of these changes (c396b80a + d88b02f7)
2019-11-01 11:50:16 +00:00
Christian Sailer
9f15457045 add else branch to avoid warning that fails the warning check 2019-11-01 11:48:07 +00:00
Christian Sailer
119f939ee1 Fix and or suppress detekt warnings 2019-11-01 11:48:07 +00:00
Dan Newton
268d129838 CORDA-3194 Replay start flow events when responding flow fails initial checkpoint commit (#2601)
* CORDA-3194 Replay start flow events when responding flow fails initial checkpoint commit

Logic has already been added to recover from initial checkpoint commit
failures on the initiating flow side but this did not suffice for
the same failure occurring on the responding flow's side.

The same idea has been added to resolve the responding flow's issue.

`ExternalMessageEvent` now has a `flowId` that is maintained on the
event. Messages can then be replayed to start/restart the flow, while
the event provides the flow id to each flow start.

Each `ExternalMessageEvent` implementation generates a random `flowId`
when constructed.

Events are stored in Artemis. This allows the solution to recover across
node restarts as the events will be pulled from artemis again when
restarting.

In the future `flowId`s will probably moved off of the events and
generated purely on the responding flow's node.

* CORDA-3194 Add test to verify that errors removing a responding flow are recoverable
2019-11-01 11:48:07 +00:00
Christian Sailer
1f71b071aa CORDA-3217 and CORDA-3195 Various bits arond SQL exceptions and flow hospital (#2605)
* Unwrap rx.OnErrorNotImplementedException so the hospital can handle the cause appropriately

* Add db failure cordapp

* Renamed folders to avoid ambiguity in gradle

* Add integration test for exception hospitalisation when thrown from an RX observable.

* Make the test slightly cleaner

* Fix the schema to actually match the requirements for my custom state. Thanks a bunch, H2.

* Switch test to use SqlException base class.

* Schedule error event if we detect that a commit or db flush has thrown (forcing the flow to error even if customer code then goes ahead to swallow the exception)

* Revert change to schedule extra error

* Add more tests for edge case with DB exceptions, changed CorDapp to suppor this an hook in the flow hospital

* Warning about unsubscribe
Check state transitioned from clean to error for hospital admission.

* Match the test to our actual expectations

* Revert "Revert change to schedule extra error"

This reverts commit 43d47937

* Prevent suppression of errors arising in `transaction()` and `jdbcConnection()`

* Test for SqlException caught trying to escape from recordTransaction and suppressed outside being intercepted.

* More tests for various error/catch combinations

* Clean up and comments

* Code reformat

* Fix test compilation
2019-11-01 11:48:07 +00:00
Stefan Iliev
03ab258fc2 Revert "CORDA-3307 - add support for environment variables in linux (#5523)" (#5643)
This reverts commit c882b221a5.
2019-10-29 17:55:58 +00:00
Stefano Franz
d693a9c1ce
TM-65 re-add local port availability check (#5618)
* re-add local port availability check

* attempt to fix issue with port allocator

* ensure the serversocket is closed
2019-10-21 16:17:17 +01:00
Stefan Iliev
c882b221a5 CORDA-3307 - add support for environment variables in linux (#5523)
* Added a new way for environment variables to be loaded, which allows for underscore based separation.

* Moved test to its own kotlin file.

* Added case insensitivity support.

* The corda. prefix is now case insensitive too.

* Removed unused variable.

* Added env variables support for driverDSL. Shadowing corda. properties raises an exception.

* Driver api stability fix.

* Changed type of cordapps param to reflect the real one, rather than what IntelliJ auto completed.

* Some detekt issue fixes. Spread operator removed, baselined api stability constructors and buggy line.

* Fixed misspelled variable.

* Reverted unintentional changes.

* Added suppress instead of changing baseline.

* Reworked logic to handle previously defined CORDA_ starting properties and handle accordingly. Fixed a bug where wrong class was used for reflection walking.

* Fix for detekt issues.

* Changed message to a more understandable one.

* Changelog + doc note, console error grammar.

* Changes according to PR review.
2019-10-21 12:01:14 +01:00
Stefano Franz
6e98adb085
use zulu for jdk in testing image (#5583)
* use zulu for jdk
add some parallel groups

* port kubesTest to Java
remove asterix from tests listed by ListTests, instead add after allocation
2019-10-14 13:35:22 +01:00
Dimos Raptis
7666ca0d80 [CORDA-3303] - Avoid flushing when inside a cascade (#5575)
* [CORDA-3303] - Avoid flushing when inside a cascade

* Remove listener infrastructure
2019-10-11 10:58:26 +01:00
Stefano Franz
298c91ce82
Webook test branch (#5528)
* add ability to index by class OR method

* disable unit tests again

* pending commit

* stream output of build for debugging

* remove git from the base image

* re-enable NodeRPCTests

* add ability to distribute tests based on TeamCity CSV output for test times

* try enabling unit tests again after test distribution has been fixed

* refactor BucketingAllocator to be a class, which bucketingAllocatorTask delegates to.

* finishing touches for improved test distribution

* create new pipelines for builds
2019-10-08 16:33:24 +02:00
josecoll
246ec8766e
CORDA-3224 JDK11 built and published artifacts to include classifier. (#5488)
* JDK11 built and published artifacts to include classifier.

* Added additional non-intuitive gradle dust sprinkled recommended by CR.

* Added additional non-intuitive gradle dust sprinkled recommended by CR.

* Bump corda gradle plugins version.

* Incorporating PR review feedback.

* jdk11Classifier
2019-10-08 10:45:10 +01:00
Ryan Fowler
6ca4b589e2 CORDA-3091: Move executor thread management into CordaRPCConnection (#5491) 2019-09-26 17:48:07 +01:00
Viktor Kolomeyko
51330c2e44 CORDA-3232: Support of multiple interfaces for RPC calls (#5495)
* CORDA-3232: Make backward compatible RPC client changes

Such that it will be able to talk to new and old server versions.

* CORDA-3232: Make backward compatible RPC server changes

Such that it will be able to talk to new and old client versions.

* CORDA-3232: Trick Detekt

* CORDA-3232: Integration test for multi-interface communication.

* CORDA-3232: Add legacy mode test.

* CORDA-3232: Making Detekt happier

* CORDA-3232: Fix Detekt baseline after merge with `4.3` branch

* CORDA-3232: Incrementing Platform version

As discussed with @lockathan

* CORDA-3232: Fix legacy test post platform version increment

* CORDA-3232: Use recursive logic to establish complete population of method names

* Revert "CORDA-3232: Incrementing Platform version"

This reverts commit d75f48aa

* CORDA-3232: Remove logic that conditions on PLATFORM_VERSION

* CORDA-3232: Making Detekt happier

* CORDA-3232: Few more changes after conversation with @mnesbit

* CORDA-3232: Make a strict match to `CordaRPCOps` on client side

Or else will fail:
net.corda.tools.shell.InteractiveShellIntegrationTest.dumpCheckpoints creates zip with json file for suspended flow

Flagging that `InternalCordaRPCOps.dumpCheckpoints` cannot be called.

* CORDA-3232: Address PR comments by @rick-r3

* CORDA-3232: Address further review input from @rick-r3

* Change the way how methods stored in the map;
* Extend test to make sure that `CordaRPCOps` can indeed be mixed with other RPC interfaces.
2019-09-26 16:01:14 +01:00
Tudor Malene
298d8ba69c CORDA-3024 Rename the webserver (#5489) 2019-09-26 10:20:49 +01:00
Ed Prosser
defa2565ea Merge branch 'release/os/4.1' of https://github.com/corda/corda into ENT-4104-4.3-EdP 2019-09-25 10:47:06 +01:00
josecoll
f4f46af706
CORDA-3209 Fix vault query for participants specified in common criteria (#5508)
* Generalise participant parsing code & additional test cases.

* Use a common predicate to expand the participants query (when specified more than once - eg. in fungible and linear query criteria).

* Introduce some re-usable functions.

* Additional code clean-up and improvements.

* Fix detekt MaxLineLength errors.
2019-09-24 10:32:06 +01:00
opticyclic
eaed69f26f Check If Quasar Is Active Using API (#5471)
* Do a proper check for whether the quasar agent is active instead of sniffing JVM args

* Use JavaAgent.isActive instead of SuspendableHelper.isJavaAgentActive for clarity
2019-09-18 17:15:56 +01:00