Commit Graph

8429 Commits

Author SHA1 Message Date
Dan Newton
ce146bdd59
Merge pull request #6427 from corda/dan/CORDA-3722-session-rollbacks-backport-4.4
BACKPORT - NOTICK Non-database error handling in `withEntityManager` (#6239)
2020-07-02 09:50:22 +01:00
Dan Newton
5f4401d16a NOTICK Non-database error handling in withEntityManager (#6239)
When a non-database exception is thrown out of a `withEntityManager`
block, always check if the session needs to be rolled back.

This means if a database error is caught and a new non-database error is
thrown out of the `withEntityManager` block, the transaction is still
rolled back. The flow can then continue progressing as normal.
2020-07-01 18:18:21 +01:00
Razvan Codreanu
bf4d733336
INFRA-371 publish nightly and publish tag to docker hub steps (#6399) 2020-06-30 20:12:27 +01:00
Dan Newton
8cd2d4891a
Merge pull request #6416 from corda/dan/CORDA-3722-session-rollbacks-backport-4.4
CORDA-3722 session rollbacks - backport
2020-06-30 13:35:14 +01:00
LankyDan
d35e8ba526 CORDA-3722 Add imports in RestrictedEntityManager
Needed after backporting a change
2020-06-30 11:54:16 +01:00
Dan Newton
efd633c7b9 CORDA-3722 withEntityManager can rollback its session (#6187)
* CORDA-3722 withEntityManager can rollback its session

Improve the handling of database transactions when using
`withEntityManager` inside a flow.

Extra changes have been included to improve the safety and
correctness of Corda around handling database transactions.

This focuses on allowing flows to catch errors that occur inside an
entity manager and handle them accordingly.

Errors can be caught in two places:

- Inside `withEntityManager`
- Outside `withEntityManager`

Further changes have been included to ensure that transactions are
rolled back correctly.

Errors caught inside `withEntityManager` require the flow to manually
`flush` the current session (the entity manager's individual session).
By manually flushing the session, a `try-catch` block can be placed
around the `flush` call, allowing possible exceptions to be caught.

Once an error is thrown from a call to `flush`, it is no longer possible
to use the same entity manager to trigger any database operations. The
only possible option is to rollback the changes from that session.
The flow can continue executing updates within the same session but they
will never be committed. What happens in this situation should be handled
by the flow. Explicitly restricting the scenario requires a lot of effort
and code. Instead, we should rely on the developer to control complex
workflows.

To continue updating the database after an error like this occurs, a new
`withEntityManager` block should be used (after catching the previous
error).

Exceptions can be caught around `withEntityManager` blocks. This allows
errors to be handled in the same way as stated above, except the need to
manually `flush` the session is removed. `withEntityManager` will
automatically `flush` a session if it has not been marked for rollback
due to an earlier error.

A `try-catch` can then be placed around the whole of the
`withEntityManager` block, allowing the error to be caught while not
committing any changes to the underlying database transaction.

To make `withEntityManager` blocks work like mini database transactions,
save points have been utilised. A new savepoint is created when opening
a `withEntityManager` block (along with a new session). It is then used
as a reference point to rollback to if the session errors and needs to
roll back. The savepoint is then released (independently from
completing successfully or failing).

Using save points means, that either all the statements inside the
entity manager are executed, or none of them are.

- A new session is created every time an entity manager is requested,
but this does not replace the flow's main underlying database session.
- `CordaPersistence.transaction` can now determine whether it needs
to execute its extra error handling code. This is needed to allow errors
escape `withEntityManager` blocks while allowing some of our exception
handling around subscribers (in `NodeVaultService`) to continue to work.
2020-06-30 11:54:16 +01:00
Dan Newton
f6b5737277
ENT-5196 handle errors during flow initialisation (#6378)
## Summary

This change deals with multiple issues:

* Errors that occur during flow initialisation.

* Errors that occur when handling the outcome of an existing flow error.

* Failures to rollback and close a database transaction when an error
occurs in `TransitionExecutorImpl`.

* Removal of create and commit transaction actions around retrying a flow.

## Errors that occur during flow initialisation

Flow initialisation has been moved into the try/catch that exists inside
`FlowStateMachineImpl.run`. This means if an error is thrown all the way
out of `initialiseFlow` (which should rarely happen) it will be caught 
and move into a flow's standard error handling path. The flow should 
then properly terminate.

`Event.Error` was changed to make the choice to rollback be optional. 
Errors during flow initialisation cause the flow to not have a open 
database transaction. Therefore there is no need to rollback.

## Errors that occur when handling the outcome of an existing flow error

When an error occurs a flow goes to the flow hospital and is given an 
outcome event to address the original error. If the transition that was 
processing the error outcome event (`StartErrorPropagation` and
`RetryFlowFromSafePoint`) has an error then the flow aborts and
nothing happens. This means that the flow is left in a runnable state.

To resolve this, we now retry the original error outcome event whenever 
another error occurs doing so.

This is done by adding a new staff member that looks for 
`ErrorStateTransitionException` thrown in the error code path of 
`TransitionExecutorImpl`. It then takes the last outcome for that flow 
and schedules it to run again. This scheduling runs with a backoff.

This means that a flow will continually retry the original error outcome
event until it completes it successfully.

## Failures to rollback and close a database transaction when an error occurs in `TransitionExecutorImpl`
   
Rolling back and closing the database transaction inside of 
`TransitionExecutorImpl` is now done inside individual try/catch blocks
as this should not prevent the flow from continuing.

## Removal of create and commit transaction actions around retrying a flow

The database commit that occurs after retrying a flow can fail which 
required some custom code just for that event to prevent inconsistent 
behaviour. The transaction was only needed for reading checkpoints from 
the database, therefore the transaction was moved into 
`retryFlowFromSafePoint` instead and the commit removed.

If we need to commit data inside of `retryFlowFromSafePoint` in the 
future, a commit should be added directly to `retryFlowFromSafePoint`. 
The commit should occur before the flow is started on a new fiber.
2020-06-30 11:50:42 +01:00
Denis Rekalov
3f03de6fbd
CORDA-3856: Add Artemis plugin for validating AMQP message header and type (#6407) 2020-06-29 09:23:29 +01:00
Waldemar Żurowski
b21a3c33cd
Added Sonatype check for regression builds (#6401) 2020-06-26 10:48:47 +01:00
Ryan Fowler
ef582900cf
NOTICK Expand the regex to match what we already do in ENT (#6400) 2020-06-25 17:26:55 +01:00
Waldemar Zurowski
3377df2212 Switched to CMD for Windows Server
* when using shell there are huge performance penalty
2020-06-20 17:26:57 +01:00
Razvan Codreanu
e021022d0d
INFRA-357 Publish build tag (#6375) 2020-06-20 11:08:52 +01:00
Waldemar Zurowski
a7644dab8b Cosmetic change for Windows Server builds 2020-06-19 22:09:02 +01:00
Waldemar Żurowski
ac0e35c854
Trying to ignore push events for Windows Server builds (#6383) 2020-06-19 22:03:09 +01:00
Waldemar Żurowski
8a601e6d62
Merge pull request #6380 from corda/merge-43-to-44
NOTICK: forward merge from Corda OS 4.3 to 4.4
2020-06-19 19:35:19 +01:00
Waldemar Zurowski
f7b38f0b6d Merge branch 'release/os/4.3' into merge-43-to-44 2020-06-19 19:32:55 +01:00
Waldemar Żurowski
23f81e73a6
Remove obsolete Jenkins configration for smoke tests (#6379) 2020-06-19 19:30:11 +01:00
Razvan Codreanu
ad21e381dc
INFRA-357 Adding JDK 11 Regression and Build tag jenkinsfile (#6374) 2020-06-19 14:10:31 +01:00
Euan Cairncross
4bfb595e57
INFRA-357 - Add release tag Jenkins build for OS 4.4 (#6362) 2020-06-18 23:46:29 +01:00
Waldemar Żurowski
494073590b
INFRA-347: Corda builds for Windows Servers (#6356) 2020-06-18 23:19:06 +01:00
Razvan Codreanu
813e9dd242
INFRA-394 enable full junit log to avoid truncation (#6368) 2020-06-18 12:52:33 +01:00
Rick Parker
fe61781889
Merge pull request #6358 from corda/adel/merge-4.3-to-4.4-17June
Merge OS 4.3 to OS 4.4
2020-06-17 18:13:55 +01:00
Chris Rankin
d0c0a1d9ba
ENT-5430: Fix deserialisation of commands containing generic types. (#6359) 2020-06-17 17:28:26 +01:00
Adel El-Beik
debc962d1d Merge remote-tracking branch 'origin/release/os/4.3' into adel/merge-4.3-to-4.4-17June 2020-06-17 10:06:34 +01:00
Razvan Codreanu
2b7c220522
INFRA-387 preallocation and deployNodes migration to Code Checks (#6339)
* preallocate for slow integration tests as well
2020-06-12 15:30:15 +01:00
Joseph Zuniga-Daly
d7e55ed4e5
CORDA-3837: When delivering a lifecycle event, copy the database context into the lifecycle event thread (#6331) 2020-06-11 11:14:51 +01:00
Razvan Codreanu
cf8cd8e004
INFRA-348 JDK 11 Compatibility job migration (#6311) 2020-06-05 15:20:03 +01:00
Ramzi El-Yafi
6ab6099075
[INFRA-368] Add deployNodes and install gradle tasks (#6302) 2020-06-03 14:05:12 +01:00
Ramzi El-Yafi
a9b8751ce7
[INFRA-351] Jenkinsfile for code check jobs (#6294) 2020-05-29 13:08:00 +01:00
Ramzi El-Yafi
ef3270e32c
[INFRA-352] Artifactory publication in Jenkins (#6276) (#6290)
* [INFRA-352] Artifactory publication in Jenkins

* Address review comments
2020-05-28 22:34:37 +01:00
Adel El-Beik
a33309a31b
CORDA-3755: Backport AttachmentURLStreamHandlerFactory memory leak (#6274)
* CORDA-3755: Switched attachments map to a WeakHashMap (#6214)

* CORDA-3772: Now specify source and target of 8 when compiling contract classes.

* CORDA-3651: addManifest now uses separate files for reading and writing. (#6026)

* CORDA-3651: addManifest now uses separate files for reading and writing.

* CORDA-3651: The jar scanning loader now closes itsself.

Co-authored-by: Adel El-Beik <adelel-beik@19LDN-MAC108.local>

Co-authored-by: Adel El-Beik <adelel-beik@19LDN-MAC108.local>
2020-05-27 11:35:15 +01:00
Matthew Nesbit
ad40b6618d
Merge pull request #6281 from corda/nnagy-os-4.3-os-4.4-20200522
Nnagy os 4.3 os 4.4 20200522
2020-05-22 17:27:06 +01:00
nikinagy
61d9dd0bb1 Merge branch 'release/os/4.3' into nnagy-os-4.3-os-4.4-20200522 2020-05-22 15:33:58 +01:00
nikinagy
8e74eea607
CORDA-3587 - adding kdocs for current behaviour of VaultQueryCriteria (#6242)
* adding kdocs for current behaviour of VaultQueryCriteria

* improving the kdocs

* address PR comments
2020-05-21 13:26:55 +01:00
nikinagy
cc8ce3ca99
empty list checks (#6262) 2020-05-19 14:04:19 +01:00
nikinagy
350066d386
fix for handling empty lists in vault query (#6231) 2020-05-11 15:41:50 +01:00
Ryan Fowler
b6dc1d8c4a
NOTICK - Reduce flow count for test to improve run time (#6213) 2020-05-05 09:38:55 +01:00
Rick Parker
cfb985244a
Merge pull request #6211 from corda/CORDA-3662-backport-from-4.5
ENT-5263: Backport CORDA-3662: Use an INNER JOIN for network map cache queries
2020-05-04 16:15:14 +01:00
Ryan Fowler
55797612b2 ENT-5237: Remove DISTINCT (backport) 2020-05-04 15:02:33 +01:00
Ryan Fowler
ebdd40049c CORDA-3662: Use an INNER JOIN for network map cache queries, (#6062)
- rename add or update function for clarity
- put removal of old nodes after retrieval of new ones to avoid gaps in the map
- plus add a test
2020-05-04 11:10:21 +01:00
Matthew Nesbit
8eda8b744f
Merge pull request #6181 from corda/nnagy-os-4.3-os-4.4-20200423
Nnagy os 4.3 os 4.4 20200423
2020-04-24 15:00:34 +01:00
nikinagy
335372cfc1 Merge branch 'release/os/4.3' into nnagy-os-4.3-os-4.4-20200423 2020-04-23 16:03:17 +01:00
Joseph Zuniga-Daly
69a4f80cd2
ENT-5141: Fix ConcurrentModificationException in FetchDataFlow (#6176)
* ENT-5141: Fix ConcurrentModificationException in FetchDataFlow

* Make detekt happy

* Make CheckpointSerializationEnvironmentRule inheritable
2020-04-23 13:11:23 +01:00
nikinagy
02d21c7bac
making sure hibernate uses UTC time zone (#6168) 2020-04-22 13:34:17 +01:00
Chris Rankin
7e13491a25
CORDA-3716: Fix Enum serializers to handle enums that override toString()
* CORDA-3716: Fix SandboxEnumSerializer to handle enums that override toString().

* Remove more uses of Enum.toString() from the Corda serializer.

* Add test coverage for this case to standard enum serializer.

* Increase maxWaitTimeout in IRSDemoTest to 150 seconds.
2020-04-20 17:49:12 +01:00
Razvan Codreanu
f4c1119727
INFRA-284 switching from local k8s label (#6156) 2020-04-17 15:53:10 +01:00
Razvan Codreanu
4210c0d81f
INFRA-284 switching from local k8s label (#6157) 2020-04-17 15:52:47 +01:00
Chris Rankin
032f008e7e
CORDA-3688: Apply @Named annotation to CRaSH commands to fix usage messages. (#6126) 2020-04-03 08:35:01 +01:00
Chris Rankin
b340766506
ENT-5039 Improved help text for commands (#6006) (#6122)
* Improved help text for commands
* Address feedback

Co-authored-by: jakubbielawa <jakub.bielawa@gmail.com>
2020-04-02 17:43:51 +01:00
Chris Rankin
f1ebaa761b
CORDA-3680: Add CorDapp custom serialisers to Driver's in-process nodes. (#6098)
* Run serialisation tests with both in-process and out-of-process nodes.

* Add custom serialisers and whitelists to Driver's AMQPServerSerializationScheme.
2020-03-27 10:01:30 +00:00