CORDA-3187: Add -XX:+HeapDumpOnOutOfMemoryError -XX:+CrashOnOutOfMemoryError to default JVM args for node (#5432)

docs/source/changelog.rst

@@ -56,6 +56,11 @@ Unreleased
 * Wildcards can now be used when specifying RPC permissions, for example ``StartFlow.foo.bar.*`` will allow users to start any flow in the
   ``foo.bar`` package. See :ref:`rpcUsers <corda_configuration_file_rpc_users>` for more information.
 
+* ``-XX:+HeapDumpOnOutOfMemoryError`` and ``-XX:+CrashOnOutOfMemoryError`` have been added to the default JVM options of the node.
+  A node which is running out of memory is now expected to stop immediately to preserve ledger consistency and avoid flaws in operations.
+  Note that it's a responsibility of a client application to handle RPC reconnection in case this happens.
+  See :ref:`setting_jvm_args` and :ref:`memory_usage_and_tuning` for further details.
+
 .. _changelog_v4.1:
 
 Version 4.1

docs/source/node-administration.rst

@@ -139,6 +139,8 @@ The following diagram illustrates Corda flow metrics visualized using hawtio:
 
 .. image:: resources/hawtio-jmx.png
 
+.. _memory_usage_and_tuning:
+
 Memory usage and tuning
 -----------------------
 
@@ -157,6 +159,11 @@ The example command above would give a 1 gigabyte Java heap.
 
 .. note:: Unfortunately the JVM does not let you limit the total memory usage of Java program, just the heap size.
 
+A node which is running out of memory is expected to stop immediately to preserve ledger consistency and avoid flaws in operations.
+Note that it's a responsibility of a client application to handle RPC reconnection in case this happens. It's also advised to have
+necessary JVM monitoring and restart infrastructure in place.
+See :ref:`setting_jvm_args` for further details on JVM out-of-memory related parameters.
+
 Hiding sensitive data
 ---------------------
 A frequent requirement is that configuration files must not expose passwords to unauthorised readers. By leveraging environment variables, it is possible to hide passwords and other similar fields.

docs/source/running-a-node.rst

@@ -43,12 +43,12 @@ There are several ways of setting JVM arguments for the node process (particular
 They are listed here in order of increasing priority, i.e. if the same flag is set in a way later in this list, it will override
 anything set earlier.
 
-:Default arguments in capsule: The capsuled corda node has default flags set to ``-Xmx512m -XX:+UseG1GC`` - this gives the node (a relatively
+:Default arguments in capsule: The capsuled Corda node has default flags set to ``-Xmx512m -XX:+UseG1GC`` - this gives the node (a relatively
    low) 512 MB of heap space and turns on the G1 garbage collector, ensuring low pause times for garbage collection.
 
 :Node configuration: The node configuration file can specify custom default JVM arguments by adding a section like:
 
-.. code-block:: none
+   .. code-block:: none
 
       custom = {
          jvmArgs: [ '-Xmx1G', '-XX:+UseG1GC' ]
@@ -70,6 +70,25 @@ anything set earlier.
 :Command line flag: You can set JVM args on the command line that apply to the launcher process and the node process as in the example
       above. This will override any value for the same flag set any other way, but will leave any other JVM arguments alone.
 
+:OutOfMemoryError handling: In addition to the JVM arguments listed above, the capsuled Corda node has two flags that cause the node to stop
+   on out-of-memory error and generate the corresponding diagnostic files::
+
+      -XX:+HeapDumpOnOutOfMemoryError -XX:+CrashOnOutOfMemoryError
+
+   With ``CrashOnOutOfMemoryError`` the node which is running out of memory is expected to stop immediately (fail-fast) to preserve ledger
+   consistency and avoid flaws in operations.
+
+   Unlike for arguments related to memory and GC, to completely replace the default out-of-memory error args, you must explicitly add
+   at least one out-of-memory error related argument into the ``custom.jvmArgs`` section. For example, the following config turns off
+   ``HeapDumpOnOutOfMemoryError`` and doesn't invoke ``CrashOnOutOfMemoryError`` option:
+
+   .. code-block:: none
+
+      custom = {
+         jvmArgs: [ '-Xmx1G', '-XX:+UseG1GC', '-XX:-HeapDumpOnOutOfMemoryError' ]
+      }
+
+
 Starting all nodes at once on a local machine from the command line
 -------------------------------------------------------------------
 

node/capsule/src/main/java/CordaCaplet.java

@@ -127,16 +127,25 @@ public class CordaCaplet extends Capsule {
         } else if (ATTR_JVM_ARGS == attr) {
             // Read JVM args from the config if specified, else leave alone.
             List<String> jvmArgs = new ArrayList<>((List<String>) super.attribute(attr));
+            boolean defaultOutOfMemoryErrorHandling = true;
             try {
                 List<String> configJvmArgs = nodeConfig.getStringList("custom.jvmArgs");
                 jvmArgs.clear();
                 jvmArgs.addAll(configJvmArgs);
                 log(LOG_VERBOSE, "Configured JVM args = " + jvmArgs);
+
+                // Switch off default OutOfMemoryError handling if any related JVM arg is specified in custom config
+                defaultOutOfMemoryErrorHandling = configJvmArgs.stream().noneMatch(arg -> arg.contains("OutOfMemoryError"));
             } catch (ConfigException.Missing e) {
                 // Ignore since it's ok to be Missing. Other errors would be unexpected.
             } catch (ConfigException e) {
                 log(LOG_QUIET, e);
             }
+            // Shutdown and print diagnostics on OutOfMemoryError.
+            if (defaultOutOfMemoryErrorHandling) {
+                jvmArgs.add("-XX:+HeapDumpOnOutOfMemoryError");
+                jvmArgs.add("-XX:+CrashOnOutOfMemoryError");
+            }
             return (T) jvmArgs;
         } else if (ATTR_SYSTEM_PROPERTIES == attr) {
             // Add system properties, if specified, from the config.