corda/docs/source/deterministic-modules.rst

.. raw:: html

    <style> .red {color:red} </style>

.. role:: red

Deterministic Corda Modules
===========================

A Corda contract's verify function should always produce the same results for the same input data. To that end,
Corda provides the following modules:
 
 #. ``core-deterministic``
 #. ``serialization-deterministic``
 #. ``jdk8u-deterministic``

These are reduced version of Corda's ``core`` and ``serialization`` modules and the OpenJDK 8 ``rt.jar``, where the
non-deterministic functionality has been removed. The intention here is that all CorDapp classes required for
contract verification should be compiled against these modules to prevent them containing non-deterministic behaviour.

.. note:: These modules are only a development aid. They cannot guarantee determinism without also including
          deterministic versions of all their dependent libraries, e.g. ``kotlin-stdlib``.

Generating the Deterministic Modules
------------------------------------

JDK 8
  ``jdk8u-deterministic`` is a "pseudo JDK" image that we can point the Java and Kotlin compilers to. It downloads the
  ``rt.jar`` containing a deterministic subset of the Java 8 APIs from the Artifactory.

  To build a new version of this JAR and upload it to the Artifactory, see the ``create-jdk8u`` module. This is a
  standalone Gradle project within the Corda repository that will clone the ``deterministic-jvm8`` branch of Corda's
  `OpenJDK repository <https://github.com/corda/openjdk>`_ and then build it. (This currently requires a C++ compiler,
  GNU Make and a UNIX-like development environment.)

Corda Modules
  ``core-deterministic`` and ``serialization-deterministic`` are generated from Corda's ``core`` and ``serialization``
  modules respectively using both `ProGuard <https://www.guardsquare.com/en/proguard>`_ and Corda's ``JarFilter`` Gradle
  plugin. Corda developers configure these tools by applying Corda's ``@KeepForDJVM`` and ``@DeleteForDJVM``
  annotations to elements of ``core`` and ``serialization`` as described `here <deterministic_annotations_>`_.

The build generates each of Corda's deterministic JARs in six steps:

 #. Some *very few* classes in the original JAR must be replaced completely. This is typically because the original
    class uses something like ``ThreadLocal``, which is not available in the deterministic Java APIs, and yet the
    class is still required by the deterministic JAR. We must keep such classes to a minimum!
 #. The patched JAR is analysed by ProGuard for the first time using the following rule:

    .. sourcecode:: groovy

        keep '@interface net.corda.core.KeepForDJVM { *; }'

    ..

    ProGuard works by calculating how much code is reachable from given "entry points", and in our case these entry
    points are the ``@KeepForDJVM`` classes. The unreachable classes are then discarded by ProGuard's ``shrink``
    option.
 #. The remaining classes may still contain non-deterministic code. However, there is no way of writing a ProGuard rule
    explicitly to discard anything. Consider the following class:

    .. sourcecode:: kotlin

        @CordaSerializable
        @KeepForDJVM
        data class UniqueIdentifier(val externalId: String?, val id: UUID) : Comparable<UniqueIdentifier> {
            @DeleteForDJVM constructor(externalId: String?) : this(externalId, UUID.randomUUID())
            @DeleteForDJVM constructor() : this(null)
            ...
        }

    ..

    While CorDapps will definitely need to handle ``UniqueIdentifier`` objects, both of the secondary constructors
    generate a new random ``UUID`` and so are non-deterministic. Hence the next "determinising" step is to pass the
    classes to the ``JarFilter`` tool, which strips out all of the elements which have been annotated as
    ``@DeleteForDJVM`` and stubs out any functions annotated with ``@StubOutForDJVM``. (Stub functions that
    return a value will throw ``UnsupportedOperationException``, whereas ``void`` or ``Unit`` stubs will do nothing.)
 #. After the ``@DeleteForDJVM`` elements have been filtered out, the classes are rescanned using ProGuard to remove
    any more code that has now become unreachable.
 #. The remaining classes define our deterministic subset. However, the ``@kotlin.Metadata`` annotations on the compiled
    Kotlin classes still contain references to all of the functions and properties that ProGuard has deleted. Therefore
    we now use the ``JarFilter`` to delete these references, as otherwise the Kotlin compiler will pretend that the
    deleted functions and properties are still present.
 #. Finally, we use ProGuard again to validate our JAR against the deterministic ``rt.jar``:

    .. literalinclude:: ../../core-deterministic/build.gradle
       :language: groovy
       :start-after: DOCSTART 01
       :end-before: DOCEND 01
    ..

    This step will fail if ProGuard spots any Java API references that still cannot be satisfied by the deterministic
    ``rt.jar``, and hence it will break the build.

Configuring IntelliJ with a Deterministic SDK
---------------------------------------------

We would like to configure IntelliJ so that it will highlight uses of non-deterministic Java APIs as :red:`not found`.
Or, more specifically, we would like IntelliJ to use the ``deterministic-rt.jar`` as a "Module SDK" for deterministic
modules rather than the ``rt.jar`` from the default project SDK, to make IntelliJ consistent with Gradle.

This is possible, but slightly tricky to configure because IntelliJ will not recognise an SDK containing only the
``deterministic-rt.jar`` as being valid. It also requires that IntelliJ delegate all build tasks to Gradle, and that
Gradle be configured to use the Project's SDK.

Creating the Deterministic SDK
    #. Create a JDK Home directory with the following contents:

        ``jre/lib/rt.jar``

       where ``rt.jar`` here is this renamed artifact:

       .. code-block:: xml

           <dependency>
               <groupId>net.corda</groupId>
               <artifactId>deterministic-rt</artifactId>
               <classifier>api</classifier>
           </dependency>

       ..

       .. note:: Gradle already creates this JDK in the project's ``jdk8u-deterministic/jdk`` directory, and you could
                 configure IntelliJ to use this location as well. However, you should also be aware that IntelliJ SDKs
                 are available for *all* projects to use.

                 To create this deterministic JDK image, execute the following:

                 .. code-block:: bash

                     $ gradlew jdk8u-deterministic:copyJdk

                 ..

    #. While IntelliJ is *not* running, locate the ``config/options/jdk.table.xml`` file in IntelliJ's configuration
       directory. Add an empty ``<jdk>`` section to this file:

       .. code-block:: xml

           <jdk version="2">
               <name value="1.8 (Deterministic)"/>
               <type value="JavaSDK"/>
               <version value="java version &quot;1.8.0&quot;"/>
               <homePath value=".. path to the deterministic JDK directory .."/>
               <roots>
               </roots>
           </jdk>

       ..

    #. Open IntelliJ and select ``File/Project Structure/Platform Settings/SDKs``. The "1.8 (Deterministic)" SDK should
       now be present. Select it and then click on the ``Classpath`` tab. Press the "Add" / "Plus" button to add
       ``rt.jar`` to the SDK's classpath. Then select the ``Annotations`` tab and include the same JAR(s) as the other
       SDKs.

Configuring the Corda Project
    #. Open the root ``build.gradle`` file and define this property:

       .. code-block:: gradle

           buildscript {
               ext {
                   ...
                   deterministic_idea_sdk = '1.8 (Deterministic)'
                   ...
               }
           }

       ..

Configuring IntelliJ
    #. Go to ``File/Settings/Build, Execution, Deployment/Build Tools/Gradle``, and configure Gradle's JVM to be the
       project's JVM.

    #. Go to ``File/Settings/Build, Execution, Deployment/Build Tools/Gradle/Runner``, and select these options:

        - Delegate IDE build/run action to Gradle
        - Run tests using the Gradle Test Runner

    #. Delete all of the ``out`` directories that IntelliJ has previously generated for each module.

    #. Go to ``View/Tool Windows/Gradle`` and click the ``Refresh all Gradle projects`` button.

These steps will enable IntelliJ's presentation compiler to use the deterministic ``rt.jar`` with the following modules:

    - ``core-deterministic``
    - ``serialization-deterministic``
    - ``core-deterministic:testing:common``

but still build everything using Gradle with the full JDK.

Testing the Deterministic Modules
---------------------------------

The ``core-deterministic:testing`` module executes some basic JUnit tests for the ``core-deterministic`` and
``serialization-deterministic`` JARs. These tests are compiled against the deterministic ``rt.jar``, although
they are still executed using the full JDK.

The ``testing`` module also has two sub-modules:

``core-deterministic:testing:data``
    This module generates test data such as serialised transactions and elliptic curve key pairs using the full
    non-deterministic ``core`` library and JDK. This data is all written into a single JAR which the ``testing``
    module adds to its classpath.

``core-deterministic:testing:common``
    This module provides the test classes which the ``testing`` and ``data`` modules need to share. It is therefore
    compiled against the deterministic API subset.


.. _deterministic_annotations:

Applying @KeepForDJVM and @DeleteForDJVM annotations
----------------------------------------------------

Corda developers need to understand how to annotate classes in the ``core`` and ``serialization`` modules correctly
in order to maintain the deterministic JARs.

.. note:: Every Kotlin class still has its own ``.class`` file, even when all of those classes share the same
          source file. Also, annotating the file:

          .. sourcecode:: kotlin

              @file:KeepForDJVM
              package net.corda.core.internal

          ..

          *does not* automatically annotate any class declared *within* this file. It merely annotates any
          accompanying Kotlin ``xxxKt`` class.

For more information about how ``JarFilter`` is processing the byte-code inside ``core`` and ``serialization``,
use Gradle's ``--info`` or ``--debug`` command-line options.

Deterministic Classes
    Classes that *must* be included in the deterministic JAR should be annotated as ``@KeepForDJVM``.

    .. literalinclude:: ../../core/src/main/kotlin/net/corda/core/KeepForDJVM.kt
       :language: kotlin
       :start-after: DOCSTART 01
       :end-before: DOCEND 01
    ..

    To preserve any Kotlin functions, properties or type aliases that have been declared outside of a ``class``,
    you should annotate the source file's ``package`` declaration instead:

    .. sourcecode:: kotlin

        @file:JvmName("InternalUtils")
        @file:KeepForDJVM
        package net.corda.core.internal

        infix fun Temporal.until(endExclusive: Temporal): Duration = Duration.between(this, endExclusive)

    ..

Non-Deterministic Elements
    Elements that *must* be deleted from classes in the deterministic JAR should be annotated as ``@DeleteForDJVM``.

    .. literalinclude:: ../../core/src/main/kotlin/net/corda/core/DeleteForDJVM.kt
        :language: kotlin
        :start-after: DOCSTART 01
        :end-before: DOCEND 01
    ..

    You must also ensure that a deterministic class's primary constructor does not reference any classes that are
    not available in the deterministic ``rt.jar``, nor have any non-deterministic default parameter values such as
    ``UUID.randomUUID()``. The biggest risk here would be that ``JarFilter`` would delete the primary constructor
    and that the class could no longer be instantiated, although ``JarFilter`` will print a warning in this case.
    However, it is also likely that the "determinised" class would have a different serialisation signature than
    its non-deterministic version and so become unserialisable on the deterministic JVM.

    Be aware that package-scoped Kotlin properties are all initialised within a common ``<clinit>`` block inside
    their host ``.class`` file. This means that when ``JarFilter`` deletes these properties, it cannot also remove
    their initialisation code. For example:

    .. sourcecode:: kotlin

        package net.corda.core

        @DeleteForDJVM
        val map: MutableMap<String, String> = ConcurrentHashMap()

    ..

    In this case, ``JarFilter`` would delete the ``map`` property but the ``<clinit>`` block would still create
    an instance of ``ConcurrentHashMap``. The solution here is to refactor the property into its own file and then
    annotate the file itself as ``@DeleteForDJVM`` instead.

Non-Deterministic Function Stubs
    Sometimes it is impossible to delete a function entirely. Or a function may have some non-deterministic code
    embedded inside it that cannot be removed. For these rare cases, there is the ``@StubOutForDJVM``
    annotation:

    .. literalinclude:: ../../core/src/main/kotlin/net/corda/core/StubOutForDJVM.kt
        :language: kotlin
        :start-after: DOCSTART 01
        :end-before: DOCEND 01
    ..

    This annotation instructs ``JarFilter`` to replace the function's body with either an empty body (for functions
    that return ``void`` or ``Unit``) or one that throws ``UnsupportedOperationException``. For example:

    .. sourcecode:: kotlin

        fun necessaryCode() {
            nonDeterministicOperations()
            otherOperations()
        }

        @StubOutForDJVM
        private fun nonDeterministicOperations() {
            // etc
        }

    ..