CORDA-1709 - The MVP blob inspector, able to inspect network service blobs (#3503)

* Cleanup and improvements to the serialisation format of JacksonSupport (needed for CORDA-1238) (#3102)

Also deprecated all the public members that shouldn't have leaked into the public API.

(cherry picked from commit 3bb95c3)

* CORDA-1238: Updated JacksonSupport to support SerializedBytes, CertPath, X509Certificate and the signature classes (#3145)

SerializedBytes are first converted to the object it represents before being serialised as a pojo.

These changes will be needed to support the the blob inspector when it will output to YAML/JSON.

(cherry picked from commit b031e66)

* Cherry picked part of commit 824adca to port over *only* the JackSupport refactoring.

* CORDA-1238: Moved the blob inspector out of experimental and wired it to JackonSupport (#3224)

The existing output format was not complete and so was deleted to avoid it becoming a tech debt. We can always resurrect it at a later point.

(cherry picked from commit 4e0378d)

* Added back support for parsing OpaqueBytes as UTF-8 strings in JacksonSupport (#3240)

(cherry picked from commit d772bc8)

* Cleaned up blob inspector doc (#3284)

(cherry picked from commit b7fbebb)

* Blobinspector: trace level logging with --verbose (#3313)

(cherry picked from commit 6a2e50b)

* Cherry picked part of commit 3046843 to fix issue with --version

* Fixes to the api file

docs/source/blob-inspector.rst

@@ -0,0 +1,67 @@
+Blob Inspector
+==============
+
+There are many benefits to having a custom binary serialisation format (see :doc:`serialization` for details) but one
+disadvantage is the inability to view the contents in a human-friendly manner. The blob inspector tool alleviates this issue
+by allowing the contents of a binary blob file (or URL end-point) to be output in either YAML or JSON. It uses
+``JacksonSupport`` to do this (see :doc:`json`).
+
+The latest version of the tool can be downloaded from `here <https://www.corda.net/downloads/>`_.
+
+To run simply pass in the file or URL as the first parameter:
+
+``java -jar blob-inspector.jar <file or URL>``
+
+Use the ``--help`` flag for a full list of command line options.
+
+When inspecting your custom data structures, there's no need to include the jars containing the class definitions for them
+in the classpath. The blob inspector (or rather the serialization framework) is able to synthesis any classes found in the
+blob that aren't on the classpath.
+
+SerializedBytes
+~~~~~~~~~~~~~~~
+
+One thing to note is that the binary blob may contain embedded ``SerializedBytes`` objects. Rather than printing these
+out as a Base64 string, the blob inspector will first materialise them into Java objects and then output those. You will
+see this when dealing with classes such as ``SignedData`` or other structures that attach a signature, such as the
+``nodeInfo-*`` files or the ``network-parameters`` file in the node's directory. For example, the output of a node-info
+file may look like:
+
+
+**-\\-format=YAML**
+::
+
+    net.corda.nodeapi.internal.SignedNodeInfo
+    ---
+    raw:
+      class: "net.corda.core.node.NodeInfo"
+      deserialized:
+        addresses:
+        - "localhost:10005"
+        legalIdentitiesAndCerts:
+        - "O=BankOfCorda, L=London, C=GB"
+        platformVersion: 4
+        serial: 1527851068715
+    signatures:
+    - !!binary |-
+      VFRy4frbgRDbCpK1Vo88PyUoj01vbRnMR3ROR2abTFk7yJ14901aeScX/CiEP+CDGiMRsdw01cXt\nhKSobAY7Dw==
+
+**-\\-format=JSON**
+::
+
+    net.corda.nodeapi.internal.SignedNodeInfo
+    {
+      "raw" : {
+        "class" : "net.corda.core.node.NodeInfo",
+        "deserialized" : {
+          "addresses" : [ "localhost:10005" ],
+          "legalIdentitiesAndCerts" : [ "O=BankOfCorda, L=London, C=GB" ],
+          "platformVersion" : 4,
+          "serial" : 1527851068715
+        }
+      },
+      "signatures" : [ "VFRy4frbgRDbCpK1Vo88PyUoj01vbRnMR3ROR2abTFk7yJ14901aeScX/CiEP+CDGiMRsdw01cXthKSobAY7Dw==" ]
+    }
+
+Notice the file is actually a serialised ``SignedNodeInfo`` object, which has a ``raw`` property of type ``SerializedBytes<NodeInfo>``.
+This property is materialised into a ``NodeInfo`` and is output under the ``deserialized`` field.

docs/source/changelog.rst

@@ -7,6 +7,27 @@ release, see :doc:`upgrade-notes`.
 Unreleased
 ==========
 
+* Changes to the JSON/YAML serialisation format from ``JacksonSupport``, which also applies to the node shell:
+
+  * ``Instant`` and ``Date`` objects are serialised as ISO-8601 formatted strings rather than timestamps
+  * ``PublicKey`` objects are serialised and looked up according to their Base58 encoded string
+  * ``Party`` objects can be deserialised by looking up their public key, in addition to their name
+  * ``NodeInfo`` objects are serialised as an object and can be looked up using the same mechanism as ``Party``
+  * ``NetworkHostAndPort`` serialised according to its ``toString()``
+  * ``PartyAndCertificate`` is serialised as the name
+  * ``SerializedBytes`` is serialised by materialising the bytes into the object it represents, and then serialising that
+    object into YAML/JSON
+  * ``X509Certificate`` is serialised as an object with key fields such as ``issuer``, ``publicKey``, ``serialNumber``, etc.
+    The encoded bytes are also serialised into the ``encoded`` field. This can be used to deserialise an ``X509Certificate``
+    back.
+  * ``CertPath`` objects are serialised as a list of ``X509Certificate`` objects.
+
+* ``fullParties`` boolean parameter added to ``JacksonSupport.createDefaultMapper`` and ``createNonRpcMapper``. If ``true``
+  then ``Party`` objects are serialised as JSON objects with the ``name`` and ``owningKey`` fields. For ``PartyAndCertificate``
+  the ``certPath`` is serialised.
+
+* Several members of ``JacksonSupport`` have been deprecated to highlight that they are internal and not to be used
+
 * ``ServiceHub`` and ``CordaRPCOps`` can now safely be used from multiple threads without incurring in database transaction problems.
 
 * Fixed an issue preventing out of process nodes started by the ``Driver`` from logging to file.

docs/source/shell.rst

@@ -148,8 +148,8 @@ Where ``newCampaign`` is a parameter of type ``Campaign``.
 
 Mappings from strings to types
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Several parameter types can automatically be mapped from strings. See the `defined parsers`_ for more information. We
-cover the most common types here.
+In addition to the types already supported by Jackson, several parameter types can automatically be mapped from strings.
+We cover the most common types here.
 
 Amount
 ~~~~~~
@@ -158,23 +158,44 @@ A parameter of type ``Amount<Currency>`` can be written as either:
 * A dollar ($), pound (£) or euro (€) symbol followed by the amount as a decimal
 * The amount as a decimal followed by the ISO currency code (e.g. "100.12 CHF")
 
+SecureHash
+~~~~~~~~~~
+A parameter of type ``SecureHash`` can be written as a hexadecimal string: ``F69A7626ACC27042FEEAE187E6BFF4CE666E6F318DC2B32BE9FAF87DF687930C``
+
 OpaqueBytes
 ~~~~~~~~~~~
-A parameter of type ``OpaqueBytes`` can be provided as a string, which will be automatically converted to
-``OpaqueBytes``.
+A parameter of type ``OpaqueBytes`` can be provided as a UTF-8 string.
+
+PublicKey and CompositeKey
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+A parameter of type ``PublicKey`` can be written as a Base58 string of its encoded format: ``GfHq2tTVk9z4eXgyQXzegw6wNsZfHcDhfw8oTt6fCHySFGp3g7XHPAyc2o6D``.
+``net.corda.core.utilities.EncodingUtils.toBase58String`` will convert a ``PublicKey`` to this string format.
 
 Party
 ~~~~~
 A parameter of type ``Party`` can be written in several ways:
 
-* By using the node's full name: ``"O=Monogram Bank,L=Sao Paulo,C=GB"``
+* By using the full name: ``"O=Monogram Bank,L=Sao Paulo,C=GB"``
 * By specifying the organisation name only: ``"Monogram Bank"``
 * By specifying any other non-ambiguous part of the name: ``"Sao Paulo"`` (if only one network node is located in Sao
   Paulo)
+* By specifying the public key (see above)
 
-Instant
-~~~~~~~
-A parameter of type ``Instant`` can be written as follows: ``"2017-12-22T00:00:00Z"``.
+NodeInfo
+~~~~~~~~
+A parameter of type ``NodeInfo`` can be written in terms of one of its identities (see ``Party`` above)
+
+AnonymousParty
+~~~~~~~~~~~~~~
+A parameter of type ``AnonymousParty`` can be written in terms of its ``PublicKey`` (see above)
+
+NetworkHostAndPort
+~~~~~~~~~~~~~~~~~~
+A parameter of type ``NetworkHostAndPort`` can be written as a "host:port" string: ``"localhost:1010"``
+
+Instant and Date
+~~~~~~~~~~~~~~~~
+A parameter of ``Instant`` and ``Date`` can be written as an ISO-8601 string: ``"2017-12-22T00:00:00Z"``
 
 Examples
 ^^^^^^^^
@@ -258,6 +279,5 @@ The shell will be enhanced over time. The currently known limitations include:
 * The ``jul`` command advertises access to logs, but it doesn't work with the logging framework we're using
 
 .. _Yaml: http://www.yaml.org/spec/1.2/spec.html
-.. _defined parsers: api/kotlin/corda/net.corda.client.jackson/-jackson-support/index.html
 .. _Groovy: http://groovy-lang.org/
 .. _CRaSH: http://www.crashub.org/

docs/source/tools-index.rst

@@ -4,6 +4,7 @@ Tools
 .. toctree::
    :maxdepth: 1
 
+   blob-inspector
    network-simulator
    demobench
    node-explorer