corda/docs/source/data-model.rst

Data model

This article covers the data model: how states, transactions and code contracts interact with each other and how they are represented in software.

Overview

We begin with the idea of a global ledger. In our model although the ledger is shared, it is not always the case that transactions and ledger entries are globally visible. In cases where a set of transactions stays within a small subgroup of users it should be possible to keep the relevant data purely within that group.

To ensure consistency in a global, shared system where not all data may be visible to all participants, we rely heavily on secure hashes like SHA-256 to identify things. The ledger is defined as a set of immutable states, which are created and destroyed by digitally signed transactions. Each transaction points to a set of states that it will consume/destroy, these are called inputs, and contains a set of new states that it will create, these are called outputs.

States contain arbitrary data, but they always contain at minimum a hash of the bytecode of a contract code file, which is a program expressed in JVM byte code that runs sandboxed inside a Java virtual machine. Contract code (or just "contracts" in the rest of this document) are globally shared pieces of business logic.

Note

In the current code dynamic loading of contracts is not implemented, so states currently point at statically created object instances. This will change in the near future.

Contracts define a verify function, which is a pure function given the entire transaction as input. To be considered valid, the transaction must be accepted by the verify function of every contract pointed to by the input and output states.

Beyond inputs and outputs, transactions may also contain commands, small data packets that the platform does not interpret itself but which can parameterise execution of the contracts. They can be thought of as arguments to the verify function. Each command has a list of public keys associated with it. The platform ensures that the transaction is signed by every key listed in the commands before the contracts start to execute. Thus, a verify function can trust that all listed keys have signed the transaction but is responsible for verifying that any keys required for the transaction to be valid from the verify function's perspective are included in the list. Public keys may be random/identityless for privacy, or linked to a well known legal identity, for example via a public key infrastructure (PKI).

Note

Linkage of keys with identities via a PKI is only partially implemented in the current code.

Commands are always embedded inside a transaction. Sometimes, there's a larger piece of data that can be reused across many different transactions. For this use case, we have attachments. Every transaction can refer to zero or more attachments by hash. Attachments are always ZIP/JAR files, which may contain arbitrary content. These files are then exposed on the classpath and so can be opened by contract code in the same manner as any JAR resources would be loaded.

Note

Attachments must be opened explicitly in the current code.

Note that there is nothing that explicitly binds together specific inputs, outputs, commands or attachments. Instead it's up to the contract code to interpret the pieces inside the transaction and ensure they fit together correctly. This is done to maximise flexibility for the contract developer.

Transactions may sometimes need to provide a contract with data from the outside world. Examples may include stock prices, facts about events or the statuses of legal entities (e.g. bankruptcy), and so on. The providers of such facts are called oracles and they provide facts to the ledger by signing transactions that contain commands they recognise, or by creating signed attachments. The commands contain the fact and the signature shows agreement to that fact.

Time is also modelled as a fact, with the signature of a special kind of service called a notary. A notary is a (very likely) decentralised service which fulfils the role that miners play in other blockchain systems: notaries ensure only one transaction can consume any given output. Additionally they may verify a timestamping command placed inside the transaction, which specifies a time window in which the transaction is considered valid for notarisation. The time window can be open ended (i.e. with a start but no end or vice versa). In this way transactions can be linked to the notary's clock.

It is possible for a single Corda network to have multiple competing notaries. Each state points to the notary that controls it. Whilst a single transaction may only consume states if they are all controlled by the same notary, a special type of transaction is provided that moves a state (or set of states) from one notary to another.

Note

Currently the platform code will not re-assign states to a single notary as needed for you, in case of a mismatch. This is a future planned feature.

As the same terminology often crops up in different distributed ledger designs, let's compare this to other systems you may be familiar with. The key differences are:

• Improved contract flexibility vs Bitcoin
• Improved scalability vs Ethereum, as well as ability to keep parts of the transaction graph private (yet still uniquely addressable)
• No reliance on proof of work
• Re-use of existing sandboxing virtual machines
• Use of type safe GCd implementation languages
• Simplified auditing

Comparison with Bitcoin

Similarities:

• The basic notion of immutable states that are consumed and created by transactions is the same.
• The notion of transactions having multiple inputs and outputs is the same. Bitcoin sometimes refers to the ledger as the unspent transaction output set (UTXO set) as a result.
• Like in Bitcoin, a contract is pure function. Contracts do not have storage or the ability to interact with anything. Given the same transaction, a contract's accept function always yields exactly the same result.
• Bitcoin output scripts are parameterised by the input scripts in the spending transaction. This is somewhat similar to our notion of a command.
• Bitcoin has a global distributed notary service; that's the famous block chain. However, there is only one. Whilst there is a notion of a "side chain", this isn't integrated with the core Bitcoin data model and thus adds large amounts of additional complexity meaning in practice side chains are not used.
• Bitcoin transactions, like ours, refer to the states they consume by using a (txhash, index) pair. The Bitcoin protocol calls these "outpoints". In our code they are known as StateRefs but the concept is identical.
• Bitcoin transactions have an associated timestamp (the time at which they are mined).

Differences:

• A Bitcoin transaction has a single, rigid data format. A "state" in Bitcoin is always a (quantity of bitcoin, script) pair and cannot hold any other data. Some people have been known to try and hack around this limitation by embedding data in semi-standardised places in the contract code so the data can be extracted through pattern matching, but this is a poor approach. Our states can include arbitrary typed data.
• A Bitcoin transaction's acceptance is controlled only by the contract code in the consumed input states. In practice this has proved limiting. Our transactions invoke not only input contracts but also the contracts of the outputs.
• A Bitcoin script can only be given a fixed set of byte arrays as the input. This means there's no way for a contract to examine the structure of the entire transaction, which severely limits what contracts can do.
• Our contracts are Turing-complete and can be written in any ordinary programming language that targets the JVM.
• Our transactions and contracts get their time from an attached timestamp rather than a block. This is important given that we use block-free conflict resolution algorithms. The timestamp can be arbitrarily precise.
• We use the term "contract" to refer to a bundle of business logic that may handle various different tasks, beyond transaction verification. For instance, currently our contracts also include code for creating valid transactions (this is often called "wallet code" in Bitcoin).

Comparison with Ethereum

Similarities:

• Like Ethereum, code runs inside a relatively powerful virtual machine and can contain complex logic. Non-assembly based programming languages can be used for contract programming.
• They are both intended for the modelling of many different kinds of financial contract.

Differences:

• The term "contract" in Ethereum refers to an instantiation of a program that is replicated and maintained by every participating node. This instantiation is very much like an object in an OO program: it can receive and send messages, update local storage and so on. In contrast, we use the term "contract" to refer to a set of functions, only one of which is a part of keeping the system synchronised (the verify function). That function is pure and stateless i.e. it may not interact with any other part of the system whilst executing.
• There is no notion of an "account", as there is in Ethereum.
• As contracts don't have any kind of mutable storage, there is no notion of a "message" as in Ethereum.
• Ethereum claims to be a platform not only for financial logic, but literally any kind of application at all. Our platform considers non-financial applications to be out of scope.

Rationale for and tradeoffs in adopting a UTXO-style model

As discussed above, Corda uses the so-called "UTXO set" model (unspent transaction output). In this model, the database does not track accounts or balances. Instead all database entries are immutable. An entry is either spent or not spent but it cannot be changed. In Bitcoin, spentness is implemented simply as deletion – the inputs of an accepted transaction are deleted and the outputs created.

This approach has some advantages and some disadvantages, which is why some platforms like Ethereum have tried (or are trying) to abstract this choice away and support a more traditional account-like model. We have explicitly chosen not to do this and our decision to adopt a UTXO-style model is a deliberate one. In the section below, the rationale for this decision and its pros and cons of this choice are outlined.

Rationale

Corda, in common with other blockchain-like platforms, is designed to bring parties to shared sets of data into consensus as to the existence, content and allowable evolutions of those data sets. However, Corda is designed with the explicit aim of avoiding, to the extent possible, the scalability and privacy implications that arise from those platforms' decisions to adopt a global broadcast model.

Whilst the privacy implications of a global consensus model are easy to understand, the scalability implications are perhaps more subtle, yet serious. In a consensus system, it is critical that all processors of a transaction reach precisely the same conclusion as to its effects. In situations where two transactions may act on the same data set, it means that the two transactions must be processed in the same order by all nodes. If this were not the case then it would be possible to devise situations where nodes processed transactions in different orders and reached different conclusions as to the state of the system. It is for this reason that systems like Ethereum effectively run single-threaded, meaning the speed of the system is limited by the single-threaded performance of the slowest machine on the network.

In Corda, we assume the data being processed represents financial agreements between identifiable parties and that these institutions will adopt the system only if a significant number of such agreements can be managed by the platform. As such, the system has to be able to support parallelisation of execution to the greatest extent possible, whilst ensuring correct transaction ordering when two transactions seek to act on the same piece of shared state.

To achieve this, we must minimise the number of parties who need to receive and process copies of any given transaction and we must minimise the extent to which two transactions seek to mutate (or supersede) any given piece of shared state.

A key design decision, therefore, is what should be the most atomic unit of shared data in the system. This decision also has profound privacy implications: the more coarsely defined the shared data units, the larger the set of actors who will likely have a stake in its accuracy and who must process and observe any update to it.

This becomes most obvious when we consider two models for representing cash balances and payments.

A simple account model for cash would define a data structure that maintained a balance at a particular bank for each "account holder". Every holder of a balance would need a copy of this structure and would thus need to process and validate every payment transaction, learning about everybody else's payments and balances in the process. All payments across that set of accounts would have to be single-threaded across the platform, limiting maximum throughput.

A more sophisticated example might create a data structure per account holder. But, even here, I would leak my account balance to anybody to whom I ever made a payment and I could only ever make one payment at a time, for the same reasons above.

A UTXO model would define a data structure that represented an instance of a claim against the bank. An account holder could hold many such instances, the aggregate of which would reveal their balance at that institution. However, the account holder now only needs to reveal to their payee those instances consumed in making a payment to that payee. This also means the payer could make several payments in parallel. A downside is that the model is harder to understand. However, we consider the privacy and scalability advantages to overwhelm the modest additional cognitive load this places on those attempting to learn the system.

In what follows, further advantages and disadvantages of this design decision are explored.

Pros

The UTXO model has these advantages:

• Immutable ledger entries gives the usual advantages that a more functional approach brings: it's easy to do analysis on a static snapshot of the data and reason about the contents.
• Because there are no accounts, it's very easy to apply transactions in parallel even for high traffic legal entities assuming sufficiently granular entries.
• Transaction ordering becomes trivial: it is impossible to mis-order transactions due to the reliance on hash functions to identify previous states. There is no need for sequence numbers or other things that are hard to provide in a fully distributed system.
• Conflict resolution boils down to the double spending problem, which places extremely minimal demands on consensus algorithms (as the variable you're trying to reach consensus on is a set of booleans).

Cons

It also comes with some pretty serious complexities that in practice must be abstracted from developers:

• Representing numeric amounts using immutable entries is unnatural. For instance, if you receive $1000 and wish to send someone $100, you have to consume the $1000 output and then create two more: a $100 for the recipient and $900 back to yourself as change. The fact that this happens can leak private information to an observer.
• Because users do need to think in terms of balances and statements, you have to layer this on top of the underlying ledger: you can't just read someone's balance out of the system. Hence, the "wallet" / position manager. Experience from those who have developed wallets for Bitcoin and other systems is that they can be complex pieces of code, although the bulk of wallets' complexity in public systems is handling the lack of finality (and key management).
• Whilst transactions can be applied in parallel, it is much harder to create them in parallel due to the need to strictly enforce a total ordering.

With respect to parallel creation, if the user is single threaded this is fine, but in a more complex situation where you might want to be preparing multiple transactions in flight this can prove a limitation – in the worst case where you have a single output that represents all your value, this forces you to serialise the creation of every transaction. If transactions can be created and signed very fast that's not a concern. If there's only a single user, that's not a concern.

Both cases are typically true in the Bitcoin world, so users don't suffer from this much. In the context of a complex business with a large pool of shared funds, in which creation of transactions may be very slow due to the need to get different humans to approve a tx using a signing device, this could quickly lead to frustrating conflicts where someone approves a transaction and then discovers that it has become a double spend and they must sign again. In the absolute worst case you could get a form of human livelock.

The tricky part about solving these problems is that the simplest way to express a payment request ("send me $1000 to public key X") inherently results in you receiving a single output, which then can prove insufficiently granular to be convenient. In the Bitcoin space Mike Hearn and Gavin Andresen designed "BIP 70" to solve this: it's a simple binary format for requesting a payment and specifying exactly how you'd like to get paid, including things like the shape of the transaction. It may seem that it's an over complex approach: could you not just immediately respend the big output back to yourself in order to split it? And yes, you could, until you hit scenarios like "the machine requesting the payment doesn't have the keys needed to spend it", which turn out to be very common. So it's really more effective for a recipient to be able to say to the sender, "here's the kind of transaction I want you to send me". The flow framework <flow-state-machines> may provide a vehicle to make such negotiations simpler.

A further challenge is privacy. Whilst our goal of not sending transactions to nodes that don't "need to know" helps, to verify a transaction you still need to verify all its dependencies and that can result in you receiving lots of transactions that involve random third parties. The problems start when you have received lots of separate payments and been careful not to make them linkable to your identity, but then you need to combine them all in a single transaction to make a payment.

Mike Hearn wrote an article about this problem and techniques to minimise it in this article from 2013. This article coined the term "merge avoidance", which has never been implemented in the Bitcoin space, although not due to lack of practicality.

A piece of future work for the wallet implementation will be to implement automated "grooming" of the wallet to "reshape" outputs to useful/standardised sizes, for example, and to send outputs of complex transactions back to their issuers for reissuance to "sever" long privacy-breaching chains.

Finally, it should be noted that some of the issues described here are not really "cons" of the UTXO model; they're just fundamental. If you used many different anonymous accounts to preserve some privacy and then needed to spend the contents of them all simultaneously, you'd hit the same problem, so it's not something that can be trivially fixed with data model changes.