* In order to validate T2, A asks its enclave whether T2 is valid.
* The enclave sees that T2 depends on T1, so it consults its sealed ledger whether it contains T1.
* If it does then this means T1 has been verified already, so the enclave moves on to the verification of T2.
* If the ledger doesn't contain T1 then the enclave needs to retrieve it from node B.
* In order to do this A's enclave needs to prove to B's enclave that it is indeed a trusted enclave B can provision T1
to. This proof is what the attestation process provides.
* Attestation is done in the clear: (TODO attestation diagram)
* A's enclave generates a keypair, the public part of which is sent to Node B in a datastructure signed by Intel,
this is called the quote(1).
* Node B's XFlow may do various checks on this datastructure that cannot be performed by B's enclave, for example
checking of the timeliness of Intel's signature(2).
* Node B's XFlow then forwards the quote to B's enclave, which will check Intel's signature and whether it trusts A'
s enclave. For the sake of simplicity we can assume this to be strict check that A is running the exact same
enclave B is.
* At this point B's enclave has established trust in A's enclave, and has the public part of the key generated by A'
s enclave.
* The nodes repeat the above process the other way around so that A's enclave establishes trust in B's and gets hold
of B's public key(3).
* Now they proceed to perform an ephemeral Diffie-Hellman key exchange using the keys in the quotes(4).
* The ephemeral key is then used to encrypt further communication. Beyond this point the nodes' flows (and anything
outside of the enclaves) have no way of seeing what data is being exchanged, all the nodes can do is forward the
encrypted messages.
* Once attestation is done B's enclave provisions T1 to A's enclave using the DH key. If there are further
dependencies those would be provisioned as well.
* A's enclave then proceeds to verify T1 using the embedded deterministic JVM to run XContract. The verified
transaction is then sealed to disk(5). We repeat this for T2.
* If verification or attestation fails at any point the enclave returns to A's XFlow with a failure. Otherwise if all
is good the enclave returns with a success. At this point A's XFlow knows that T2 is valid, but hasn't seen T1 in
the clear.
(1) This is simplified, the actual protocol is a bit different. Namely the quote is not generated every time A requires provisioning, but is rather generated periodically.
(2) There is a way to do this check inside the enclave, however it requires switching on of the Intel ME which in general isn't available on machines in the cloud and is known to have vulnerabilities.
(3) We need symmetric trust even if the secrets seem to only flow from B to A. Node B may try to fake being an enclave to fish for information from A.
(4) The generated keys in the quotes are used to authenticate the respective parts of the DH key exchange.
(5) Sealing means encryption of data using a key unique to the enclave and CPU. The data may be subsequently unsealed (decrypted) by the enclave, even if the enclave was restarted. Also note that there is another layer of abstraction needed which we don't detail here, needed for redundancy of the encryption key.
To summarise, the journey of T1 is:
1. Initially it's sitting encrypted in B's storage.
2. B's enclave decrypts it using its seal key specific to B's enclave + CPU combination.
3. B's enclave encrypts it using the ephemeral DH key.
4. The encrypted transaction is sent to A. The safety of this (namely that A's enclave doesn't reveal the transaction to node A) hinges on B's enclave's trust in A's enclave, which is expressed as a check of A's enclave measurement during attestation, which in turn requires auditing of A's enclave code and reproducing of the measurement.
5. A's enclave decrypts the transaction using the DH key.
6. A's enclave verifies the transaction using a deterministic JVM.
7. A's enclave encrypts the transaction using A's seal key specific to A's enclave + CPU combination.
8. The encrypted transaction is stored in A's storage.
As we can see in this model each non-notary node runs their own SGX enclave and related storage. Validation of the
backchain happens by secure provisioning of it between enclaves, plus subsequent verification and storage. However
there is one important thing missing from the example (actually it has several, but those are mostly technical detail):
the notary!
In reality we cannot establish the full validity of T2 at this point of negotiation, we need to first notarise it.
This model gives us some flexibility in this regard: we can use a validating notary (also running SGX) or a
non-validating one. This indicates that the enclave API should be split in two, mirroring the signature check choice
in SignedTransaction.verify. Only when the transaction is fully signed and notarised should it be persisted (sealed).
This model has both advantages and disadvantages. On one hand it is the closest to what we have now - we (and users)
are familiar with this model, we can fairly easily nest it into the existing codebase and it gives us flexibility with
regards to notary modes. On the other hand it is a compromising answer to the regulatory problem. If we use non-
validating notaries then the backchain storage is restricted to participants, however consider the following example:
if we have a transaction X that parties A and B can process legally, but a later transaction Y that has X in its
backchain is sent for verification to party C, then C will process and store X as well, which may be illegal.
## Privacy model + notary provisioning
This model would work similarly to the previous one, except non-notary nodes wouldn't need to run SGX or care about
storage of the encrypted ledger, it would all be done in notary nodes. Nodes would connect to SGX capable notary nodes,
and after attestation the nodes can be sure that the notary has run verification before signing.
This fixes the choice of using validating notaries, as notaries would be the only entities capable of verification:
only they have access to the full backchain inside enclaves.
Note that because we still provision the full backchain between notary members for verification, we don't necessarily
need a BFT consensus on validity - if an enclave is compromised an invalid transaction will be detected at the next
backchain provisioning.
This model reduces the number of responsibilities of a non-notary node, in particular it wouldn't need to provide
storage for the backchain or verification, but could simply trust notary signatures. Also it wouldn't need to host SGX
enclaves, only partake in the DH exchange with notary enclaves. The node's responsibilities would be reduced to the
orchestration of ledger updates (flows) and related bookkeeping (vault, network map). This split would also enable us
to be flexible with regards to the update orchestration: trust in the validity of the ledger would cease to depend on
the transaction resolution currently embedded into flows - we could provide a from-scratch light-weight implementation
of a "node" (say a mobile app) that doesn't use flows and related code at all, it just needs to be able to connect to
notary enclaves to notarise, validity is taken care of by notaries.
Note that although we wouldn't require validation checks from non-notary nodes, in theory it would be safe to allow
them to do so (if they want a stronger-than-BFT guarantee).
Of course this model has disadvantages too. From the regulatory point of view it is a strictly worse solution than the
non-notary provisioning model: the backchain would be provisioned between notary nodes not owned by actual
participants in the backchain. It also disables us from using non-validating notaries.
## Integrity model + non-notary provisioning
In this model we would trust SGX-backed signatures and related attestation datastructures (quote over signature key
signed by Intel) as proof of validity. When node A and B are negotiating a transaction it's enough to provision SGX
signatures over the dependency hashes to one another, there's no need to provision the full backchain.
This sounds very simple and efficient, and it's even more private than the privacy model as we're only passing
signatures around, not transactions. However there are a couple of issues that need addressing: If an SGX enclave is
compromised a malicious node can provide a signature over an invalid transaction that checks out, and nobody will ever
know about it, because the original transaction will never be verified. One way we can mitigate this is by requiring a
BFT consensus signature, or perhaps a threshold signature is enough. We could decouple verification into "verifying
oracles" which verify in SGX and return signatures over transaction hashes, and require a certain number of them to
convince the notary to notarise and subsequent nodes to trust validity. Another issue is enclave updates. If we find a
vulnerability in an enclave and update it, what happens to the already signed backchain? Historical transactions have
signatures that are rooted in SGX quotes belonging to old untrusted enclave code. One option is to simply have a
cutoff date before which we accept old signatures. This requires a consensus-backed timestamp on the notary signature.
Another option would be to keep the old ledger around and re-verify it with the new enclaves. However if we do this we
lose the benefits of the integrity model - we get back the regulatory issue, and we don't gain the performance benefits.
## Integrity model + notary provisioning
This is similar to the previous model, only once again non-notary nodes wouldn't need to care about verifying or
collecting proofs of validity before sending the transaction off for notarisation. All of the complexity would be
hidden by notary nodes, which may use validating oracles or perhaps combine consensus over validity with consensus
over spending. This model would be a very clean separation of concerns which solves the regulatory problem (almost)
and is quite efficient as we don't need to keep provisioning the chain. One potential issue with regards to regulation
is the tip of the ledger (the transaction being notarised) - this is sent to notaries and although it is not stored it
may still be against the law to receive it and hold it in volatile memory, even inside an enclave. I'm unfamiliar with
the legal details of whether this is good enough. If this is an issue, one way we could address this would be to scope
the validity checks required for notarisation within legal boundaries and only require "full" consensus on the
spentness check. Of course this has the downside that ledger participants outside of the regulatory boundary need to
trust the BFT-SGX of the scope. I'm not sure whether it's possible to do any better, after all we can't send the
transaction body outside the scope in any shape or form.
## Threat model
In all models we have the following actors, which may or may not overlap depending on the model:
* Notary quorum members
* Non-notary nodes/entities interacting with the ledger
* Identities owning the verifying enclave hosting infrastructure
* Identities owning the encrypted ledger/signature storage infrastructure
* R3 = enclave whitelisting identity
* Network Map = contract whitelisting identity
* Intel
We have two major ways of compromise:
* compromise of a non-enclave entity (notary, node, R3, Network Map, storage)
* compromise of an enclave.
In the case of **notaries** compromise means malicious signatures, for **nodes** it's malicious transactions, for **R3**
it's signing malicious enclaves, for **Network Map** it's signing malicious contracts, for **storage** it's read-write
access to encrypted data, and for **Intel** it's forging of quotes or signing over invalid ones.
A compromise of an **enclave** means some form of access to the enclave's temporary identity key. This may happen
through direct hardware compromise (extracting of fuse values) and subsequent forging of a quote, or leaking of secrets
through weakness of the enclave-host boundary or other side-channels like Spectre(hacking). In any case it allows an
adversary to impersonate an enclave and therefore to intercept enclave traffic and forge signatures.
The actors relevant to SGX are enclave hosts, storage infrastructure owners, regular nodes and R3.
* **Enclave hosts**: enclave code is specifically written with malicious (compromised) hosts in mind. That said we
cannot be 100% secure against yet undiscovered side channel attacks and other vulnerabilities, so we need to be
prepared for the scenario where enclaves get compromised. The privacy model effectively solves this problem by
always provisioning and re-verifying the backchain. An impersonated enclave may be able to see what's on the ledger,
but tampering with it will not check out at the next provisioning. On the other hand if a compromise happens in the
integrity model an attacker can forge a signature over validity. We can mitigate this with a BFT guarantee by
requiring a consensus over validity. This way we effectively provide the same guarantee for validity as notaries
provide with regards to double spend.
* **Storage infrastructure owner**:
* A malicious actor would need to crack the encryption key to decrypt transactions
or transaction signatures. Although this is highly unlikely, we can mitigate by preparing for and forcing of key
updates (i.e. we won't provision new transactions to enclaves using old keys).
* What an attacker *can* do is simply erase encrypted data (or perhaps re-encrypt as part of ransomware), blocking
subsequent resolution and verification. In the non-notary provisioning models we can't really mitigate this as the
tip of the ledger (or signature over) may only be stored by a single non-notary entity (assumed to be compromised).
However if we require consensus over validity between notary or non-notary entities (e.g. validating oracles) then
this implicitly provides redundancy of storage.
* Furthermore storage owners can spy on the enclave's activity by observing access patterns to the encrypted blobs.
We can mitigate by implementing ORAM storage.
* **Regular nodes**: if a regular node is compromised the attacker may gain access to the node's long term key that
allows them to Diffie-Hellman with an enclave, or get the ephemeral DH value calculated during attestation directly.
This means they can man-in-the-middle between the node and the enclave. From the ledger's point of view we are
prepared for this scenario as we never leak sensitive information to the node from the enclave, however it opens the
possibility that the attacker can fake enclave replies (e.g. validity checks) and can sniff on secrets flowing from
the node to the enclave. We can mitigate the fake enclave replies by requiring an extra signature on messages.
Sniffing cannot really be mitigated, but one could argue that if the transient DH key (that lives temporarily in
volatile memory) or long term key (that probably lives in an HSM) was leaked then the attacker has access to node
secrets anyway.
* **R3**: the entity that's whitelisting enclaves effectively controls attestation trust, which means they can
backdoor the ledger by whitelisting a secret-revealing/signature-forging enclave. One way to mitigate this is by
requiring a threshold signature/consensus over new trusted enclave measurements. Another way would be to use "canary"
keys controlled by neutral parties. These parties' responsibility would simply be to publish enclave measurements (and
perhaps the reproducing build) to the public before signing over them. The "publicity" and signature would be checked
during attestation, so a quote with a non-public measurement would be rejected. Although this wouldn't prevent
backdoors (unless the parties also do auditing), it would make them public.
* **Intel**: There are two ways a compromised Intel can interact with the ledger maliciously, both provide a backdoor.
* It can sign over invalid quotes. This can be mitigated by implementing our own attestation service. Intel told us
we'll be able to do this in the future (by downloading a set of certificates tied to CPU+CPUSVN combos that may be
used to check QE signatures).
* It can produce valid quotes without an enclave. This is due to the fact that they store one half of the SGX-
specific fuse values in order to validate quotes flexibly. One way to circumvent this would be to only use the
other half of the fuse values (the seal values) which they don't store (or so they claim). However this requires
our own "enrollment" process of CPUs where we replicate the provisioning process based off of seal values and
verify manually that the provisioning public key comes from the CPU. And even if we do this all we did was move
the requirement of trust from Intel to R3.
Note however that even if an attacker compromises Intel and decides to backdoor they would need to connect to the
ledger participants in order to take advantage. The flow framework and the business network concept act as a form of
ACL on data that would make an Intel backdoor quite useless.
## Summary
As we can see we have a number of options here, all of them have advantages and disadvantages.
#### Privacy + non-notary
**Pros**:
* Closest to our current non-SGX model
* Strong guarantee of validity
* Flexible with respect to notary modes
**Cons**:
* Regulatory problem about provisioning of ledger
* Relies on ledger participants to do validation checks
* No redundancy across ledger participants
#### Privacy + notary
**Pros**:
* Strong guarantee of validity
* Separation of concerns, allows lightweight ledger participants
* Redundancy across notary nodes
**Cons**:
* Regulatory problem about provisioning of ledger
#### Integrity + non-notary
**Pros**:
* Efficient validity checks
* No storage of sensitive transaction body only signatures
* Relies on ledger participants to do validation checks
* No redundancy across ledger participants
#### Integrity + notary
**Pros**:
* Efficient validity check
* No storage of sensitive transaction body only signatures
* Separation of concerns, allows lightweight ledger participants
* Redundancy across notary nodes
**Cons**:
* Only BFT guarantee over validity
* Temporary storage of transaction in RAM may be against regulation
Personally I'm strongly leaning towards an integrity model where SGX compromise is mitigated by a BFT consensus over validity (perhaps done by a validating oracle cluster). This would solve the regulatory problem, it would be efficient and the infrastructure would have a very clean separation of concerns between notary and non-notary nodes, allowing lighter-weight interaction with the ledger.
