Allmydata "Tahoe" Architecture

OVERVIEW

The high-level view of this system consists of three layers: the mesh, the
virtual drive, and the application that sits on top.

The lowest layer is the "mesh" or "cloud", basically a DHT (Distributed Hash
Table) which maps URIs to data. The URIs are relatively-short ascii strings
(currently about 140 bytes), and they are used as references to an immutable
arbitrary-length sequence of data bytes. This data is distributed around the
cloud in a large number of nodes, such that a statistically unlikely number
of nodes would have to be unavailable for the data to be unavailable.

The middle layer is the virtual drive: a tree-shaped data structure in which
the intermediate nodes are directories and the leaf nodes are files. Each
file contains both the URI of the file's data and all the necessary metadata
(MIME type, filename, ctime/mtime, etc) required to present the file to a
user in a meaningful way (displaying it in a web browser, or on a desktop).

The top layer is where the applications that use this virtual drive operate.
Allmydata uses this for a backup service, in which the application copies the
files to be backed up from the local disk into the virtual drive on a
periodic basis. By providing read-only access to the same virtual drive
later, a user can recover older versions of their files. Other sorts of
applications can run on top of the virtual drive, of course, anything that
has a use for a secure, robust, distributed filestore.

Note: some of the description below indicates design targets rather than
actual code present in the current release. Please take a look at roadmap.txt
to get an idea of how much of this has been implemented so far.


THE BIG CLOUD OF PEERS

Underlying the mesh/cloud is a large collection of peer nodes. These are
processes running on a wide variety of computers, all of which know about
each other in some way or another. They establish TCP connections to one
another using Foolscap, an encrypted+authenticated remote message passing
library (using TLS connections and self-authenticating identifiers called
"FURLs").

Each peer offers certain services to the others. The primary service is the
StorageServer, which offers to hold data for a limited period of time (a
"lease"). Each StorageServer has a quota, and it will reject lease requests
that would cause it to consume more space than it wants to provide. When a
lease expires, the data is deleted. Peers might renew their leases.

This storage is used to hold "shares", which are themselves used to store
files in the mesh. There are many shares for each file, typically around 100
(the exact number depends upon the tradeoffs made between reliability,
overhead, and storage space consumed). The files are indexed by a piece of
the URI called the "verifierid", which is derived from the contents of the
file. Leases are indexed by verifierid, and a single StorageServer may hold
multiple shares for the corresponding file. Multiple peers can hold leases on
the same file, in which case the shares will be kept alive until the last
lease expires. The typical lease is expected to be for one month: enough time
for interested parties to renew it, but not so long that abandoned data
consumes unreasonable space. Peers are expected to "delete" (drop leases) on
data that they know they no longer want: lease expiration is meant as a
safety measure.

In this release, peers learn about each other through the "introducer". Each
peer connects to this central introducer at startup, and receives a list of
all other peers from it. Each peer then connects to all other peers, creating
a full-mesh topology. Future versions will reduce the number of connections
considerably, to enable the mesh to scale larger than a full-mesh allows.


FILE ENCODING

When a file is to be added to the mesh, it is first encrypted using a key
that is derived from the hash of the file itself. The encrypted file is then
broken up into segments so it can be processed in small pieces (to minimize
the memory footprint of both encode and decode operations, and to increase
the so-called "alacrity": how quickly can the download operation provide
validated data to the user). Each segment is erasure coded, which creates
encoded blocks that are larger than the input segment, such that only a
subset of the output blocks are required to reconstruct the segment. These
blocks are then combined into "shares", such that a subset of the shares can
be used to reconstruct the whole file. The shares are then deposited in
StorageServers in other peers.

A tagged hash of the original file is called the "fileid", while a
differently-tagged hash of the original file provides the encryption key. A
tagged hash of the *encrypted* file is called the "verifierid", and is used
for both peer selection (described below) and to index shares within the
StorageServers on the selected peers.

The URI contains the verifierid, the encryption key, any encoding parameters
necessary to perform the eventual decoding process, and some additional
hashes that allow the download process to validate the data it receives.

On the download side, the node that wishes to turn a URI into a sequence of
bytes will obtain the necessary shares from remote nodes, break them into
blocks, use erasure-decoding to turn them into segments of crypttext, use the
decryption key to convert that into plaintext, then emit the plaintext bytes
to the output target (which could be a file on disk, or it could be streamed
directly to a web browser or media player).

All hashes use SHA256, and a different tag is used for each purpose.
Netstrings are used where necessary to insure these tags cannot be confused
with the data to be hashed. All encryption uses AES in CTR mode. The erasure
coding is performed with zfec (a python wrapper around Rizzo's FEC library).
A Merkle Hash Tree is used to validate the encoded blocks before they are fed
into the decode process, and a second tree is used to validate the shares
before they are retrieved. The hash tree root is put into the URI.


URIs

Each URI represents a specific set of bytes. Think of it like a hash
function: you feed in a bunch of bytes, and you get out a URI. The URI is
deterministically derived from the input data: changing even one bit of the
input data will result in a drastically different URI. The URI provides both
"identification" and "location": you can use it to locate a set of bytes that
are probably the same as the original file, and you can also use it to
validate that these potential bytes are indeed the ones that you were looking
for.

URIs refer to an immutable set of bytes. If you modify a file and upload the
new one to the mesh, you will get a different URI. URIs do not represent
filenames at all, just the data that a filename might point to at some given
point in time. This is why the "mesh" layer is insufficient to provide a
virtual drive: an actual filesystem requires human-meaningful names and
mutability, while URIs provide neither. URIs sit on the "global+secure" edge
of Zooko's Triangle[1]. They are self-authenticating, meaning that nobody can
trick you into using the wrong data.

The URI should be considered as a "read capability" for the corresponding
data: anyone who knows the full URI has the ability to read the given data.
There is a subset of the URI (which leaves out the encryption key and fileid)
which is called the "verification capability": it allows the holder to
retrieve and validate the crypttext, but not the plaintext. Once the
crypttext is available, the erasure-coded shares can be regenerated. This
will allow a file-repair process to maintain and improve the robustness of
files without being able to read their contents.

The lease mechanism will also involve a "delete" capability, by which a peer
which uploaded a file can indicate that they don't want it anymore. It is not
truly a delete capability because other peers might be holding leases on the
same data, and it should not be deleted until the lease count (i.e. reference
count) goes to zero, so perhaps "cancel-the-lease capability" is more
accurate. The plan is to store this capability next to the URI in the virtual
drive structure.


PEER SELECTION

When a file is uploaded, the encoded shares are sent to other peers. But to
which ones? The "peer selection" algorithm is used to make this choice.

In the current version, the verifierid is used to consistently-permute the
set of all peers (by sorting the peers by HASH(verifierid+peerid)). This
places the peers around a 2^256-sized ring, like the rim of a big clock. The
100-or-so shares are then placed around the same ring (at 0, 1/100*2^256,
2/100*2^256, ... 99/100*2^256). Imagine that we start at 0 with an empty
basket in hand and proceed clockwise. When we come to a share, we pick it up
and put it in the basket. When we come to a peer, we ask that peer if they
will give us a lease for every share in our basket.

The peer will grant us leases for some of those shares and reject others (if
they are full or almost full). If they reject all our requests, we remove
them from the ring, because they are full and thus unhelpful. Each share they
accept is removed from the basket. The remainder stay in the basket as we
continue walking clockwise.

We keep walking, accumulating shares and distributing them to peers, until
either we find a home for all shares, or there are no peers left in the ring
(because they are all full). If we run out of peers before we run out of
shares, the upload may be considered a failure, depending upon how many
shares we were able to place. The current parameters try to place 100 shares,
of which 25 must be retrievable to recover the file, and the peer selection
algorithm is happy if it was able to place at least 75 shares. These numbers
are adjustable: 25-out-of-100 means an expansion factor of 4x (every file in
the mesh consumes four times as much space when totalled across all
StorageServers), but is highly reliable (the actual reliability is a binomial
distribution function of the expected availability of the individual peers,
but in general it goes up very quickly with the expansion factor).

If the file has been uploaded before (or if two uploads are happening at the
same time), a peer might already have shares for the same file we are
proposing to send to them. In this case, those shares are removed from the
list and assumed to be available (or will be soon). This reduces the number
of uploads that must be performed.

When downloading a file, the current release just asks all known peers for
any shares they might have, chooses the minimal necessary subset, then starts
downloading and processing those shares. A later release will use the full
algorithm to reduce the number of queries that must be sent out. This
algorithm uses the same consistent-hashing permutation as on upload, but
instead of one walker with one basket, we have 100 walkers (one per share).
They each proceed clockwise until they find a peer: this peer is the most
likely to be the same one to which the share was originally uploaded, and is
put on the "A" list. The next peer that each walker encounters is put on the
"B" list, etc. All the "A" list peers are asked for any shares they might
have. If enough of them can provide a share, the download phase begins and
those shares are retrieved and decoded. If not, the "B" list peers are
contacted, etc. This routine will eventually find all the peers that have
shares, and will find them quickly if there is significant overlap between
the set of peers that were present when the file was uploaded and the set of
peers that are present as it is downloaded (i.e. if the "peerlist stability"
is high). Some limits may be imposed in large meshes to avoid querying a
million peers; this provides a tradeoff between the work spent to discover
that a file is unrecoverable and the probability that a retrieval will fail
when it couldhave succeeded if we had just tried a little bit harder. The
appropriate value of this tradeoff will depend upon the size of the mesh.

Other peer selection algorithms are being evaluated. One of them (known as
"tahoe 2") uses the same consistent hash, starts at 0 and requests one lease
per peer until it gets 100 of them. This is likely to get better overlap
(since a single insertion or deletion will still leave 99 overlapping peers),
but is non-ideal in other ways (TODO: what were they?).

Another algorithm (known as "denver airport"[2]) uses the permuted hash to
decide on an approximate target for each share, then sends lease requests via
Chord routing (to avoid maintaining a large number of long-term connections).
The request includes the contact information of the uploading node, and asks
that the node which eventually accepts the lease should contact the uploader
directly. The shares are then transferred over direct connections rather than
through multiple Chord hops. Download uses the same approach.


FILETREE: THE VIRTUAL DRIVE LAYER

The "virtual drive" layer is responsible for mapping human-meaningful
pathnames (directories and filenames) to pieces of data. The actual bytes
inside these files are referenced by URI, but the "filetree" is where the
directory names, file names, and metadata are kept.

The current release has a very simplistic filetree model. There is a single
globally-shared directory structure, which maps filename to URI. This
structure is maintained in a central node (which happens to be the same node
that houses the Introducer), by writing URIs to files in a local filesystem.

A future release (probably the next one) will offer each application the
ability to have a separate file tree. Each tree can reference others. Some
trees are redirections, while others actually contain subdirectories full of
filenames. The redirections may be mutable by some users but not by others,
allowing both read-only and read-write views of the same data. This will
enable individual users to have their own personal space, with links to
spaces that are shared with specific other users, and other spaces that are
globally visible. Eventually the application layer will present these pieces
in a way that allows the sharing of a specific file or the creation of a
"virtual CD" as easily as dragging a folder onto a user icon.

The URIs described above are "Content Hash Key" (CHK) identifiers[3], in
which the identifier refers to a specific sequence of bytes. In this project,
CHK identifiers are used for both files and immutable directories (the tree
of directory and file nodes are serialized into a sequence of bytes, which is
then uploaded and turned into a URI). There is a separate kind of upload, not
yet implemented, called SSK (short for Signed Subspace Key), in which the URI
refers to a mutable slot. Some users have a write-capability to this slot,
allowing them to change the data that it refers to. Others only have a
read-capability, merely letting them read the current contents. These SSK
slots can be used to provide mutability in the filetree, so that users can
actually change the contents of their virtual drive. Redirection nodes can
also provide mutability, such as a central service which allows a user to set
the current URI of their top-level filetree. SSK slots provide a
decentralized way to accomplish this mutability, whereas centralized
redirection nodes are more vulnerable to single-point-of-failure issues.


FILE REPAIRER

Each node is expected to explicitly drop leases on files that it knows it no
longer wants (the "delete" operation). Nodes are also expected to renew
leases on files that still exist in their filetrees. When nodes are offline
for an extended period of time, their files may decay (both because of leases
expiring and because of StorageServers going offline). A File Verifier is
used to check on the health of any given file, and a File Repairer is used to
to keep desired files alive. The two are conceptually distinct (the repairer
is run if the verifier decides it is necessary), but in practice they will be
closely related, and may run in the same process.

The repairer process does not get the full URI of the file to be maintained:
it merely gets the "repairer capability" subset, which does not include the
decryption key. The File Verifier uses that data to find out which peers
ought to hold shares for this file, and to see if those peers are still
around and willing to provide the data. If the file is not healthy enough,
the File Repairer is invoked to download the crypttext, regenerate any
missing shares, and upload them to new peers. The goal of the File Repairer
is to finish up with a full set of 100 shares.

There are a number of engineering issues to be resolved here. The bandwidth,
disk IO, and CPU time consumed by the verification/repair process must be
balanced against the robustness that it provides to the mesh. The nodes
involved in repair will have very different access patterns than normal
nodes, such that these processes may need to be run on hosts with more memory
or network connectivity than usual. The frequency of repair runs directly
affects the resources consumed. In some cases, verification of multiple files
can be performed at the same time, and repair of files can be delegated off
to other nodes.

The security model we are currently using assumes that peers who claim to
hold a share will actually provide it when asked. (We validate the data they
provide before using it in any way, but if enough peers claim to hold the
data and are wrong, the file will not be repaired, and may decay beyond
recoverability). There are several interesting approaches to mitigate this
threat, ranging from challenges to provide a keyed hash of the allegedly-held
data (using "buddy nodes", in which two peers hold the same block, and check
up on each other), to the original MojoNation economic model.


SECURITY

Data validity (the promise that the downloaded data will match the originally
uploaded data) is provided by the hash embedded the URI. Data security (the
promise that the data is only readable by people with the URI) is provided by
the encryption key embedded in the URI. Data availability (the hope that data
which has been uploaded in the past will be downloadable in the future) is
provided by the mesh, which distributes failures in a way that reduces the
correspondence between individual node failure and file recovery failure.

The capability-based security model is used throughout this project. Filetree
operations are expressed in terms of distinct read and write capabilities.
The URI of a file is the read-capability: knowing the URI is equivalent to
the ability to read the corresponding data.


RELIABILITY

File encoding and peer selection parameters can be adjusted to achieve
different goals. Each choice results in a number of properties; there are
many tradeoffs.

First, some terms: the erasure-coding algorithm is described as K-out-of-N
(for this release, the default values are K=25 and N=100). Each mesh will
have some number of peers; this number will rise and fall over time as peers
join, drop out, come back, and leave forever. Files are of various sizes,
some are popular, others are rare. Peers have various capacities, variable
upload/download bandwidths, and network latency. Most of the mathematical
models that look at peer failure assume some average (and independent)
probability 'P' of a given peer being available: this can be high (servers
tend to be online and available >90% of the time) or low (laptops tend to be
turned on for an hour then disappear for several days). Files are encoded in
segments of a given maximum size, which affects memory usage.

The ratio of N/K is the "expansion factor". Higher expansion factors improve
reliability very quickly (the binomial distribution curve is very sharp), but
consumes much more mesh capacity. The absolute value of K affects the
granularity of the binomial curve (1-out-of-2 is much worse than
50-out-of-100), but high values asymptotically approach a constant that
depends upon 'P' (i.e. 500-of-1000 is not much better than 50-of-100).

Likewise, the total number of peers in the network affects the same
granularity: having only one peer means a single point of failure, no matter
how many copies of the file you make. Independent peers (with uncorrelated
failures) are necessary to hit the mathematical ideals: if you have 100 nodes
but they are all in the same office building, then a single power failure
will take out all of them at once. The "Sybil Attack" is where a single
attacker convinces you that they are actually multiple servers, so that you
think you are using a large number of independent peers, but in fact you have
a single point of failure (where the attacker turns off all their machines at
once). Large meshes, with lots of truly-independent peers, will enable the
use of lower expansion factors to achieve the same reliability, but increase
overhead because each peer needs to know something about every other, and the
rate at which peers come and go will be higher (requiring network maintenance
traffic). Also, the File Repairer work will increase with larger meshes,
although then the job can be distributed out to more peers.

Higher values of N increase overhead: more shares means more Merkle hashes
that must be included with the data, and more peers to contact to retrieve
the shares. Smaller segment sizes reduce memory usage (since each segment
must be held in memory while erasure coding runs) and increases "alacrity"
(since downloading can validate a smaller piece of data faster, delivering it
to the target sooner), but also increase overhead (because more blocks means
more Merkle hashes to validate them).

In general, small private meshes should work well, but the participants will
have to decide between storage overhead and reliability. Large stable meshes
will be able to reduce the expansion factor down to a bare minimum while
still retaining high reliability, but large unstable meshes (where nodes are
coming and going very quickly) may require more repair/verification bandwidth
than actual upload/download traffic.


------------------------------

[1]: http://en.wikipedia.org/wiki/Zooko%27s_triangle
[2]: all of these names are derived from the location where they were
     concocted, in this case in a car ride from Boulder to DEN
[3]: the terms CHK and SSK come from Freenet,
     http://wiki.freenetproject.org/FreenetCHKPages ,
     although we use "SSK" in a slightly different way