From 9cac3261a74833470399e0888368d010d75c0703 Mon Sep 17 00:00:00 2001 From: Brian Warner Date: Sat, 21 Jul 2007 17:29:01 -0700 Subject: [PATCH] docs/uri.txt: document current URI formats --- docs/uri.txt | 156 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 156 insertions(+) create mode 100644 docs/uri.txt diff --git a/docs/uri.txt b/docs/uri.txt new file mode 100644 index 000000000..14ea8db37 --- /dev/null +++ b/docs/uri.txt @@ -0,0 +1,156 @@ + += Tahoe URIs = + +Each file and directory in a Tahoe filesystem is described by a "URI". There +are different kinds of URIs for different kinds of objects, and there are +different kinds of URIs to provide different kinds of access to those +objects. + +Each URI provides both '''location''' and '''identification''' properties. +'''location''' means that holding the URI is sufficient to locate the data it +represents (this means it contains a storage index or a lookup key, whatever +is necessary to find the place or places where the data is being kept). +'''identification''' means that the URI also serves to validate the data: an +attacker who wants to trick you into into using the wrong data will be +limited in their abilities by the identification properties of the URI. + +Some URIs are subsets of others. In particular, if you know a URI which +allows you to modify some object, you can produce a weaker read-only URI and +give it to someone else, and they will be able to read that object but not +modify it. Each URI represents some '''capability''', and some capabilities +are derived from others. + +source:src/allmydata/uri.py is the main place where URIs are processed. It is +the authoritative definition point for all the the URI types described +herein. + +== File URIs == + +The lowest layer of the Tahoe architecture (the "grid") is reponsible for +mapping URIs to data. This is basically a distributed hash table, in which +the URI is the key, and some sequence of bytes is the value. + +At present, all the entries in this DHT are immutable. That means that each +URI represents a fixed chunk of data. The URI itself is derived from the data +when it is uploaded into the grid, and can be used to locate and download +that data from the grid at some time in the future. + +It is important to note that the "files" described by these URIs are just a +bunch of bytes, and that __no__ filenames or other metadata is retained at +this layer. The vdrive layer (which sits above the grid layer) is entirely +responsible for directories and filenames and the like. + +=== CHI URIs === + +CHK (Content Hash Keyed) files are immutable sequences of bytes. They are +uploaded in a distributed fashion using a "storage index" (for the "location" +property), and encrypted using a "read key". A secure hash of the data is +computed to help validate the data afterwards (providing the "identification" +property). All of these pieces, plus information about the file's size and +the number of shares into which it has been distributed, are put into the +"CHK" uri. + +In the current release, these URIs contain both the storage index and the +read key. The plan is to derive the storage index from the read key, so only +the latter will appear in the URI. + +The current format for CHK URIs is the concatenation of the following +strings: + + URI:CHK:(storage-index):(key):(hash):(needed-shares):(total-shares):(size) + +Where (storage-index) is the base32 encoding of the (binary) storage index +(which itself is a SHA-256 hash), (key) is the base32 encoding of the 16-byte +AES read key, (hash) is the base32 encoding of the SHA-256 hash of the URI +Extension Block, (needed-shares) is an ascii decimal representation of the +number of shares required to reconstruct this file, (total-shares) is the +same representation of the total number of shares created, and (size) is an +ascii decimal representation of the size of the data represented by this URI. + +For example, the following is a CHK URI, generated from the contents of the +architecture.txt document that lives next to this one in the source tree: + +URI:CHK:khzth3n672elnovimdpaczwuyukwm42vkdzenbqoaj6sqebk3zjq====:ihrbeov7lbvoduupd4qblysj7a======:bg5agsdt62jb34hxvxmdsbza6do64f4fg5anxxod2buttbo6udzq====:3:10:28733 + + +=== LIT URIs === + +LITeral files are also an immutable sequence of bytes, but they are so short +that the data is stored inside the URI itself. These are used for files of 55 +bytes or shorter, which is the point at which the LIT URI is the same length +as a CHK URI would be. + +LIT URIs do not require an upload or download phase, as their data is stored +directly in the URI. + +The format of a LIT URI is simply a fixed prefix concatenated with the base32 +encoding of the file's data: + + URI:LIT:bjuw4y3movsgkidbnrwg26lemf2gcl3xmvrc6kropbuhi3lmbi====== + +The LIT URI for an empty file is "URI:LIT:", and the LIT URI for a 5-byte +file that contains the string "hello" is "URI:LIT:nbswy3dp". + +=== Mutable File URIs === + +The current release does not provide for mutable files, hence all file URIs +correspond to immutable data. Future releases will probably add mutable +files, creating a new class of Mutable File URIs. These URIs will contain the +hash of a public key and also a symmetric read- or write- key. The URI refers +to a "mutable slot" into which arbitrary data can be uploaded at various +times. Each time this kind of URI is submitted to the Downloader, the caller +will receive the current contents of the slot (i.e. the data that was most +recently uploaded to it). The public key will be used to validate the data. + +Note that this form of validation is limited to confirming that the data +retrieved matches __some__ data that was uploaded in the past. The downloader +may still be vulnerable to replay attacks, although the distributed storage +mechanism will probably minimize this vulnerability. + +== Directory URIs == + +The grid layer provides a mapping from URI to data. To turn this into a graph +of directories and files, the "vdrive" layer (which sits on top of the grid +layer) needs to keep track of "directory nodes", or "dirnodes" for short. +source:docs/dirnodes.txt describes how these work. + +In the current release, each dirnode is stored (in encrypted form) on a +single "vdrive server". The Foolscap FURL that points at this server is kept +inside the "dirnode URI", as well as the read-key or write-key used in the +encryption. There are two forms of dirnode URIs: the read-write form contains +the write-key (from which the read-key can be derived by hashing), while the +read-only form only contains the read-key. The storage index is derived from +the read-key, so both kinds of URIs implicitly contain the storage index. + +The format of a read-write directory URI is the literal string "URI:DIR:", +followed by the FURL of the vdrive server, another ":", then the +base32-encoded representation of the write-key. For example: + + URI:DIR:pb://ugltpehrf73gnb4qbjigxmmzbmznjxo6@10.0.0.16:59571,127.0.0.1:59571/vdrive:x2amqa52r6kqe7iemndilvtntm====== + +A read-only directory URI is similar: "DIR-RO" is used instead of "DIR", and +the read-key is used instead of the write-key: + + URI:DIR-RO:pb://ugltpehrf73gnb4qbjigxmmzbmznjxo6@10.0.0.16:59571,127.0.0.1:59571/vdrive:l4dqkt3lianmxecxv7nol3ka2i====== + +== Internal Usage of URIs == + +The classes in source:src/allmydata/uri.py are used to pack and unpack these +various kinds of URIs. Three Interfaces are defined (IURI, IFileURI, and +IDirnodeURI) which are implemented by these classes, and string-to-URI-class +conversion routines have been registered as adapters, so that code which +wants to extract e.g. the size of a CHK or LIT uri can do: + +{{{ +print IFileURI(uri).get_size() +}}} + +If the URI does not represent a CHK or LIT uri (for example, if it was for a +directory instead), the adaptation will fail, raising a TypeError inside the +IFileURI() call. + +Several utility methods are provided on these objects. The most important is +{{{ to_string() }}}, which returns the string form of the URI. Therefore {{{ +IURI(uri).to_string == uri }}} is true for any valid URI. See the IURI class +in source:src/allmydata/interfaces.py for more details. +