ExternalVendorCode/tahoe-lafs
1
0
Fork 0
mirror of https://github.com/tahoe-lafs/tahoe-lafs.git synced 2025-05-02 17:12:59 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Copyediting.

Browse Source 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
This commit is contained in:
Daira Hopwood 2015-04-02 16:15:39 +01:00
parent 14c3bfd12e
commit 8fabcbe54b
1 changed files with 34 additions and 32 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

66
docs/proposed/magic-folder/filesystem-integration.rst
View File

@ -1,10 +1,10 @@
Magic Folders local filesystem integration design Magic Folder local filesystem integration design
================================================= ================================================
*Scope* *Scope*
This document describes how to integrate the local filesystem with Magic This document describes how to integrate the local filesystem with Magic
Folders in an efficient and reliable manner. For now we ignore Remote to Folder in an efficient and reliable manner. For now we ignore Remote to
Local synchronization; the design and implementation of this is scheduled Local synchronization; the design and implementation of this is scheduled
for a later time. We also ignore multiple writers for the same Magic for a later time. We also ignore multiple writers for the same Magic
Folder, which may or may not be supported in future. The design here will Folder, which may or may not be supported in future. The design here will
@ -20,15 +20,15 @@ under the local directory and adds every file to a first-in first-out
"scan queue". When processing the scan queue, redundant uploads are "scan queue". When processing the scan queue, redundant uploads are
avoided by using the same mechanism the Tahoe backup command uses: we avoided by using the same mechanism the Tahoe backup command uses: we
keep track of previous uploads by recording each file's metadata such as keep track of previous uploads by recording each file's metadata such as
size, CTIME and MTIME. This information is stored in a database referred size, ``ctime`` and ``mtime``. This information is stored in a database,
to from now on as the magic folder db. Using this recorded state, we referred to from now on as the magic folder db. Using this recorded
ensure that when Magic Folder is subsequently started, the local state, we ensure that when Magic Folder is subsequently started, the
directory tree can be scanned quickly by comparing current filesystem local directory tree can be scanned quickly by comparing current
metadata with the previously recorded metadata. Each file referenced in filesystem metadata with the previously recorded metadata. Each file
the scan queue is uploaded only if its metadata differs at the time it is referenced in the scan queue is uploaded only if its metadata differs at
processed. If a change event is detected for a file that is already the time it is processed. If a change event is detected for a file that
queued (and therefore will be processed later), the redundant event is is already queued (and therefore will be processed later), the redundant
ignored. event is ignored.
To implement the magic folder db, we will use an SQLite schema that To implement the magic folder db, we will use an SQLite schema that
initially is the existing Tahoe-LAFS backup schema. This schema may initially is the existing Tahoe-LAFS backup schema. This schema may
@ -39,31 +39,33 @@ database file and separate mutex lock just for Magic Folder. This avoids
usability problems related to mutual exclusion. (If a single file and usability problems related to mutual exclusion. (If a single file and
lock were used, a backup would block Magic Folder updates for a long lock were used, a backup would block Magic Folder updates for a long
time, and a user would not be able to tell when backups are possible time, and a user would not be able to tell when backups are possible
because Magic Folder acquires a lock at arbitrary times.) because Magic Folder would acquire a lock at arbitrary times.)
*Eventual consistency property* *Eventual consistency property*
It is not possible to prevent local writes to a file while it is being During the process of reading a file in order to upload it, it is not
read in order to upload it. Such writes will result in temporary possible to prevent further local writes. Such writes will result in
inconsistency. Eventual consistency is reached when the queue of pending temporary inconsistency (that is, the uploaded file will not reflect
uploads is empty. That is, a consistent snapshot will be achieved what the contents of the local file were at any specific time). Eventual
eventually when local writes to the target folder cease for a consistency is reached when the queue of pending uploads is empty. That
sufficiently long period of time. is, a consistent snapshot will be achieved eventually when local writes
to the target folder cease for a sufficiently long period of time.
*Detecting filesystem changes* *Detecting filesystem changes*
For the Linux implementation we will use the inotify Linux kernel For the Linux implementation, we will use the inotify Linux kernel
subsystem to gather events on the local Magic Folder directory tree. This subsystem to gather events on the local Magic Folder directory tree. This
implementation was already present in Tahoe-LAFS 1.9.0, but needs to be implementation was already present in Tahoe-LAFS 1.9.0, but needs to be
changed to gather directory creation and move events, as well as events changed to gather directory creation and move events, in addition to the
indicating that a file has been written. events indicating that a file has been written that are gathered by the
current code.
For the Windows implementation we will use the ``ReadDirectoryChangesW`` For the Windows implementation, we will use the ``ReadDirectoryChangesW``
Win32 API. The prototype implementation simulates the inotify API in Win32 API. The prototype implementation simulates a Python interface to
terms of ``ReadDirectoryChangesW``, allowing most of the code to be the inotify API in terms of ``ReadDirectoryChangesW``, allowing most of
shared across platforms. the code to be shared across platforms.
When we detect the creation of a new directory below the local Magic When we detect the creation of a new directory below the local Magic
Folder directory, we create it in the Tahoe-LAFS filesystem, and also Folder directory, we create it in the Tahoe-LAFS filesystem, and also
@ -75,23 +77,23 @@ is moved to be under the local Magic Folder directory.
*User interface* *User interface*
The Magic Folders local filesystem integration will initially have a The Magic Folder local filesystem integration will initially have a
provisional configuration file-based interface that may not be ideal from provisional configuration file-based interface that may not be ideal from
a usability perspective. Creating our local filesystem integration in a usability perspective. Creating our local filesystem integration in
this manner will allow us to use and test it indepently of the rest of this manner will allow us to use and test it independently of the rest of
the Magic Folder software components. We will focus greater attention on the Magic Folder software components. We will focus greater attention on
user interface design as a later milestone in our development roadmap. user interface design as a later milestone in our development roadmap.
The configuration file, ``tahoe.cfg``, must define a target local The configuration file, ``tahoe.cfg``, must define a target local
directory to be synchronized. Provisionally, this configuration will directory to be synchronized. Provisionally, this configuration will
replace the current "drop-upload" section:: replace the current ``[drop_upload]`` section::
[magic_folders] [magic_folder]
enabled = true enabled = true
local.directory = "/home/human" local.directory = "/home/human"
When a filesystem directory is first "added" to Magic Folders, the user When a filesystem directory is first configured for Magic Folder, the user
needs to create the remote Tahoe-LAFS directory using ``tahoe mkdir``, needs to create the remote Tahoe-LAFS directory using ``tahoe mkdir``,
and configure the Magic-Folder-enabled node with its URI (e.g. by putting and configure the Magic-Folder-enabled node with its URI (e.g. by putting
it in a file ``private/magic_folder_dircap``). If there are existing it in a file ``private/magic_folder_dircap``). If there are existing
files in the local directory, they will be uploaded as a result of the files in the local directory, they will be uploaded as a result of the