From 461f42e7bc560e6abceff135728a1bcda1913c3c Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Thu, 13 Feb 2020 10:14:25 -0500 Subject: [PATCH 01/25] news fragment --- newsfragments/3284.removed | 1 + 1 file changed, 1 insertion(+) create mode 100644 newsfragments/3284.removed diff --git a/newsfragments/3284.removed b/newsfragments/3284.removed new file mode 100644 index 000000000..7e31d352a --- /dev/null +++ b/newsfragments/3284.removed @@ -0,0 +1 @@ +The Magic Folder frontend has been split out into a stand-alone project. The functionality is no longer part of Tahoe-LAFS itself. Learn more at . From 76d667865e8f10449a541b1427952670fc225c70 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Thu, 13 Feb 2020 10:21:45 -0500 Subject: [PATCH 02/25] Remove some top-level packaging related to magic-folders --- Makefile | 6 ------ setup.py | 5 +---- 2 files changed, 1 insertion(+), 10 deletions(-) diff --git a/Makefile b/Makefile index 63d9db980..6e0009ddf 100644 --- a/Makefile +++ b/Makefile @@ -42,12 +42,6 @@ upload-osx-pkg: # echo not uploading tahoe-lafs-osx-pkg because this is not trunk but is branch \"${BB_BRANCH}\" ; \ # fi -.PHONY: smoketest -smoketest: - -python ./src/allmydata/test/check_magicfolder_smoke.py kill - -rm -rf smoke_magicfolder/ - python ./src/allmydata/test/check_magicfolder_smoke.py - # code coverage-based testing is disabled temporarily, as we switch to tox. # This will eventually be added to a tox environment. The following comments # and variable settings are retained as notes for that future effort. diff --git a/setup.py b/setup.py index 72561a9d9..caa37f16e 100644 --- a/setup.py +++ b/setup.py @@ -62,9 +62,7 @@ install_requires = [ # version of cryptography will *really* be installed. "cryptography >= 2.6", - # * On Linux we need at least Twisted 10.1.0 for inotify support - # used by the drop-upload frontend. - # * We also need Twisted 10.1.0 for the FTP frontend in order for + # * We need Twisted 10.1.0 for the FTP frontend in order for # Twisted's FTP server to support asynchronous close. # * The SFTP frontend depends on Twisted 11.0.0 to fix the SSH server # rekeying bug @@ -354,7 +352,6 @@ setup(name="tahoe-lafs", # also set in __init__.py # https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2392 for some # discussion. ':sys_platform=="win32"': ["pywin32 != 226"], - ':sys_platform!="win32" and sys_platform!="linux2"': ["watchdog"], # For magic-folder on "darwin" (macOS) and the BSDs "test": [ # Pin a specific pyflakes so we don't have different folks # disagreeing on what is or is not a lint issue. We can bump From e3a6d759627ad76bf28eaf84dbdbd8a1719f2302 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Thu, 13 Feb 2020 10:23:27 -0500 Subject: [PATCH 03/25] Remove the obvious documentation references --- docs/configuration.rst | 7 - docs/frontends/magic-folder.rst | 148 --- docs/index.rst | 2 - docs/magic-folder-howto.rst | 176 ---- docs/magic-wormhole-invites.rst | 4 +- docs/proposed/index.rst | 4 - .../magic-folder/filesystem-integration.rst | 118 --- .../multi-party-conflict-detection.rst | 373 ------- .../magic-folder/remote-to-local-sync.rst | 951 ------------------ .../magic-folder/user-interface-design.rst | 205 ---- 10 files changed, 1 insertion(+), 1987 deletions(-) delete mode 100644 docs/frontends/magic-folder.rst delete mode 100644 docs/magic-folder-howto.rst delete mode 100644 docs/proposed/magic-folder/filesystem-integration.rst delete mode 100644 docs/proposed/magic-folder/multi-party-conflict-detection.rst delete mode 100644 docs/proposed/magic-folder/remote-to-local-sync.rst delete mode 100644 docs/proposed/magic-folder/user-interface-design.rst diff --git a/docs/configuration.rst b/docs/configuration.rst index 02a97d2b8..ab4751a04 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -82,7 +82,6 @@ Client/server nodes provide one or more of the following services: * web-API service * SFTP service * FTP service -* Magic Folder service * helper service * storage service. @@ -719,12 +718,6 @@ SFTP, FTP for instructions on configuring these services, and the ``[sftpd]`` and ``[ftpd]`` sections of ``tahoe.cfg``. -Magic Folder - - A node running on Linux or Windows can be configured to automatically - upload files that are created or changed in a specified local directory. - See :doc:`frontends/magic-folder` for details. - Storage Server Configuration ============================ diff --git a/docs/frontends/magic-folder.rst b/docs/frontends/magic-folder.rst deleted file mode 100644 index 694958563..000000000 --- a/docs/frontends/magic-folder.rst +++ /dev/null @@ -1,148 +0,0 @@ -.. -*- coding: utf-8-with-signature -*- - -================================ -Tahoe-LAFS Magic Folder Frontend -================================ - -1. `Introduction`_ -2. `Configuration`_ -3. `Known Issues and Limitations With Magic-Folder`_ - - -Introduction -============ - -The Magic Folder frontend synchronizes local directories on two or more -clients, using a Tahoe-LAFS grid for storage. Whenever a file is created -or changed under the local directory of one of the clients, the change is -propagated to the grid and then to the other clients. - -The implementation of the "drop-upload" frontend, on which Magic Folder is -based, was written as a prototype at the First International Tahoe-LAFS -Summit in June 2011. In 2015, with the support of a grant from the -`Open Technology Fund`_, it was redesigned and extended to support -synchronization between clients. It currently works on Linux and Windows. - -Magic Folder is not currently in as mature a state as the other frontends -(web, CLI, SFTP and FTP). This means that you probably should not rely on -all changes to files in the local directory to result in successful uploads. -There might be (and have been) incompatible changes to how the feature is -configured. - -We are very interested in feedback on how well this feature works for you, and -suggestions to improve its usability, functionality, and reliability. - -.. _`Open Technology Fund`: https://www.opentech.fund/ - - -Configuration -============= - -The Magic Folder frontend runs as part of a gateway node. To set it up, you -must use the tahoe magic-folder CLI. For detailed information see our -:doc:`Magic-Folder CLI design -documentation<../proposed/magic-folder/user-interface-design>`. For a -given Magic-Folder collective directory you need to run the ``tahoe -magic-folder create`` command. After that the ``tahoe magic-folder invite`` -command must used to generate an *invite code* for each member of the -magic-folder collective. A confidential, authenticated communications channel -should be used to transmit the invite code to each member, who will be -joining using the ``tahoe magic-folder join`` command. - -These settings are persisted in the ``[magic_folder]`` section of the -gateway's ``tahoe.cfg`` file. - -``[magic_folder]`` - -``enabled = (boolean, optional)`` - - If this is ``True``, Magic Folder will be enabled. The default value is - ``False``. - -``local.directory = (UTF-8 path)`` - - This specifies the local directory to be monitored for new or changed - files. If the path contains non-ASCII characters, it should be encoded - in UTF-8 regardless of the system's filesystem encoding. Relative paths - will be interpreted starting from the node's base directory. - -You should not normally need to set these fields manually because they are -set by the ``tahoe magic-folder create`` and/or ``tahoe magic-folder join`` -commands. Use the ``--help`` option to these commands for more information. - -After setting up a Magic Folder collective and starting or restarting each -gateway, you can confirm that the feature is working by copying a file into -any local directory, and checking that it appears on other clients. -Large files may take some time to appear. - -The 'Operational Statistics' page linked from the Welcome page shows counts -of the number of files uploaded, the number of change events currently -queued, and the number of failed uploads. The 'Recent Uploads and Downloads' -page and the node :doc:`log<../logging>` may be helpful to determine the -cause of any failures. - - -.. _Known Issues in Magic-Folder: - -Known Issues and Limitations With Magic-Folder -============================================== - -This feature only works on Linux and Windows. There is a ticket to add -support for Mac OS X and BSD-based systems (`#1432`_). - -The only way to determine whether uploads have failed is to look at the -'Operational Statistics' page linked from the Welcome page. This only shows -a count of failures, not the names of files. Uploads are never retried. - -The Magic Folder frontend performs its uploads sequentially (i.e. it waits -until each upload is finished before starting the next), even when there -would be enough memory and bandwidth to efficiently perform them in parallel. -A Magic Folder upload can occur in parallel with an upload by a different -frontend, though. (`#1459`_) - -On Linux, if there are a large number of near-simultaneous file creation or -change events (greater than the number specified in the file -``/proc/sys/fs/inotify/max_queued_events``), it is possible that some events -could be missed. This is fairly unlikely under normal circumstances, because -the default value of ``max_queued_events`` in most Linux distributions is -16384, and events are removed from this queue immediately without waiting for -the corresponding upload to complete. (`#1430`_) - -The Windows implementation might also occasionally miss file creation or -change events, due to limitations of the underlying Windows API -(ReadDirectoryChangesW). We do not know how likely or unlikely this is. -(`#1431`_) - -Some filesystems may not support the necessary change notifications. -So, it is recommended for the local directory to be on a directly attached -disk-based filesystem, not a network filesystem or one provided by a virtual -machine. - -The ``private/magic_folder_dircap`` and ``private/collective_dircap`` files -cannot use an alias or path to specify the upload directory. (`#1711`_) - -If a file in the upload directory is changed (actually relinked to a new -file), then the old file is still present on the grid, and any other caps -to it will remain valid. Eventually it will be possible to use -:doc:`../garbage-collection` to reclaim the space used by these files; however -currently they are retained indefinitely. (`#2440`_) - -Unicode filenames are supported on both Linux and Windows, but on Linux, the -local name of a file must be encoded correctly in order for it to be uploaded. -The expected encoding is that printed by -``python -c "import sys; print sys.getfilesystemencoding()"``. - -On Windows, local directories with non-ASCII names are not currently working. -(`#2219`_) - -On Windows, when a node has Magic Folder enabled, it is unresponsive to Ctrl-C -(it can only be killed using Task Manager or similar). (`#2218`_) - -.. _`#1430`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1430 -.. _`#1431`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1431 -.. _`#1432`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1432 -.. _`#1459`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1459 -.. _`#1711`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1711 -.. _`#2218`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2218 -.. _`#2219`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2219 -.. _`#2440`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2440 diff --git a/docs/index.rst b/docs/index.rst index 566e78d4f..98b4f241b 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -20,7 +20,6 @@ Contents: frontends/CLI frontends/webapi frontends/FTP-and-SFTP - frontends/magic-folder frontends/download-status known_issues @@ -37,7 +36,6 @@ Contents: expenses cautions write_coordination - magic-folder-howto backupdb anonymity-configuration diff --git a/docs/magic-folder-howto.rst b/docs/magic-folder-howto.rst deleted file mode 100644 index b368972b8..000000000 --- a/docs/magic-folder-howto.rst +++ /dev/null @@ -1,176 +0,0 @@ -.. _magic-folder-howto: - -========================= -Magic Folder Set-up Howto -========================= - -#. `This document`_ -#. `Setting up a local test grid`_ -#. `Setting up Magic Folder`_ -#. `Testing`_ - - -This document -============= - -This is preliminary documentation of how to set up Magic Folder using a test -grid on a single Linux or Windows machine, with two clients and one server. -It is aimed at a fairly technical audience. - -For an introduction to Magic Folder and how to configure it -more generally, see :doc:`frontends/magic-folder`. - -It it possible to adapt these instructions to run the nodes on -different machines, to synchronize between three or more clients, -to mix Windows and Linux clients, and to use multiple servers -(if the Tahoe-LAFS encoding parameters are changed). - - -Setting up a local test grid -============================ - -Linux ------ - -Run these commands:: - - mkdir ../grid - bin/tahoe create-introducer ../grid/introducer - bin/tahoe start ../grid/introducer - export FURL=`cat ../grid/introducer/private/introducer.furl` - bin/tahoe create-node --introducer="$FURL" ../grid/server - bin/tahoe create-client --introducer="$FURL" ../grid/alice - bin/tahoe create-client --introducer="$FURL" ../grid/bob - - -Windows -------- - -Run:: - - mkdir ..\grid - bin\tahoe create-introducer ..\grid\introducer - bin\tahoe start ..\grid\introducer - -Leave the introducer running in that Command Prompt, -and in a separate Command Prompt (with the same current -directory), run:: - - set /p FURL=<..\grid\introducer\private\introducer.furl - bin\tahoe create-node --introducer=%FURL% ..\grid\server - bin\tahoe create-client --introducer=%FURL% ..\grid\alice - bin\tahoe create-client --introducer=%FURL% ..\grid\bob - - -Both Linux and Windows ----------------------- - -(Replace ``/`` with ``\`` for Windows paths.) - -Edit ``../grid/alice/tahoe.cfg``, and make the following -changes to the ``[node]`` and ``[client]`` sections:: - - [node] - nickname = alice - web.port = tcp:3457:interface=127.0.0.1 - - [client] - shares.needed = 1 - shares.happy = 1 - shares.total = 1 - -Edit ``../grid/bob/tahoe.cfg``, and make the following -change to the ``[node]`` section, and the same change as -above to the ``[client]`` section:: - - [node] - nickname = bob - web.port = tcp:3458:interface=127.0.0.1 - -Note that when running nodes on a single machine, -unique port numbers must be used for each node (and they -must not clash with ports used by other server software). -Here we have used the default of 3456 for the server, -3457 for alice, and 3458 for bob. - -Now start all of the nodes (the introducer should still be -running from above):: - - bin/tahoe start ../grid/server - bin/tahoe start ../grid/alice - bin/tahoe start ../grid/bob - -On Windows, a separate Command Prompt is needed to run each -node. - -Open a web browser on http://127.0.0.1:3457/ and verify that -alice is connected to the introducer and one storage server. -Then do the same for http://127.0.0.1:3568/ to verify that -bob is connected. Leave all of the nodes running for the -next stage. - - -Setting up Magic Folder -======================= - -Linux ------ - -Run:: - - mkdir -p ../local/alice ../local/bob - bin/tahoe -d ../grid/alice magic-folder create magic: alice ../local/alice - bin/tahoe -d ../grid/alice magic-folder invite magic: bob >invitecode - export INVITECODE=`cat invitecode` - bin/tahoe -d ../grid/bob magic-folder join "$INVITECODE" ../local/bob - - bin/tahoe restart ../grid/alice - bin/tahoe restart ../grid/bob - -Windows -------- - -Run:: - - mkdir ..\local\alice ..\local\bob - bin\tahoe -d ..\grid\alice magic-folder create magic: alice ..\local\alice - bin\tahoe -d ..\grid\alice magic-folder invite magic: bob >invitecode - set /p INVITECODE=`) invites and joins work this way -as well. +Introducer with new clients. This is a two-part process. Alice runs a grid and wishes to have her friend Bob use it as a client. She runs ``tahoe invite bob`` which diff --git a/docs/proposed/index.rst b/docs/proposed/index.rst index a052baeff..d01d92d2d 100644 --- a/docs/proposed/index.rst +++ b/docs/proposed/index.rst @@ -14,8 +14,4 @@ index only lists the files that are in .rst format. :maxdepth: 2 leasedb - magic-folder/filesystem-integration - magic-folder/remote-to-local-sync - magic-folder/user-interface-design - magic-folder/multi-party-conflict-detection http-storage-node-protocol diff --git a/docs/proposed/magic-folder/filesystem-integration.rst b/docs/proposed/magic-folder/filesystem-integration.rst deleted file mode 100644 index 589db2322..000000000 --- a/docs/proposed/magic-folder/filesystem-integration.rst +++ /dev/null @@ -1,118 +0,0 @@ -Magic Folder local filesystem integration design -================================================ - -*Scope* - -This document describes how to integrate the local filesystem with Magic -Folder in an efficient and reliable manner. For now we ignore Remote to -Local synchronization; the design and implementation of this is scheduled -for a later time. We also ignore multiple writers for the same Magic -Folder, which may or may not be supported in future. The design here will -be updated to account for those features in later Objectives. Objective 3 -may require modifying the database schema or operation, and Objective 5 -may modify the User interface. - -Tickets on the Tahoe-LAFS trac with the `otf-magic-folder-objective2`_ -keyword are within the scope of the local filesystem integration for -Objective 2. - -.. _otf-magic-folder-objective2: https://tahoe-lafs.org/trac/tahoe-lafs/query?status=!closed&keywords=~otf-magic-folder-objective2 - -.. _filesystem_integration-local-scanning-and-database: - -*Local scanning and database* - -When a Magic-Folder-enabled node starts up, it scans all directories -under the local directory and adds every file to a first-in first-out -"scan queue". When processing the scan queue, redundant uploads are -avoided by using the same mechanism the Tahoe backup command uses: we -keep track of previous uploads by recording each file's metadata such as -size, ``ctime`` and ``mtime``. This information is stored in a database, -referred to from now on as the magic folder db. Using this recorded -state, we ensure that when Magic Folder is subsequently started, the -local directory tree can be scanned quickly by comparing current -filesystem metadata with the previously recorded metadata. Each file -referenced in the scan queue is uploaded only if its metadata differs at -the time it is processed. If a change event is detected for a file that -is already queued (and therefore will be processed later), the redundant -event is ignored. - -To implement the magic folder db, we will use an SQLite schema that -initially is the existing Tahoe-LAFS backup schema. This schema may -change in later objectives; this will cause no backward compatibility -problems, because this new feature will be developed on a branch that -makes no compatibility guarantees. However we will have a separate SQLite -database file and separate mutex lock just for Magic Folder. This avoids -usability problems related to mutual exclusion. (If a single file and -lock were used, a backup would block Magic Folder updates for a long -time, and a user would not be able to tell when backups are possible -because Magic Folder would acquire a lock at arbitrary times.) - - -*Eventual consistency property* - -During the process of reading a file in order to upload it, it is not -possible to prevent further local writes. Such writes will result in -temporary inconsistency (that is, the uploaded file will not reflect -what the contents of the local file were at any specific time). Eventual -consistency is reached when the queue of pending uploads is empty. That -is, a consistent snapshot will be achieved eventually when local writes -to the target folder cease for a sufficiently long period of time. - - -*Detecting filesystem changes* - -For the Linux implementation, we will use the `inotify`_ Linux kernel -subsystem to gather events on the local Magic Folder directory tree. This -implementation was already present in Tahoe-LAFS 1.9.0, but needs to be -changed to gather directory creation and move events, in addition to the -events indicating that a file has been written that are gathered by the -current code. - -.. _`inotify`: https://en.wikipedia.org/wiki/Inotify - -For the Windows implementation, we will use the ``ReadDirectoryChangesW`` -Win32 API. The prototype implementation simulates a Python interface to -the inotify API in terms of ``ReadDirectoryChangesW``, allowing most of -the code to be shared across platforms. - -The alternative of using `NTFS Change Journals`_ for Windows was -considered, but appears to be more complicated and does not provide any -additional functionality over the scanning approach described above. -The Change Journal mechanism is also only available for NTFS filesystems, -but FAT32 filesystems are still common in user installations of Windows. - -.. _`NTFS Change Journals`: https://msdn.microsoft.com/en-us/library/aa363803%28VS.85%29.aspx - -When we detect the creation of a new directory below the local Magic -Folder directory, we create it in the Tahoe-LAFS filesystem, and also -scan the new local directory for new files. This scan is necessary to -avoid missing events for creation of files in a new directory before it -can be watched, and to correctly handle cases where an existing directory -is moved to be under the local Magic Folder directory. - - -*User interface* - -The Magic Folder local filesystem integration will initially have a -provisional configuration file-based interface that may not be ideal from -a usability perspective. Creating our local filesystem integration in -this manner will allow us to use and test it independently of the rest of -the Magic Folder software components. We will focus greater attention on -user interface design as a later milestone in our development roadmap. - -The configuration file, ``tahoe.cfg``, must define a target local -directory to be synchronized. Provisionally, this configuration will -replace the current ``[drop_upload]`` section:: - - [magic_folder] - enabled = true - local.directory = "/home/human" - -When a filesystem directory is first configured for Magic Folder, the user -needs to create the remote Tahoe-LAFS directory using ``tahoe mkdir``, -and configure the Magic-Folder-enabled node with its URI (e.g. by putting -it in a file ``private/magic_folder_dircap``). If there are existing -files in the local directory, they will be uploaded as a result of the -initial scan described earlier. - diff --git a/docs/proposed/magic-folder/multi-party-conflict-detection.rst b/docs/proposed/magic-folder/multi-party-conflict-detection.rst deleted file mode 100644 index fb1ae8339..000000000 --- a/docs/proposed/magic-folder/multi-party-conflict-detection.rst +++ /dev/null @@ -1,373 +0,0 @@ -Multi-party Conflict Detection -============================== - -The current Magic-Folder remote conflict detection design does not properly detect remote conflicts -for groups of three or more parties. This design is specified in the "Fire Dragon" section of this document: -https://github.com/tahoe-lafs/tahoe-lafs/blob/2551.wip.2/docs/proposed/magic-folder/remote-to-local-sync.rst#fire-dragons-distinguishing-conflicts-from-overwrites - -This Tahoe-LAFS trac ticket comment outlines a scenario with -three parties in which a remote conflict is falsely detected: - -.. _`ticket comment`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2551#comment:22 - - -Summary and definitions -======================= - -Abstract file: a file being shared by a Magic Folder. - -Local file: a file in a client's local filesystem corresponding to an abstract file. - -Relative path: the path of an abstract or local file relative to the Magic Folder root. - -Version: a snapshot of an abstract file, with associated metadata, that is uploaded by a Magic Folder client. - -A version is associated with the file's relative path, its contents, and -mtime and ctime timestamps. Versions also have a unique identity. - -Follows relation: -* If and only if a change to a client's local file at relative path F that results in an upload of version V', -was made when the client already had version V of that file, then we say that V' directly follows V. -* The follows relation is the irreflexive transitive closure of the "directly follows" relation. - -The follows relation is transitive and acyclic, and therefore defines a DAG called the -Version DAG. Different abstract files correspond to disconnected sets of nodes in the Version DAG -(in other words there are no "follows" relations between different files). - -The DAG is only ever extended, not mutated. - -The desired behaviour for initially classifying overwrites and conflicts is as follows: - -* if a client Bob currently has version V of a file at relative path F, and it sees a new version V' - of that file in another client Alice's DMD, such that V' follows V, then the write of the new version - is initially an overwrite and should be to the same filename. -* if, in the same situation, V' does not follow V, then the write of the new version should be - classified as a conflict. - -The existing :doc:`remote-to-local-sync` document defines when an initial -overwrite should be reclassified as a conflict. - -The above definitions completely specify the desired solution of the false -conflict behaviour described in the `ticket comment`_. However, they do not give -a concrete algorithm to compute the follows relation, or a representation in the -Tahoe-LAFS file store of the metadata needed to compute it. - -We will consider two alternative designs, proposed by Leif Ryge and -Zooko Wilcox-O'Hearn, that aim to fill this gap. - - - -Leif's Proposal: Magic-Folder "single-file" snapshot design -=========================================================== - -Abstract --------- - -We propose a relatively simple modification to the initial Magic Folder design which -adds merkle DAGs of immutable historical snapshots for each file. The full history -does not necessarily need to be retained, and the choice of how much history to retain -can potentially be made on a per-file basis. - -Motivation: ------------ - -no SPOFs, no admins -``````````````````` - -Additionally, the initial design had two cases of excess authority: - -1. The magic folder administrator (inviter) has everyone's write-caps and is thus essentially "root" -2. Each client shares ambient authority and can delete anything or everything and - (assuming there is not a conflict) the data will be deleted from all clients. So, each client - is effectively "root" too. - -Thus, while it is useful for file synchronization, the initial design is a much less safe place -to store data than in a single mutable tahoe directory (because more client computers have the -possibility to delete it). - - -Glossary --------- - -- merkle DAG: like a merkle tree but with multiple roots, and with each node potentially having multiple parents -- magic folder: a logical directory that can be synchronized between many clients - (devices, users, ...) using a Tahoe-LAFS storage grid -- client: a Magic-Folder-enabled Tahoe-LAFS client instance that has access to a magic folder -- DMD: "distributed mutable directory", a physical Tahoe-LAFS mutable directory. - Each client has the write cap to their own DMD, and read caps to all other client's DMDs - (as in the original Magic Folder design). -- snapshot: a reference to a version of a file; represented as an immutable directory containing - an entry called "content" (pointing to the immutable file containing the file's contents), - and an entry called "parent0" (pointing to a parent snapshot), and optionally parent1 through - parentN pointing at other parents. The Magic Folder snapshot object is conceptually very similar - to a git commit object, except for that it is created automatically and it records the history of an - individual file rather than an entire repository. Also, commits do not need to have authors - (although an author field could be easily added later). -- deletion snapshot: immutable directory containing no content entry (only one or more parents) -- capability: a Tahoe-LAFS diminishable cryptographic capability -- cap: short for capability -- conflict: the situation when another client's current snapshot for a file is different than our current snapshot, and is not a descendant of ours. -- overwrite: the situation when another client's current snapshot for a file is a (not necessarily direct) descendant of our current snapshot. - - -Overview --------- - -This new design will track the history of each file using "snapshots" which are -created at each upload. Each snapshot will specify one or more parent snapshots, -forming a directed acyclic graph. A Magic-Folder user's DMD uses a flattened directory -hierarchy naming scheme, as in the original design. But, instead of pointing directly -at file contents, each file name will link to that user's latest snapshot for that file. - -Inside the dmd there will also be an immutable directory containing the client's subscriptions -(read-caps to other clients' dmds). - -Clients periodically poll each other's DMDs. When they see the current snapshot for a file is -different than their own current snapshot for that file, they immediately begin downloading its -contents and then walk backwards through the DAG from the new snapshot until they find their own -snapshot or a common ancestor. - -For the common ancestor search to be efficient, the client will need to keep a local store (in the magic folder db) of all of the snapshots -(but not their contents) between the oldest current snapshot of any of their subscriptions and their own current snapshot. -See "local cache purging policy" below for more details. - -If the new snapshot is a descendant of the client's existing snapshot, then this update -is an "overwrite" - like a git fast-forward. So, when the download of the new file completes it can overwrite -the existing local file with the new contents and update its dmd to point at the new snapshot. - -If the new snapshot is not a descendant of the client's current snapshot, then the update is a -conflict. The new file is downloaded and named $filename.conflict-$user1,$user2 (including a list -of other subscriptions who have that version as their current version). - -Changes to the local .conflict- file are not tracked. When that file disappears -(either by deletion, or being renamed) a new snapshot for the conflicting file is -created which has two parents - the client's snapshot prior to the conflict, and the -new conflicting snapshot. If multiple .conflict files are deleted or renamed in a short -period of time, a single conflict-resolving snapshot with more than two parents can be created. - -! I think this behavior will confuse users. - -Tahoe-LAFS snapshot objects ---------------------------- - -These Tahoe-LAFS snapshot objects only track the history of a single file, not a directory hierarchy. -Snapshot objects contain only two field types: -- ``Content``: an immutable capability of the file contents (omitted if deletion snapshot) -- ``Parent0..N``: immutable capabilities representing parent snapshots - -Therefore in this system an interesting side effect of this Tahoe snapshot object is that there is no -snapshot author. The only notion of an identity in the Magic-Folder system is the write capability of the user's DMD. - -The snapshot object is an immutable directory which looks like this: -content -> immutable cap to file content -parent0 -> immutable cap to a parent snapshot object -parent1..N -> more parent snapshots - - -Snapshot Author Identity ------------------------- - -Snapshot identity might become an important feature so that bad actors -can be recognized and other clients can stop "subscribing" to (polling for) updates from them. - -Perhaps snapshots could be signed by the user's Magic-Folder write key for this purpose? Probably a bad idea to reuse the write-cap key for this. Better to introduce ed25519 identity keys which can (optionally) sign snapshot contents and store the signature as another member of the immutable directory. - - -Conflict Resolution -------------------- - -detection of conflicts -`````````````````````` - -A Magic-Folder client updates a given file's current snapshot link to a snapshot which is a descendent -of the previous snapshot. For a given file, let's say "file1", Alice can detect that Bob's DMD has a "file1" -that links to a snapshot which conflicts. Two snapshots conflict if one is not an ancestor of the other. - - -a possible UI for resolving conflicts -````````````````````````````````````` - -If Alice links a conflicting snapshot object for a file named "file1", -Bob and Carole will see a file in their Magic-Folder called "file1.conflicted.Alice". -Alice conversely will see an additional file called "file1.conflicted.previous". -If Alice wishes to resolve the conflict with her new version of the file then -she simply deletes the file called "file1.conflicted.previous". If she wants to -choose the other version then she moves it into place: - - mv file1.conflicted.previous file1 - - -This scheme works for N number of conflicts. Bob for instance could choose -the same resolution for the conflict, like this: - - mv file1.Alice file1 - - -Deletion propagation and eventual Garbage Collection ----------------------------------------------------- - -When a user deletes a file, this is represented by a link from their DMD file -object to a deletion snapshot. Eventually all users will link this deletion -snapshot into their DMD. When all users have the link then they locally cache -the deletion snapshot and remove the link to that file in their DMD. -Deletions can of course be undeleted; this means creating a new snapshot -object that specifies itself a descent of the deletion snapshot. - -Clients periodically renew leases to all capabilities recursively linked -to in their DMD. Files which are unlinked by ALL the users of a -given Magic-Folder will eventually be garbage collected. - -Lease expirey duration must be tuned properly by storage servers such that -Garbage Collection does not occur too frequently. - - - -Performance Considerations --------------------------- - -local changes -````````````` - -Our old scheme requires two remote Tahoe-LAFS operations per local file modification: -1. upload new file contents (as an immutable file) -2. modify mutable directory (DMD) to link to the immutable file cap - -Our new scheme requires three remote operations: -1. upload new file contents (as in immutable file) -2. upload immutable directory representing Tahoe-LAFS snapshot object -3. modify mutable directory (DMD) to link to the immutable snapshot object - -remote changes -`````````````` - -Our old scheme requires one remote Tahoe-LAFS operation per remote file modification (not counting the polling of the dmd): -1. Download new file content - -Our new scheme requires a minimum of two remote operations (not counting the polling of the dmd) for conflicting downloads, or three remote operations for overwrite downloads: -1. Download new snapshot object -2. Download the content it points to -3. If the download is an overwrite, modify the DMD to indicate that the downloaded version is their current version. - -If the new snapshot is not a direct descendant of our current snapshot or the other party's previous snapshot we saw, we will also need to download more snapshots to determine if it is a conflict or an overwrite. However, those can be done in -parallel with the content download since we will need to download the content in either case. - -While the old scheme is obviously more efficient, we think that the properties provided by the new scheme make it worth the additional cost. - -Physical updates to the DMD overiouslly need to be serialized, so multiple logical updates should be combined when an update is already in progress. - -conflict detection and local caching -```````````````````````````````````` - -Local caching of snapshots is important for performance. -We refer to the client's local snapshot cache as the ``magic-folder db``. - -Conflict detection can be expensive because it may require the client -to download many snapshots from the other user's DMD in order to try -and find it's own current snapshot or a descendent. The cost of scanning -the remote DMDs should not be very high unless the client conducting the -scan has lots of history to download because of being offline for a long -time while many new snapshots were distributed. - - -local cache purging policy -`````````````````````````` - -The client's current snapshot for each file should be cached at all times. -When all clients' views of a file are synchronized (they all have the same -snapshot for that file), no ancestry for that file needs to be cached. -When clients' views of a file are *not* synchronized, the most recent -common ancestor of all clients' snapshots must be kept cached, as must -all intermediate snapshots. - - -Local Merge Property --------------------- - -Bob can in fact, set a pre-existing directory (with files) as his new Magic-Folder directory, resulting -in a merge of the Magic-Folder with Bob's local directory. Filename collisions will result in conflicts -because Bob's new snapshots are not descendent's of the existing Magic-Folder file snapshots. - - -Example: simultaneous update with four parties: - -1. A, B, C, D are in sync for file "foo" at snapshot X -2. A and B simultaneously change the file, creating snapshots XA and XB (both descendants of X). -3. C hears about XA first, and D hears about XB first. Both accept an overwrite. -4. All four parties hear about the other update they hadn't heard about yet. -5. Result: - - everyone's local file "foo" has the content pointed to by the snapshot in their DMD's "foo" entry - - A and C's DMDs each have the "foo" entry pointing at snapshot XA - - B and D's DMDs each have the "foo" entry pointing at snapshot XB - - A and C have a local file called foo.conflict-B,D with XB's content - - B and D have a local file called foo.conflict-A,C with XA's content - -Later: - - - Everyone ignores the conflict, and continue updating their local "foo". but slowly enough that there are no further conflicts, so that A and C remain in sync with eachother, and B and D remain in sync with eachother. - - - A and C's foo.conflict-B,D file continues to be updated with the latest version of the file B and D are working on, and vice-versa. - - - A and C edit the file at the same time again, causing a new conflict. - - - Local files are now: - - A: "foo", "foo.conflict-B,D", "foo.conflict-C" - - C: "foo", "foo.conflict-B,D", "foo.conflict-A" - - B and D: "foo", "foo.conflict-A", "foo.conflict-C" - - - Finally, D decides to look at "foo.conflict-A" and "foo.conflict-C", and they manually integrate (or decide to ignore) the differences into their own local file "foo". - - - D deletes their conflict files. - - - D's DMD now points to a snapshot that is a descendant of everyone else's current snapshot, resolving all conflicts. - - - The conflict files on A, B, and C disappear, and everyone's local file "foo" contains D's manually-merged content. - - -Daira: I think it is too complicated to include multiple nicknames in the .conflict files -(e.g. "foo.conflict-B,D"). It should be sufficient to have one file for each other client, -reflecting that client's latest version, regardless of who else it conflicts with. - - -Zooko's Design (as interpreted by Daira) -======================================== - -A version map is a mapping from client nickname to version number. - -Definition: a version map M' strictly-follows a mapping M iff for every entry c->v -in M, there is an entry c->v' in M' such that v' > v. - - -Each client maintains a 'local version map' and a 'conflict version map' for each file -in its magic folder db. -If it has never written the file, then the entry for its own nickname in the local version -map is zero. The conflict version map only contains entries for nicknames B where -"$FILENAME.conflict-$B" exists. - -When a client A uploads a file, it increments the version for its own nickname in its -local version map for the file, and includes that map as metadata with its upload. - -A download by client A from client B is an overwrite iff the downloaded version map -strictly-follows A's local version map for that file; in this case A replaces its local -version map with the downloaded version map. Otherwise it is a conflict, and the -download is put into "$FILENAME.conflict-$B"; in this case A's -local version map remains unchanged, and the entry B->v taken from the downloaded -version map is added to its conflict version map. - -If client A deletes or renames a conflict file "$FILENAME.conflict-$B", then A copies -the entry for B from its conflict version map to its local version map, deletes -the entry for B in its conflict version map, and performs another upload (with -incremented version number) of $FILENAME. - - -Example: - A, B, C = (10, 20, 30) everyone agrees. - A updates: (11, 20, 30) - B updates: (10, 21, 30) - -C will see either A or B first. Both would be an overwrite, if considered alone. - - - diff --git a/docs/proposed/magic-folder/remote-to-local-sync.rst b/docs/proposed/magic-folder/remote-to-local-sync.rst deleted file mode 100644 index 32c2fd5da..000000000 --- a/docs/proposed/magic-folder/remote-to-local-sync.rst +++ /dev/null @@ -1,951 +0,0 @@ -Magic Folder design for remote-to-local sync -============================================ - -Scope ------ - -In this Objective we will design remote-to-local synchronization: - -* How to efficiently determine which objects (files and directories) have - to be downloaded in order to bring the current local filesystem into sync - with the newly-discovered version of the remote filesystem. -* How to distinguish overwrites, in which the remote side was aware of - your most recent version and overwrote it with a new version, from - conflicts, in which the remote side was unaware of your most recent - version when it published its new version. The latter needs to be raised - to the user as an issue the user will have to resolve and the former must - not bother the user. -* How to overwrite the (stale) local versions of those objects with the - newly acquired objects, while preserving backed-up versions of those - overwritten objects in case the user didn't want this overwrite and wants - to recover the old version. - -Tickets on the Tahoe-LAFS trac with the `otf-magic-folder-objective4`_ -keyword are within the scope of the remote-to-local synchronization -design. - -.. _otf-magic-folder-objective4: https://tahoe-lafs.org/trac/tahoe-lafs/query?status=!closed&keywords=~otf-magic-folder-objective4 - - -Glossary -'''''''' - -Object: a file or directory - -DMD: distributed mutable directory - -Folder: an abstract directory that is synchronized between clients. -(A folder is not the same as the directory corresponding to it on -any particular client, nor is it the same as a DMD.) - -Collective: the set of clients subscribed to a given Magic Folder. - -Descendant: a direct or indirect child in a directory or folder tree - -Subfolder: a folder that is a descendant of a magic folder - -Subpath: the path from a magic folder to one of its descendants - -Write: a modification to a local filesystem object by a client - -Read: a read from a local filesystem object by a client - -Upload: an upload of a local object to the Tahoe-LAFS file store - -Download: a download from the Tahoe-LAFS file store to a local object - -Pending notification: a local filesystem change that has been detected -but not yet processed. - - -Representing the Magic Folder in Tahoe-LAFS -------------------------------------------- - -Unlike the local case where we use inotify or ReadDirectoryChangesW to -detect filesystem changes, we have no mechanism to register a monitor for -changes to a Tahoe-LAFS directory. Therefore, we must periodically poll -for changes. - -An important constraint on the solution is Tahoe-LAFS' ":doc:`write -coordination directive<../../write_coordination>`", which prohibits -concurrent writes by different storage clients to the same mutable object: - - Tahoe does not provide locking of mutable files and directories. If - there is more than one simultaneous attempt to change a mutable file - or directory, then an UncoordinatedWriteError may result. This might, - in rare cases, cause the file or directory contents to be accidentally - deleted. The user is expected to ensure that there is at most one - outstanding write or update request for a given file or directory at - a time. One convenient way to accomplish this is to make a different - file or directory for each person or process that wants to write. - -Since it is a goal to allow multiple users to write to a Magic Folder, -if the write coordination directive remains the same as above, then we -will not be able to implement the Magic Folder as a single Tahoe-LAFS -DMD. In general therefore, we will have multiple DMDs —spread across -clients— that together represent the Magic Folder. Each client in a -Magic Folder collective polls the other clients' DMDs in order to detect -remote changes. - -Six possible designs were considered for the representation of subfolders -of the Magic Folder: - -1. All subfolders written by a given Magic Folder client are collapsed -into a single client DMD, containing immutable files. The child name of -each file encodes the full subpath of that file relative to the Magic -Folder. - -2. The DMD tree under a client DMD is a direct copy of the folder tree -written by that client to the Magic Folder. Not all subfolders have -corresponding DMDs; only those to which that client has written files or -child subfolders. - -3. The directory tree under a client DMD is a ``tahoe backup`` structure -containing immutable snapshots of the folder tree written by that client -to the Magic Folder. As in design 2, only objects written by that client -are present. - -4. *Each* client DMD contains an eventually consistent mirror of all -files and folders written by *any* Magic Folder client. Thus each client -must also copy changes made by other Magic Folder clients to its own -client DMD. - -5. *Each* client DMD contains a ``tahoe backup`` structure containing -immutable snapshots of all files and folders written by *any* Magic -Folder client. Thus each client must also create another snapshot in its -own client DMD when changes are made by another client. (It can potentially -batch changes, subject to latency requirements.) - -6. The write coordination problem is solved by implementing `two-phase -commit`_. Then, the representation consists of a single DMD tree which is -written by all clients. - -.. _`two-phase commit`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1755 - -Here is a summary of advantages and disadvantages of each design: - -+----------------------------+ -| Key | -+=======+====================+ -| \+\+ | major advantage | -+-------+--------------------+ -| \+ | minor advantage | -+-------+--------------------+ -| ‒ | minor disadvantage | -+-------+--------------------+ -| ‒ ‒ | major disadvantage | -+-------+--------------------+ -| ‒ ‒ ‒ | showstopper | -+-------+--------------------+ - - -123456+: All designs have the property that a recursive add-lease operation -starting from a *collective directory* containing all of the client DMDs, -will find all of the files and directories used in the Magic Folder -representation. Therefore the representation is compatible with :doc:`garbage -collection <../../garbage-collection>`, even when a pre-Magic-Folder client -does the lease marking. - -123456+: All designs avoid "breaking" pre-Magic-Folder clients that read -a directory or file that is part of the representation. - -456++: Only these designs allow a readcap to one of the client -directories —or one of their subdirectories— to be directly shared -with other Tahoe-LAFS clients (not necessarily Magic Folder clients), -so that such a client sees all of the contents of the Magic Folder. -Note that this was not a requirement of the OTF proposal, although it -is useful. - -135+: A Magic Folder client has only one mutable Tahoe-LAFS object to -monitor per other client. This minimizes communication bandwidth for -polling, or alternatively the latency possible for a given polling -bandwidth. - -1236+: A client does not need to make changes to its own DMD that repeat -changes that another Magic Folder client had previously made. This reduces -write bandwidth and complexity. - -1‒: If the Magic Folder has many subfolders, their files will all be -collapsed into the same DMD, which could get quite large. In practice a -single DMD can easily handle the number of files expected to be written -by a client, so this is unlikely to be a significant issue. - -123‒ ‒: In these designs, the set of files in a Magic Folder is -represented as the union of the files in all client DMDs. However, -when a file is modified by more than one client, it will be linked -from multiple client DMDs. We therefore need a mechanism, such as a -version number or a monotonically increasing timestamp, to determine -which copy takes priority. - -35‒ ‒: When a Magic Folder client detects a remote change, it must -traverse an immutable directory structure to see what has changed. -Completely unchanged subtrees will have the same URI, allowing some of -this traversal to be shortcutted. - -24‒ ‒ ‒: When a Magic Folder client detects a remote change, it must -traverse a mutable directory structure to see what has changed. This is -more complex and less efficient than traversing an immutable structure, -because shortcutting is not possible (each DMD retains the same URI even -if a descendant object has changed), and because the structure may change -while it is being traversed. Also the traversal needs to be robust -against cycles, which can only occur in mutable structures. - -45‒ ‒: When a change occurs in one Magic Folder client, it will propagate -to all the other clients. Each client will therefore see multiple -representation changes for a single logical change to the Magic Folder -contents, and must suppress the duplicates. This is particularly -problematic for design 4 where it interacts with the preceding issue. - -4‒ ‒ ‒, 5‒ ‒: There is the potential for client DMDs to get "out of sync" -with each other, potentially for long periods if errors occur. Thus each -client must be able to "repair" its client directory (and its -subdirectory structure) concurrently with performing its own writes. This -is a significant complexity burden and may introduce failure modes that -could not otherwise happen. - -6‒ ‒ ‒: While two-phase commit is a well-established protocol, its -application to Tahoe-LAFS requires significant design work, and may still -leave some corner cases of the write coordination problem unsolved. - - -+------------------------------------------------+-----------------------------------------+ -| Design Property | Designs Proposed | -+================================================+======+======+======+======+======+======+ -| **advantages** | *1* | *2* | *3* | *4* | *5* | *6* | -+------------------------------------------------+------+------+------+------+------+------+ -| Compatible with garbage collection |\+ |\+ |\+ |\+ |\+ |\+ | -+------------------------------------------------+------+------+------+------+------+------+ -| Does not break old clients |\+ |\+ |\+ |\+ |\+ |\+ | -+------------------------------------------------+------+------+------+------+------+------+ -| Allows direct sharing | | | |\+\+ |\+\+ |\+\+ | -+------------------------------------------------+------+------+------+------+------+------+ -| Efficient use of bandwidth |\+ | |\+ | |\+ | | -+------------------------------------------------+------+------+------+------+------+------+ -| No repeated changes |\+ |\+ |\+ | | |\+ | -+------------------------------------------------+------+------+------+------+------+------+ -| **disadvantages** | *1* | *2* | *3* | *4* | *5* | *6* | -+------------------------------------------------+------+------+------+------+------+------+ -| Can result in large DMDs |‒ | | | | | | -+------------------------------------------------+------+------+------+------+------+------+ -| Need version number to determine priority |‒ ‒ |‒ ‒ |‒ ‒ | | | | -+------------------------------------------------+------+------+------+------+------+------+ -| Must traverse immutable directory structure | | |‒ ‒ | |‒ ‒ | | -+------------------------------------------------+------+------+------+------+------+------+ -| Must traverse mutable directory structure | |‒ ‒ ‒ | |‒ ‒ ‒ | | | -+------------------------------------------------+------+------+------+------+------+------+ -| Must suppress duplicate representation changes | | | |‒ ‒ |‒ ‒ | | -+------------------------------------------------+------+------+------+------+------+------+ -| "Out of sync" problem | | | |‒ ‒ ‒ |‒ ‒ | | -+------------------------------------------------+------+------+------+------+------+------+ -| Unsolved design problems | | | | | |‒ ‒ ‒ | -+------------------------------------------------+------+------+------+------+------+------+ - - -Evaluation of designs -''''''''''''''''''''' - -Designs 2 and 3 have no significant advantages over design 1, while -requiring higher polling bandwidth and greater complexity due to the need -to create subdirectories. These designs were therefore rejected. - -Design 4 was rejected due to the out-of-sync problem, which is severe -and possibly unsolvable for mutable structures. - -For design 5, the out-of-sync problem is still present but possibly -solvable. However, design 5 is substantially more complex, less efficient -in bandwidth/latency, and less scalable in number of clients and -subfolders than design 1. It only gains over design 1 on the ability to -share directory readcaps to the Magic Folder (or subfolders), which was -not a requirement. It would be possible to implement this feature in -future by switching to design 6. - -For the time being, however, design 6 was considered out-of-scope for -this project. - -Therefore, design 1 was chosen. That is: - - All subfolders written by a given Magic Folder client are collapsed - into a single client DMD, containing immutable files. The child name - of each file encodes the full subpath of that file relative to the - Magic Folder. - -Each directory entry in a DMD also stores a version number, so that the -latest version of a file is well-defined when it has been modified by -multiple clients. - -To enable representing empty directories, a client that creates a -directory should link a corresponding zero-length file in its DMD, -at a name that ends with the encoded directory separator character. - -We want to enable dynamic configuration of the membership of a Magic -Folder collective, without having to reconfigure or restart each client -when another client joins. To support this, we have a single collective -directory that links to all of the client DMDs, named by their client -nicknames. If the collective directory is mutable, then it is possible -to change its contents in order to add clients. Note that a client DMD -should not be unlinked from the collective directory unless all of its -files are first copied to some other client DMD. - -A client needs to be able to write to its own DMD, and read from other DMDs. -To be consistent with the `Principle of Least Authority`_, each client's -reference to its own DMD is a write capability, whereas its reference -to the collective directory is a read capability. The latter transitively -grants read access to all of the other client DMDs and the files linked -from them, as required. - -.. _`Principle of Least Authority`: http://www.eros-os.org/papers/secnotsep.pdf - -Design and implementation of the user interface for maintaining this -DMD structure and configuration will be addressed in Objectives 5 and 6. - -During operation, each client will poll for changes on other clients -at a predetermined frequency. On each poll, it will reread the collective -directory (to allow for added or removed clients), and then read each -client DMD linked from it. - -"Hidden" files, and files with names matching the patterns used for backup, -temporary, and conflicted files, will be ignored, i.e. not synchronized -in either direction. A file is hidden if it has a filename beginning with -"." (on any platform), or has the hidden or system attribute on Windows. - - -Conflict Detection and Resolution ---------------------------------- - -The combination of local filesystems and distributed objects is -an example of shared state concurrency, which is highly error-prone -and can result in race conditions that are complex to analyze. -Unfortunately we have no option but to use shared state in this -situation. - -We call the resulting design issues "dragons" (as in "Here be dragons"), -which as a convenient mnemonic we have named after the classical -Greek elements Earth, Fire, Air, and Water. - -Note: all filenames used in the following sections are examples, -and the filename patterns we use in the actual implementation may -differ. The actual patterns will probably include timestamps, and -for conflicted files, the nickname of the client that last changed -the file. - - -Earth Dragons: Collisions between local filesystem operations and downloads -''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' - -Write/download collisions -~~~~~~~~~~~~~~~~~~~~~~~~~ - -Suppose that Alice's Magic Folder client is about to write a -version of ``foo`` that it has downloaded in response to a remote -change. - -The criteria for distinguishing overwrites from conflicts are -described later in the `Fire Dragons`_ section. Suppose that the -remote change has been initially classified as an overwrite. -(As we will see, it may be reclassified in some circumstances.) - -.. _`Fire Dragons`: #fire-dragons-distinguishing-conflicts-from-overwrites - -Note that writing a file that does not already have an entry in the -:ref:`magic folder db` is -initially classed as an overwrite. - -A *write/download collision* occurs when another program writes -to ``foo`` in the local filesystem, concurrently with the new -version being written by the Magic Folder client. We need to -ensure that this does not cause data loss, as far as possible. - -An important constraint on the design is that on Windows, it is -not possible to rename a file to the same name as an existing -file in that directory. Also, on Windows it may not be possible to -delete or rename a file that has been opened by another process -(depending on the sharing flags specified by that process). -Therefore we need to consider carefully how to handle failure -conditions. - -In our proposed design, Alice's Magic Folder client follows -this procedure for an overwrite in response to a remote change: - -1. Write a temporary file, say ``.foo.tmp``. -2. Use the procedure described in the `Fire Dragons_` section - to obtain an initial classification as an overwrite or a - conflict. (This takes as input the ``last_downloaded_uri`` - field from the directory entry of the changed ``foo``.) -3. Set the ``mtime`` of the replacement file to be at least *T* seconds - before the current local time. Stat the replacement file - to obtain its ``mtime`` and ``ctime`` as stored in the local - filesystem, and update the file's last-seen statinfo in - the magic folder db with this information. (Note that the - retrieved ``mtime`` may differ from the one that was set due - to rounding.) -4. Perform a ''file replacement'' operation (explained below) - with backup filename ``foo.backup``, replaced file ``foo``, - and replacement file ``.foo.tmp``. If any step of this - operation fails, reclassify as a conflict and stop. - -To reclassify as a conflict, attempt to rename ``.foo.tmp`` to -``foo.conflicted``, suppressing errors. - -The implementation of file replacement differs between Unix -and Windows. On Unix, it can be implemented as follows: - -* 4a. Stat the replaced path, and set the permissions of the - replacement file to be the same as the replaced file, - bitwise-or'd with octal 600 (``rw-------``). If the replaced - file does not exist, set the permissions according to the - user's umask. If there is a directory at the replaced path, - fail. -* 4b. Attempt to move the replaced file (``foo``) to the - backup filename (``foo.backup``). If an ``ENOENT`` error - occurs because the replaced file does not exist, ignore this - error and continue with steps 4c and 4d. -* 4c. Attempt to create a hard link at the replaced filename - (``foo``) pointing to the replacement file (``.foo.tmp``). -* 4d. Attempt to unlink the replacement file (``.foo.tmp``), - suppressing errors. - -Note that, if there is no conflict, the entry for ``foo`` -recorded in the :ref:`magic folder -db` will -reflect the ``mtime`` set in step 3. The move operation in step -4b will cause a ``MOVED_FROM`` event for ``foo``, and the link -operation in step 4c will cause an ``IN_CREATE`` event for -``foo``. However, these events will not trigger an upload, -because they are guaranteed to be processed only after the file -replacement has finished, at which point the last-seen statinfo -recorded in the database entry will exactly match the metadata -for the file's inode on disk. (The two hard links — ``foo`` -and, while it still exists, ``.foo.tmp`` — share the same inode -and therefore the same metadata.) - -On Windows, file replacement can be implemented by a call to -the `ReplaceFileW`_ API (with the -``REPLACEFILE_IGNORE_MERGE_ERRORS`` flag). If an error occurs -because the replaced file does not exist, then we ignore this -error and attempt to move the replacement file to the replaced -file. - -Similar to the Unix case, the `ReplaceFileW`_ operation will -cause one or more change notifications for ``foo``. The replaced -``foo`` has the same ``mtime`` as the replacement file, and so any -such notification(s) will not trigger an unwanted upload. - -.. _`ReplaceFileW`: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365512%28v=vs.85%29.aspx - -To determine whether this procedure adequately protects against data -loss, we need to consider what happens if another process attempts to -update ``foo``, for example by renaming ``foo.other`` to ``foo``. -This requires us to analyze all possible interleavings between the -operations performed by the Magic Folder client and the other process. -(Note that atomic operations on a directory are totally ordered.) -The set of possible interleavings differs between Windows and Unix. - -On Unix, for the case where the replaced file already exists, we have: - -* Interleaving A: the other process' rename precedes our rename in - step 4b, and we get an ``IN_MOVED_TO`` event for its rename by - step 2. Then we reclassify as a conflict; its changes end up at - ``foo`` and ours end up at ``foo.conflicted``. This avoids data - loss. - -* Interleaving B: its rename precedes ours in step 4b, and we do - not get an event for its rename by step 2. Its changes end up at - ``foo.backup``, and ours end up at ``foo`` after being linked there - in step 4c. This avoids data loss. - -* Interleaving C: its rename happens between our rename in step 4b, - and our link operation in step 4c of the file replacement. The - latter fails with an ``EEXIST`` error because ``foo`` already - exists. We reclassify as a conflict; the old version ends up at - ``foo.backup``, the other process' changes end up at ``foo``, and - ours at ``foo.conflicted``. This avoids data loss. - -* Interleaving D: its rename happens after our link in step 4c, and - causes an ``IN_MOVED_TO`` event for ``foo``. Its rename also changes - the ``mtime`` for ``foo`` so that it is different from the ``mtime`` - calculated in step 3, and therefore different from the metadata - recorded for ``foo`` in the magic folder db. (Assuming no system - clock changes, its rename will set an ``mtime`` timestamp - corresponding to a time after step 4c, which is after the timestamp - *T* seconds before step 4a, provided that *T* seconds is - sufficiently greater than the timestamp granularity.) Therefore, an - upload will be triggered for ``foo`` after its change, which is - correct and avoids data loss. - -If the replaced file did not already exist, an ``ENOENT`` error -occurs at step 4b, and we continue with steps 4c and 4d. The other -process' rename races with our link operation in step 4c. If the -other process wins the race then the effect is similar to -Interleaving C, and if we win the race this it is similar to -Interleaving D. Either case avoids data loss. - - -On Windows, the internal implementation of `ReplaceFileW`_ is similar -to what we have described above for Unix; it works like this: - -* 4a′. Copy metadata (which does not include ``mtime``) from the - replaced file (``foo``) to the replacement file (``.foo.tmp``). - -* 4b′. Attempt to move the replaced file (``foo``) onto the - backup filename (``foo.backup``), deleting the latter if it - already exists. - -* 4c′. Attempt to move the replacement file (``.foo.tmp``) to the - replaced filename (``foo``); fail if the destination already - exists. - -Notice that this is essentially the same as the algorithm we use -for Unix, but steps 4c and 4d on Unix are combined into a single -step 4c′. (If there is a failure at steps 4c′ after step 4b′ has -completed, the `ReplaceFileW`_ call will fail with return code -``ERROR_UNABLE_TO_MOVE_REPLACEMENT_2``. However, it is still -preferable to use this API over two `MoveFileExW`_ calls, because -it retains the attributes and ACLs of ``foo`` where possible. -Also note that if the `ReplaceFileW`_ call fails with -``ERROR_FILE_NOT_FOUND`` because the replaced file does not exist, -then the replacment operation ignores this error and continues with -the equivalent of step 4c′, as on Unix.) - -However, on Windows the other application will not be able to -directly rename ``foo.other`` onto ``foo`` (which would fail because -the destination already exists); it will have to rename or delete -``foo`` first. Without loss of generality, let's say ``foo`` is -deleted. This complicates the interleaving analysis, because we -have two operations done by the other process interleaving with -three done by the magic folder process (rather than one operation -interleaving with four as on Unix). - -So on Windows, for the case where the replaced file already exists, -we have: - -* Interleaving A′: the other process' deletion of ``foo`` and its - rename of ``foo.other`` to ``foo`` both precede our rename in - step 4b. We get an event corresponding to its rename by step 2. - Then we reclassify as a conflict; its changes end up at ``foo`` - and ours end up at ``foo.conflicted``. This avoids data loss. - -* Interleaving B′: the other process' deletion of ``foo`` and its - rename of ``foo.other`` to ``foo`` both precede our rename in - step 4b. We do not get an event for its rename by step 2. - Its changes end up at ``foo.backup``, and ours end up at ``foo`` - after being moved there in step 4c′. This avoids data loss. - -* Interleaving C′: the other process' deletion of ``foo`` precedes - our rename of ``foo`` to ``foo.backup`` done by `ReplaceFileW`_, - but its rename of ``foo.other`` to ``foo`` does not, so we get - an ``ERROR_FILE_NOT_FOUND`` error from `ReplaceFileW`_ indicating - that the replaced file does not exist. We ignore this error and - attempt to move ``foo.tmp`` to ``foo``, racing with the other - process which is attempting to move ``foo.other`` to ``foo``. - If we win the race, then our changes end up at ``foo``, and the - other process' move fails. If the other process wins the race, - then its changes end up at ``foo``, our move fails, and we - reclassify as a conflict, so that our changes end up at - ``foo.conflicted``. Either possibility avoids data loss. - -* Interleaving D′: the other process' deletion and/or rename happen - during the call to `ReplaceFileW`_, causing the latter to fail. - There are two subcases: - - * if the error is ``ERROR_UNABLE_TO_MOVE_REPLACEMENT_2``, then - ``foo`` is renamed to ``foo.backup`` and ``.foo.tmp`` remains - at its original name after the call. - * for all other errors, ``foo`` and ``.foo.tmp`` both remain at - their original names after the call. - - In both subcases, we reclassify as a conflict and rename ``.foo.tmp`` - to ``foo.conflicted``. This avoids data loss. - -* Interleaving E′: the other process' deletion of ``foo`` and attempt - to rename ``foo.other`` to ``foo`` both happen after all internal - operations of `ReplaceFileW`_ have completed. This causes deletion - and rename events for ``foo`` (which will in practice be merged due - to the pending delay, although we don't rely on that for - correctness). The rename also changes the ``mtime`` for ``foo`` so - that it is different from the ``mtime`` calculated in step 3, and - therefore different from the metadata recorded for ``foo`` in the - magic folder db. (Assuming no system clock changes, its rename will - set an ``mtime`` timestamp corresponding to a time after the - internal operations of `ReplaceFileW`_ have completed, which is - after the timestamp *T* seconds before `ReplaceFileW`_ is called, - provided that *T* seconds is sufficiently greater than the timestamp - granularity.) Therefore, an upload will be triggered for ``foo`` - after its change, which is correct and avoids data loss. - -.. _`MoveFileExW`: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365240%28v=vs.85%29.aspx - -If the replaced file did not already exist, we get an -``ERROR_FILE_NOT_FOUND`` error from `ReplaceFileW`_, and attempt to -move ``foo.tmp`` to ``foo``. This is similar to Interleaving C, and -either possibility for the resulting race avoids data loss. - -We also need to consider what happens if another process opens ``foo`` -and writes to it directly, rather than renaming another file onto it: - -* On Unix, open file handles refer to inodes, not paths. If the other - process opens ``foo`` before it has been renamed to ``foo.backup``, - and then closes the file, changes will have been written to the file - at the same inode, even if that inode is now linked at ``foo.backup``. - This avoids data loss. - -* On Windows, we have two subcases, depending on whether the sharing - flags specified by the other process when it opened its file handle - included ``FILE_SHARE_DELETE``. (This flag covers both deletion and - rename operations.) - - i. If the sharing flags *do not* allow deletion/renaming, the - `ReplaceFileW`_ operation will fail without renaming ``foo``. - In this case we will end up with ``foo`` changed by the other - process, and the downloaded file still in ``foo.tmp``. - This avoids data loss. - - ii. If the sharing flags *do* allow deletion/renaming, then - data loss or corruption may occur. This is unavoidable and - can be attributed to other process making a poor choice of - sharing flags (either explicitly if it used `CreateFile`_, or - via whichever higher-level API it used). - -.. _`CreateFile`: https://msdn.microsoft.com/en-us/library/windows/desktop/aa363858%28v=vs.85%29.aspx - -Note that it is possible that another process tries to open the file -between steps 4b and 4c (or 4b′ and 4c′ on Windows). In this case the -open will fail because ``foo`` does not exist. Nevertheless, no data -will be lost, and in many cases the user will be able to retry the -operation. - -Above we only described the case where the download was initially -classified as an overwrite. If it was classed as a conflict, the -procedure is the same except that we choose a unique filename -for the conflicted file (say, ``foo.conflicted_unique``). We write -the new contents to ``.foo.tmp`` and then rename it to -``foo.conflicted_unique`` in such a way that the rename will fail -if the destination already exists. (On Windows this is a simple -rename; on Unix it can be implemented as a link operation followed -by an unlink, similar to steps 4c and 4d above.) If this fails -because another process wrote ``foo.conflicted_unique`` after we -chose the filename, then we retry with a different filename. - - -Read/download collisions -~~~~~~~~~~~~~~~~~~~~~~~~ - -A *read/download collision* occurs when another program reads -from ``foo`` in the local filesystem, concurrently with the new -version being written by the Magic Folder client. We want to -ensure that any successful attempt to read the file by the other -program obtains a consistent view of its contents. - -On Unix, the above procedure for writing downloads is sufficient -to achieve this. There are three cases: - -* A. The other process opens ``foo`` for reading before it is - renamed to ``foo.backup``. Then the file handle will continue to - refer to the old file across the rename, and the other process - will read the old contents. - -* B. The other process attempts to open ``foo`` after it has been - renamed to ``foo.backup``, and before it is linked in step c. - The open call fails, which is acceptable. - -* C. The other process opens ``foo`` after it has been linked to - the new file. Then it will read the new contents. - -On Windows, the analysis is very similar, but case A′ needs to -be split into two subcases, depending on the sharing mode the other -process uses when opening the file for reading: - -* A′. The other process opens ``foo`` before the Magic Folder - client's attempt to rename ``foo`` to ``foo.backup`` (as part - of the implementation of `ReplaceFileW`_). The subcases are: - - i. The other process uses sharing flags that deny deletion and - renames. The `ReplaceFileW`_ call fails, and the download is - reclassified as a conflict. The downloaded file ends up at - ``foo.conflicted``, which is correct. - - ii. The other process uses sharing flags that allow deletion - and renames. The `ReplaceFileW`_ call succeeds, and the - other process reads inconsistent data. This can be attributed - to a poor choice of sharing flags by the other process. - -* B′. The other process attempts to open ``foo`` at the point - during the `ReplaceFileW`_ call where it does not exist. - The open call fails, which is acceptable. - -* C′. The other process opens ``foo`` after it has been linked to - the new file. Then it will read the new contents. - - -For both write/download and read/download collisions, we have -considered only interleavings with a single other process, and -only the most common possibilities for the other process' -interaction with the file. If multiple other processes are -involved, or if a process performs operations other than those -considered, then we cannot say much about the outcome in general; -however, we believe that such cases will be much less common. - - - -Fire Dragons: Distinguishing conflicts from overwrites -'''''''''''''''''''''''''''''''''''''''''''''''''''''' - -When synchronizing a file that has changed remotely, the Magic Folder -client needs to distinguish between overwrites, in which the remote -side was aware of your most recent version (if any) and overwrote it -with a new version, and conflicts, in which the remote side was unaware -of your most recent version when it published its new version. Those two -cases have to be handled differently — the latter needs to be raised -to the user as an issue the user will have to resolve and the former -must not bother the user. - -For example, suppose that Alice's Magic Folder client sees a change -to ``foo`` in Bob's DMD. If the version it downloads from Bob's DMD -is "based on" the version currently in Alice's local filesystem at -the time Alice's client attempts to write the downloaded file ‒or if -there is no existing version in Alice's local filesystem at that time‒ -then it is an overwrite. Otherwise it is initially classified as a -conflict. - -This initial classification is used by the procedure for writing a -file described in the `Earth Dragons`_ section above. As explained -in that section, we may reclassify an overwrite as a conflict if an -error occurs during the write procedure. - -.. _`Earth Dragons`: #earth-dragons-collisions-between-local-filesystem-operations-and-downloads - -In order to implement this policy, we need to specify how the -"based on" relation between file versions is recorded and updated. - -We propose to record this information: - -* in the :ref:`magic folder - db`, for - local files; -* in the Tahoe-LAFS directory metadata, for files stored in the - Magic Folder. - -In the magic folder db we will add a *last-downloaded record*, -consisting of ``last_downloaded_uri`` and ``last_downloaded_timestamp`` -fields, for each path stored in the database. Whenever a Magic Folder -client downloads a file, it stores the downloaded version's URI and -the current local timestamp in this record. Since only immutable -files are used, the URI will be an immutable file URI, which is -deterministically and uniquely derived from the file contents and -the Tahoe-LAFS node's :doc:`convergence secret<../../convergence-secret>`. - -(Note that the last-downloaded record is updated regardless of -whether the download is an overwrite or a conflict. The rationale -for this to avoid "conflict loops" between clients, where every -new version after the first conflict would be considered as another -conflict.) - -Later, in response to a local filesystem change at a given path, the -Magic Folder client reads the last-downloaded record associated with -that path (if any) from the database and then uploads the current -file. When it links the uploaded file into its client DMD, it -includes the ``last_downloaded_uri`` field in the metadata of the -directory entry, overwriting any existing field of that name. If -there was no last-downloaded record associated with the path, this -field is omitted. - -Note that ``last_downloaded_uri`` field does *not* record the URI of -the uploaded file (which would be redundant); it records the URI of -the last download before the local change that caused the upload. -The field will be absent if the file has never been downloaded by -this client (i.e. if it was created on this client and no change -by any other client has been detected). - -A possible refinement also takes into account the -``last_downloaded_timestamp`` field from the magic folder db, and -compares it to the timestamp of the change that caused the upload -(which should be later, assuming no system clock changes). -If the duration between these timestamps is very short, then we -are uncertain about whether the process on Bob's system that wrote -the local file could have taken into account the last download. -We can use this information to be conservative about treating -changes as conflicts. So, if the duration is less than a configured -threshold, we omit the ``last_downloaded_uri`` field from the -metadata. This will have the effect of making other clients treat -this change as a conflict whenever they already have a copy of the -file. - -Conflict/overwrite decision algorithm -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Now we are ready to describe the algorithm for determining whether a -download for the file ``foo`` is an overwrite or a conflict (refining -step 2 of the procedure from the `Earth Dragons`_ section). - -Let ``last_downloaded_uri`` be the field of that name obtained from -the directory entry metadata for ``foo`` in Bob's DMD (this field -may be absent). Then the algorithm is: - -* 2a. Attempt to "stat" ``foo`` to get its *current statinfo* (size - in bytes, ``mtime``, and ``ctime``). If Alice has no local copy - of ``foo``, classify as an overwrite. - -* 2b. Read the following information for the path ``foo`` from the - local magic folder db: - - * the *last-seen statinfo*, if any (this is the size in - bytes, ``mtime``, and ``ctime`` stored in the ``local_files`` - table when the file was last uploaded); - * the ``last_uploaded_uri`` field of the ``local_files`` table - for this file, which is the URI under which the file was last - uploaded. - -* 2c. If any of the following are true, then classify as a conflict: - - * i. there are pending notifications of changes to ``foo``; - * ii. the last-seen statinfo is either absent (i.e. there is - no entry in the database for this path), or different from the - current statinfo; - * iii. either ``last_downloaded_uri`` or ``last_uploaded_uri`` - (or both) are absent, or they are different. - - Otherwise, classify as an overwrite. - - -Air Dragons: Collisions between local writes and uploads -'''''''''''''''''''''''''''''''''''''''''''''''''''''''' - -Short of filesystem-specific features on Unix or the `shadow copy service`_ -on Windows (which is per-volume and therefore difficult to use in this -context), there is no way to *read* the whole contents of a file -atomically. Therefore, when we read a file in order to upload it, we -may read an inconsistent version if it was also being written locally. - -.. _`shadow copy service`: https://technet.microsoft.com/en-us/library/ee923636%28v=ws.10%29.aspx - -A well-behaved application can avoid this problem for its writes: - -* On Unix, if another process modifies a file by renaming a temporary - file onto it, then we will consistently read either the old contents - or the new contents. -* On Windows, if the other process uses sharing flags to deny reads - while it is writing a file, then we will consistently read either - the old contents or the new contents, unless a sharing error occurs. - In the case of a sharing error we should retry later, up to a - maximum number of retries. - -In the case of a not-so-well-behaved application writing to a file -at the same time we read from it, the magic folder will still be -eventually consistent, but inconsistent versions may be visible to -other users' clients. - -In Objective 2 we implemented a delay, called the *pending delay*, -after the notification of a filesystem change and before the file is -read in order to upload it (Tahoe-LAFS ticket `#1440`_). If another -change notification occurs within the pending delay time, the delay -is restarted. This helps to some extent because it means that if -files are written more quickly than the pending delay and less -frequently than the pending delay, we shouldn't encounter this -inconsistency. - -.. _`#1440`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1440 - -The likelihood of inconsistency could be further reduced, even for -writes by not-so-well-behaved applications, by delaying the actual -upload for a further period —called the *stability delay*— after the -file has finished being read. If a notification occurs between the -end of the pending delay and the end of the stability delay, then -the read would be aborted and the notification requeued. - -This would have the effect of ensuring that no write notifications -have been received for the file during a time window that brackets -the period when it was being read, with margin before and after -this period defined by the pending and stability delays. The delays -are intended to account for asynchronous notification of events, and -caching in the filesystem. - -Note however that we cannot guarantee that the delays will be long -enough to prevent inconsistency in any particular case. Also, the -stability delay would potentially affect performance significantly -because (unlike the pending delay) it is not overlapped when there -are multiple files on the upload queue. This performance impact -could be mitigated by uploading files in parallel where possible -(Tahoe-LAFS ticket `#1459`_). - -We have not yet decided whether to implement the stability delay, and -it is not planned to be implemented for the OTF objective 4 milestone. -Ticket `#2431`_ has been opened to track this idea. - -.. _`#1459`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1459 -.. _`#2431`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2431 - -Note that the situation of both a local process and the Magic Folder -client reading a file at the same time cannot cause any inconsistency. - - -Water Dragons: Handling deletion and renames -'''''''''''''''''''''''''''''''''''''''''''' - -Deletion of a file -~~~~~~~~~~~~~~~~~~ - -When a file is deleted from the filesystem of a Magic Folder client, -the most intuitive behavior is for it also to be deleted under that -name from other clients. To avoid data loss, the other clients should -actually rename their copies to a backup filename. - -It would not be sufficient for a Magic Folder client that deletes -a file to implement this simply by removing the directory entry from -its DMD. Indeed, the entry may not exist in the client's DMD if it -has never previously changed the file. - -Instead, the client links a zero-length file into its DMD and sets -``deleted: true`` in the directory entry metadata. Other clients -take this as a signal to rename their copies to the backup filename. - -Note that the entry for this zero-length file has a version number as -usual, and later versions may restore the file. - -When the downloader deletes a file (or renames it to a filename -ending in ``.backup``) in response to a remote change, a local -filesystem notification will occur, and we must make sure that this -is not treated as a local change. To do this we have the downloader -set the ``size`` field in the magic folder db to ``None`` (SQL NULL) -just before deleting the file, and suppress notifications for which -the local file does not exist, and the recorded ``size`` field is -``None``. - -When a Magic Folder client restarts, we can detect files that had -been downloaded but were deleted while it was not running, because -their paths will have last-downloaded records in the magic folder db -with a ``size`` other than ``None``, and without any corresponding -local file. - -Deletion of a directory -~~~~~~~~~~~~~~~~~~~~~~~ - -Local filesystems (unlike a Tahoe-LAFS filesystem) normally cannot -unlink a directory that has any remaining children. Therefore a -Magic Folder client cannot delete local copies of directories in -general, because they will typically contain backup files. This must -be done manually on each client if desired. - -Nevertheless, a Magic Folder client that deletes a directory should -set ``deleted: true`` on the metadata entry for the corresponding -zero-length file. This avoids the directory being recreated after -it has been manually deleted from a client. - -Renaming -~~~~~~~~ - -It is sufficient to handle renaming of a file by treating it as a -deletion and an addition under the new name. - -This also applies to directories, although users may find the -resulting behavior unintuitive: all of the files under the old name -will be renamed to backup filenames, and a new directory structure -created under the new name. We believe this is the best that can be -done without imposing unreasonable implementation complexity. - - -Summary -------- - -This completes the design of remote-to-local synchronization. -We realize that it may seem very complicated. Anecdotally, proprietary -filesystem synchronization designs we are aware of, such as Dropbox, -are said to incur similar or greater design complexity. diff --git a/docs/proposed/magic-folder/user-interface-design.rst b/docs/proposed/magic-folder/user-interface-design.rst deleted file mode 100644 index ea976bb00..000000000 --- a/docs/proposed/magic-folder/user-interface-design.rst +++ /dev/null @@ -1,205 +0,0 @@ -Magic Folder user interface design -================================== - -Scope ------ - -In this Objective we will design a user interface to allow users to conveniently -and securely indicate which folders on some devices should be "magically" linked -to which folders on other devices. - -This is a critical usability and security issue for which there is no known perfect -solution, but which we believe is amenable to a "good enough" trade-off solution. -This document explains the design and justifies its trade-offs in terms of security, -usability, and time-to-market. - -Tickets on the Tahoe-LAFS trac with the `otf-magic-folder-objective6`_ -keyword are within the scope of the user interface design. - -.. _otf-magic-folder-objective6: https://tahoe-lafs.org/trac/tahoe-lafs/query?status=!closed&keywords=~otf-magic-folder-objective6 - -Glossary -'''''''' - -Object: a file or directory - -DMD: distributed mutable directory - -Folder: an abstract directory that is synchronized between clients. -(A folder is not the same as the directory corresponding to it on -any particular client, nor is it the same as a DMD.) - -Collective: the set of clients subscribed to a given Magic Folder. - -Diminishing: the process of deriving, from an existing capability, -another capability that gives less authority (for example, deriving a -read cap from a read/write cap). - - -Design Constraints ------------------- - -The design of the Tahoe-side representation of a Magic Folder, and the -polling mechanism that the Magic Folder clients will use to detect remote -changes was discussed in :doc:`remote-to-local-sync`, -and we will not revisit that here. The assumption made by that design was -that each client would be configured with the following information: - -* a write cap to its own *client DMD*. -* a read cap to a *collective directory*. - -The collective directory contains links to each client DMD named by the -corresponding client's nickname. - -This design was chosen to allow straightforward addition of clients without -requiring each existing client to change its configuration. - -Note that each client in a Magic Folder collective has the authority to add, -modify or delete any object within the Magic Folder. It is also able to control -to some extent whether its writes will be treated by another client as overwrites -or as conflicts. However, there is still a reliability benefit to preventing a -client from accidentally modifying another client's DMD, or from accidentally -modifying the collective directory in a way that would lose data. This motivates -ensuring that each client only has access to the caps above, rather than, say, -every client having a write cap to the collective directory. - -Another important design constraint is that we cannot violate the :doc:`write -coordination directive<../../write_coordination>`; that is, we cannot write to -the same mutable directory from multiple clients, even during the setup phase -when adding a client. - -Within these constraints, for usability we want to minimize the number of steps -required to configure a Magic Folder collective. - - -Proposed Design ---------------- - -Three ``tahoe`` subcommands are added:: - - tahoe magic-folder create MAGIC: [MY_NICKNAME LOCAL_DIR] - - Create an empty Magic Folder. The MAGIC: local alias is set - to a write cap which can be used to refer to this Magic Folder - in future ``tahoe magic-folder invite`` commands. - - If MY_NICKNAME and LOCAL_DIR are given, the current client - immediately joins the newly created Magic Folder with that - nickname and local directory. - - - tahoe magic-folder invite MAGIC: THEIR_NICKNAME - - Print an "invitation" that can be used to invite another - client to join a Magic Folder, with the given nickname. - - The invitation must be sent to the user of the other client - over a secure channel (e.g. PGP email, OTR, or ssh). - - This command will normally be run by the same client that - created the Magic Folder. However, it may be run by a - different client if the ``MAGIC:`` alias is copied to - the ``private/aliases`` file of that other client, or if - ``MAGIC:`` is replaced by the write cap to which it points. - - - tahoe magic-folder join INVITATION LOCAL_DIR - - Accept an invitation created by ``tahoe magic-folder invite``. - The current client joins the specified Magic Folder, which will - appear in the local filesystem at the given directory. - - -There are no commands to remove a client or to revoke an -invitation, although those are possible features that could -be added in future. (When removing a client, it is necessary -to copy each file it added to some other client's DMD, if it -is the most recent version of that file.) - - -Implementation -'''''''''''''' - -For "``tahoe magic-folder create MAGIC: [MY_NICKNAME LOCAL_DIR]``" : - -1. Run "``tahoe create-alias MAGIC:``". -2. If ``MY_NICKNAME`` and ``LOCAL_DIR`` are given, do the equivalent of:: - - INVITATION=`tahoe invite-magic-folder MAGIC: MY_NICKNAME` - tahoe join-magic-folder INVITATION LOCAL_DIR - - -For "``tahoe magic-folder invite COLLECTIVE_WRITECAP NICKNAME``" : - -(``COLLECTIVE_WRITECAP`` can, as a special case, be an alias such as ``MAGIC:``.) - -1. Create an empty client DMD. Let its write URI be ``CLIENT_WRITECAP``. -2. Diminish ``CLIENT_WRITECAP`` to ``CLIENT_READCAP``, and - diminish ``COLLECTIVE_WRITECAP`` to ``COLLECTIVE_READCAP``. -3. Run "``tahoe ln CLIENT_READCAP COLLECTIVE_WRITECAP/NICKNAME``". -4. Print "``COLLECTIVE_READCAP+CLIENT_WRITECAP``" as the invitation, - accompanied by instructions on how to accept the invitation and - the need to send it over a secure channel. - - -For "``tahoe magic-folder join INVITATION LOCAL_DIR``" : - -1. Parse ``INVITATION`` as ``COLLECTIVE_READCAP+CLIENT_WRITECAP``. -2. Write ``CLIENT_WRITECAP`` to the file ``magic_folder_dircap`` - under the client's ``private`` directory. -3. Write ``COLLECTIVE_READCAP`` to the file ``collective_dircap`` - under the client's ``private`` directory. -4. Edit the client's ``tahoe.cfg`` to set - ``[magic_folder] enabled = True`` and - ``[magic_folder] local.directory = LOCAL_DIR``. - - -Discussion ----------- - -The proposed design has a minor violation of the -`Principle of Least Authority`_ in order to reduce the number -of steps needed. The invoker of "``tahoe magic-folder invite``" -creates the client DMD on behalf of the invited client, and -could retain its write cap (which is part of the invitation). - -.. _`Principle of Least Authority`: http://www.eros-os.org/papers/secnotsep.pdf - -A possible alternative design would be for the invited client -to create its own client DMD, and send it back to the inviter -to be linked into the collective directory. However this would -require another secure communication and another command -invocation per client. Given that, as mentioned earlier, each -client in a Magic Folder collective already has the authority -to add, modify or delete any object within the Magic Folder, -we considered the potential security/reliability improvement -here not to be worth the loss of usability. - -We also considered a design where each client had write access to -the collective directory. This would arguably be a more serious -violation of the Principle of Least Authority than the one above -(because all clients would have excess authority rather than just -the inviter). In any case, it was not clear how to make such a -design satisfy the :doc:`write coordination -directive<../../write_coordination>`, because the collective -directory would have needed to be written to by multiple clients. - -The reliance on a secure channel to send the invitation to its -intended recipient is not ideal, since it may involve additional -software such as clients for PGP, OTR, ssh etc. However, we believe -that this complexity is necessary rather than incidental, because -there must be some way to distinguish the intended recipient from -potential attackers who would try to become members of the Magic -Folder collective without authorization. By making use of existing -channels that have likely already been set up by security-conscious -users, we avoid reinventing the wheel or imposing substantial extra -implementation costs. - -The length of an invitation will be approximately the combined -length of a Tahoe-LAFS read cap and write cap. This is several -lines long, but still short enough to be cut-and-pasted successfully -if care is taken. Errors in copying the invitation can be detected -since Tahoe-LAFS cap URIs are self-authenticating. - -The implementation of the ``tahoe`` subcommands is straightforward -and raises no further difficult design issues. From 6bd2cc47b30d0f230b4c084cb7f50acd7c9ed228 Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Thu, 13 Feb 2020 10:26:48 -0500 Subject: [PATCH 04/25] Remove some git configuration related to magic-folder --- .gitignore | 1 - 1 file changed, 1 deletion(-) diff --git a/.gitignore b/.gitignore index bd76fc2ce..ee3b02b8f 100644 --- a/.gitignore +++ b/.gitignore @@ -43,7 +43,6 @@ zope.interface-*.egg /.tox/ /docs/_build/ /coverage.xml -/smoke_magicfolder/ /.hypothesis/ # This is the plaintext of the private environment needed for some CircleCI From 1fba2d6ea2d6ca59c427a58f034d8ad9c1081c6c Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Thu, 13 Feb 2020 10:26:57 -0500 Subject: [PATCH 05/25] Remove the magic-folder integration tests --- integration/conftest.py | 91 ------ integration/test_aaa_aardvark.py | 4 - integration/test_magic_folder.py | 462 ------------------------------- integration/test_tor.py | 2 +- integration/util.py | 4 - 5 files changed, 1 insertion(+), 562 deletions(-) delete mode 100644 integration/test_magic_folder.py diff --git a/integration/conftest.py b/integration/conftest.py index 10927f358..a85bc15f6 100644 --- a/integration/conftest.py +++ b/integration/conftest.py @@ -367,97 +367,6 @@ def bob(reactor, temp_dir, introducer_furl, flog_gatherer, storage_nodes, reques return process -@pytest.fixture(scope='session') -@log_call(action_type=u"integration:alice:invite", include_args=["temp_dir"]) -def alice_invite(reactor, alice, temp_dir, request): - node_dir = join(temp_dir, 'alice') - - with start_action(action_type=u"integration:alice:magic_folder:create"): - # FIXME XXX by the time we see "client running" in the logs, the - # storage servers aren't "really" ready to roll yet (uploads fairly - # consistently fail if we don't hack in this pause...) - proto = _CollectOutputProtocol() - _tahoe_runner_optional_coverage( - proto, - reactor, - request, - [ - 'magic-folder', 'create', - '--poll-interval', '2', - '--basedir', node_dir, 'magik:', 'alice', - join(temp_dir, 'magic-alice'), - ] - ) - pytest_twisted.blockon(proto.done) - - with start_action(action_type=u"integration:alice:magic_folder:invite") as a: - proto = _CollectOutputProtocol() - _tahoe_runner_optional_coverage( - proto, - reactor, - request, - [ - 'magic-folder', 'invite', - '--basedir', node_dir, 'magik:', 'bob', - ] - ) - pytest_twisted.blockon(proto.done) - invite = proto.output.getvalue() - a.add_success_fields(invite=invite) - - with start_action(action_type=u"integration:alice:magic_folder:restart"): - # before magic-folder works, we have to stop and restart (this is - # crappy for the tests -- can we fix it in magic-folder?) - try: - alice.transport.signalProcess('TERM') - pytest_twisted.blockon(alice.transport.exited) - except ProcessExitedAlready: - pass - with start_action(action_type=u"integration:alice:magic_folder:magic-text"): - magic_text = 'Completed initial Magic Folder scan successfully' - pytest_twisted.blockon(_run_node(reactor, node_dir, request, magic_text)) - await_client_ready(alice) - return invite - - -@pytest.fixture(scope='session') -@log_call( - action_type=u"integration:magic_folder", - include_args=["alice_invite", "temp_dir"], -) -def magic_folder(reactor, alice_invite, alice, bob, temp_dir, request): - print("pairing magic-folder") - bob_dir = join(temp_dir, 'bob') - proto = _CollectOutputProtocol() - _tahoe_runner_optional_coverage( - proto, - reactor, - request, - [ - 'magic-folder', 'join', - '--poll-interval', '1', - '--basedir', bob_dir, - alice_invite, - join(temp_dir, 'magic-bob'), - ] - ) - pytest_twisted.blockon(proto.done) - - # before magic-folder works, we have to stop and restart (this is - # crappy for the tests -- can we fix it in magic-folder?) - try: - print("Sending TERM to Bob") - bob.transport.signalProcess('TERM') - pytest_twisted.blockon(bob.transport.exited) - except ProcessExitedAlready: - pass - - magic_text = 'Completed initial Magic Folder scan successfully' - pytest_twisted.blockon(_run_node(reactor, bob_dir, request, magic_text)) - await_client_ready(bob) - return (join(temp_dir, 'magic-alice'), join(temp_dir, 'magic-bob')) - - @pytest.fixture(scope='session') def chutney(reactor, temp_dir): chutney_dir = join(temp_dir, 'chutney') diff --git a/integration/test_aaa_aardvark.py b/integration/test_aaa_aardvark.py index e4698a12a..4a2ef71a6 100644 --- a/integration/test_aaa_aardvark.py +++ b/integration/test_aaa_aardvark.py @@ -16,7 +16,3 @@ def test_create_introducer(introducer): def test_create_storage(storage_nodes): print("Created {} storage nodes".format(len(storage_nodes))) - - -def test_create_alice_bob_magicfolder(magic_folder): - print("Alice and Bob have paired magic-folders") diff --git a/integration/test_magic_folder.py b/integration/test_magic_folder.py deleted file mode 100644 index 32323cb64..000000000 --- a/integration/test_magic_folder.py +++ /dev/null @@ -1,462 +0,0 @@ -import sys -import time -import shutil -from os import mkdir, unlink, utime -from os.path import join, exists, getmtime - -import util - -import pytest_twisted - - -# tests converted from check_magicfolder_smoke.py -# see "conftest.py" for the fixtures (e.g. "magic_folder") - -def test_eliot_logs_are_written(alice, bob, temp_dir): - # The integration test configuration arranges for this logging - # configuration. Verify it actually does what we want. - # - # The alice and bob arguments looks unused but they actually tell pytest - # to set up all the magic-folder stuff. The assertions here are about - # side-effects of that setup. - assert exists(join(temp_dir, "alice", "logs", "eliot.json")) - assert exists(join(temp_dir, "bob", "logs", "eliot.json")) - - -def test_alice_writes_bob_receives(magic_folder): - alice_dir, bob_dir = magic_folder - - with open(join(alice_dir, "first_file"), "w") as f: - f.write("alice wrote this") - - util.await_file_contents(join(bob_dir, "first_file"), "alice wrote this") - return - - -def test_alice_writes_bob_receives_multiple(magic_folder): - """ - When Alice does a series of updates, Bob should just receive them - with no .backup or .conflict files being produced. - """ - alice_dir, bob_dir = magic_folder - - unwanted_files = [ - join(bob_dir, "multiple.backup"), - join(bob_dir, "multiple.conflict") - ] - - # first update - with open(join(alice_dir, "multiple"), "w") as f: - f.write("alice wrote this") - - util.await_file_contents( - join(bob_dir, "multiple"), "alice wrote this", - error_if=unwanted_files, - ) - - # second update - with open(join(alice_dir, "multiple"), "w") as f: - f.write("someone changed their mind") - - util.await_file_contents( - join(bob_dir, "multiple"), "someone changed their mind", - error_if=unwanted_files, - ) - - # third update - with open(join(alice_dir, "multiple"), "w") as f: - f.write("absolutely final version ship it") - - util.await_file_contents( - join(bob_dir, "multiple"), "absolutely final version ship it", - error_if=unwanted_files, - ) - - # forth update, but both "at once" so one should conflict - time.sleep(2) - with open(join(alice_dir, "multiple"), "w") as f: - f.write("okay one more attempt") - with open(join(bob_dir, "multiple"), "w") as f: - f.write("...but just let me add") - - bob_conflict = join(bob_dir, "multiple.conflict") - alice_conflict = join(alice_dir, "multiple.conflict") - - found = util.await_files_exist([ - bob_conflict, - alice_conflict, - ]) - - assert len(found) > 0, "Should have found a conflict" - print("conflict found (as expected)") - - -def test_alice_writes_bob_receives_old_timestamp(magic_folder): - alice_dir, bob_dir = magic_folder - fname = join(alice_dir, "ts_file") - ts = time.time() - (60 * 60 * 36) # 36 hours ago - - with open(fname, "w") as f: - f.write("alice wrote this") - utime(fname, (time.time(), ts)) - - fname = join(bob_dir, "ts_file") - util.await_file_contents(fname, "alice wrote this") - # make sure the timestamp is correct - assert int(getmtime(fname)) == int(ts) - return - - -def test_bob_writes_alice_receives(magic_folder): - alice_dir, bob_dir = magic_folder - - with open(join(bob_dir, "second_file"), "w") as f: - f.write("bob wrote this") - - util.await_file_contents(join(alice_dir, "second_file"), "bob wrote this") - return - - -def test_alice_deletes(magic_folder): - # alice writes a file, waits for bob to get it and then deletes it. - alice_dir, bob_dir = magic_folder - - with open(join(alice_dir, "delfile"), "w") as f: - f.write("alice wrote this") - - util.await_file_contents(join(bob_dir, "delfile"), "alice wrote this") - - # bob has the file; now alices deletes it - unlink(join(alice_dir, "delfile")) - - # bob should remove his copy, but preserve a backup - util.await_file_vanishes(join(bob_dir, "delfile")) - util.await_file_contents(join(bob_dir, "delfile.backup"), "alice wrote this") - return - - -def test_alice_creates_bob_edits(magic_folder): - alice_dir, bob_dir = magic_folder - - # alice writes a file - with open(join(alice_dir, "editfile"), "w") as f: - f.write("alice wrote this") - - util.await_file_contents(join(bob_dir, "editfile"), "alice wrote this") - - # now bob edits it - with open(join(bob_dir, "editfile"), "w") as f: - f.write("bob says foo") - - util.await_file_contents(join(alice_dir, "editfile"), "bob says foo") - - -def test_bob_creates_sub_directory(magic_folder): - alice_dir, bob_dir = magic_folder - - # bob makes a sub-dir, with a file in it - mkdir(join(bob_dir, "subdir")) - with open(join(bob_dir, "subdir", "a_file"), "w") as f: - f.write("bob wuz here") - - # alice gets it - util.await_file_contents(join(alice_dir, "subdir", "a_file"), "bob wuz here") - - # now bob deletes it again - shutil.rmtree(join(bob_dir, "subdir")) - - # alice should delete it as well - util.await_file_vanishes(join(alice_dir, "subdir", "a_file")) - # i *think* it's by design that the subdir won't disappear, - # because a "a_file.backup" should appear... - util.await_file_contents(join(alice_dir, "subdir", "a_file.backup"), "bob wuz here") - - -def test_bob_creates_alice_deletes_bob_restores(magic_folder): - alice_dir, bob_dir = magic_folder - - # bob creates a file - with open(join(bob_dir, "boom"), "w") as f: - f.write("bob wrote this") - - util.await_file_contents( - join(alice_dir, "boom"), - "bob wrote this" - ) - - # alice deletes it (so bob should as well .. but keep a backup) - unlink(join(alice_dir, "boom")) - util.await_file_vanishes(join(bob_dir, "boom")) - assert exists(join(bob_dir, "boom.backup")) - - # bob restore it, with new contents - unlink(join(bob_dir, "boom.backup")) - with open(join(bob_dir, "boom"), "w") as f: - f.write("bob wrote this again, because reasons") - - # XXX double-check this behavior is correct! - - # alice sees bob's update, but marks it as a conflict (because - # .. she previously deleted it? does that really make sense) - - util.await_file_contents( - join(alice_dir, "boom"), - "bob wrote this again, because reasons", - ) - - -def test_bob_creates_alice_deletes_alice_restores(magic_folder): - alice_dir, bob_dir = magic_folder - - # bob creates a file - with open(join(bob_dir, "boom2"), "w") as f: - f.write("bob wrote this") - - util.await_file_contents( - join(alice_dir, "boom2"), - "bob wrote this" - ) - - # alice deletes it (so bob should as well) - unlink(join(alice_dir, "boom2")) - util.await_file_vanishes(join(bob_dir, "boom2")) - - # alice restore it, with new contents - with open(join(alice_dir, "boom2"), "w") as f: - f.write("alice re-wrote this again, because reasons") - - util.await_file_contents( - join(bob_dir, "boom2"), - "alice re-wrote this again, because reasons" - ) - - -def test_bob_conflicts_with_alice_fresh(magic_folder): - # both alice and bob make a file at "the same time". - alice_dir, bob_dir = magic_folder - - # either alice or bob will "win" by uploading to the DMD first. - with open(join(bob_dir, 'alpha'), 'w') as f0, open(join(alice_dir, 'alpha'), 'w') as f1: - f0.write("this is bob's alpha\n") - f1.write("this is alice's alpha\n") - - # there should be conflicts - _bob_conflicts_alice_await_conflicts('alpha', alice_dir, bob_dir) - - -def test_bob_conflicts_with_alice_preexisting(magic_folder): - # both alice and bob edit a file at "the same time" (similar to - # above, but the file already exists before the edits) - alice_dir, bob_dir = magic_folder - - # have bob create the file - with open(join(bob_dir, 'beta'), 'w') as f: - f.write("original beta (from bob)\n") - util.await_file_contents(join(alice_dir, 'beta'), "original beta (from bob)\n") - - # both alice and bob now have a "beta" file, at version 0 - - # either alice or bob will "win" by uploading to the DMD first - # (however, they should both detect a conflict) - with open(join(bob_dir, 'beta'), 'w') as f: - f.write("this is bob's beta\n") - with open(join(alice_dir, 'beta'), 'w') as f: - f.write("this is alice's beta\n") - - # both alice and bob should see a conflict - _bob_conflicts_alice_await_conflicts("beta", alice_dir, bob_dir) - - -def _bob_conflicts_alice_await_conflicts(name, alice_dir, bob_dir): - """ - shared code between _fresh and _preexisting conflict test - """ - found = util.await_files_exist( - [ - join(bob_dir, '{}.conflict'.format(name)), - join(alice_dir, '{}.conflict'.format(name)), - ], - ) - - assert len(found) >= 1, "should be at least one conflict" - assert open(join(bob_dir, name), 'r').read() == "this is bob's {}\n".format(name) - assert open(join(alice_dir, name), 'r').read() == "this is alice's {}\n".format(name) - - alice_conflict = join(alice_dir, '{}.conflict'.format(name)) - bob_conflict = join(bob_dir, '{}.conflict'.format(name)) - if exists(bob_conflict): - assert open(bob_conflict, 'r').read() == "this is alice's {}\n".format(name) - if exists(alice_conflict): - assert open(alice_conflict, 'r').read() == "this is bob's {}\n".format(name) - - -@pytest_twisted.inlineCallbacks -def test_edmond_uploads_then_restarts(reactor, request, temp_dir, introducer_furl, flog_gatherer, storage_nodes): - """ - ticket 2880: if a magic-folder client uploads something, then - re-starts a spurious .backup file should not appear - """ - - edmond_dir = join(temp_dir, 'edmond') - edmond = yield util._create_node( - reactor, request, temp_dir, introducer_furl, flog_gatherer, - "edmond", web_port="tcp:9985:interface=localhost", - storage=False, - ) - - - magic_folder = join(temp_dir, 'magic-edmond') - mkdir(magic_folder) - created = False - # create a magic-folder - # (how can we know that the grid is ready?) - for _ in range(10): # try 10 times - try: - proto = util._CollectOutputProtocol() - transport = reactor.spawnProcess( - proto, - sys.executable, - [ - sys.executable, '-m', 'allmydata.scripts.runner', - 'magic-folder', 'create', - '--poll-interval', '2', - '--basedir', edmond_dir, - 'magik:', - 'edmond_magic', - magic_folder, - ] - ) - yield proto.done - created = True - break - except Exception as e: - print("failed to create magic-folder: {}".format(e)) - time.sleep(1) - - assert created, "Didn't create a magic-folder" - - # to actually-start the magic-folder we have to re-start - edmond.transport.signalProcess('TERM') - yield edmond.transport.exited - edmond = yield util._run_node(reactor, edmond.node_dir, request, 'Completed initial Magic Folder scan successfully') - util.await_client_ready(edmond) - - # add a thing to the magic-folder - with open(join(magic_folder, "its_a_file"), "w") as f: - f.write("edmond wrote this") - - # fixme, do status-update attempts in a loop below - time.sleep(5) - - # let it upload; poll the HTTP magic-folder status API until it is - # uploaded - from allmydata.scripts.magic_folder_cli import _get_json_for_fragment - - with open(join(edmond_dir, u'private', u'api_auth_token'), 'rb') as f: - token = f.read() - - uploaded = False - for _ in range(10): - options = { - "node-url": open(join(edmond_dir, u'node.url'), 'r').read().strip(), - } - try: - magic_data = _get_json_for_fragment( - options, - 'magic_folder?t=json', - method='POST', - post_args=dict( - t='json', - name='default', - token=token, - ) - ) - for mf in magic_data: - if mf['status'] == u'success' and mf['path'] == u'its_a_file': - uploaded = True - break - except Exception as e: - time.sleep(1) - - assert uploaded, "expected to upload 'its_a_file'" - - # re-starting edmond right now would "normally" trigger the 2880 bug - - # kill edmond - edmond.transport.signalProcess('TERM') - yield edmond.transport.exited - time.sleep(1) - edmond = yield util._run_node(reactor, edmond.node_dir, request, 'Completed initial Magic Folder scan successfully') - util.await_client_ready(edmond) - - # XXX how can we say for sure if we've waited long enough? look at - # tail of logs for magic-folder ... somethingsomething? - print("waiting 20 seconds to see if a .backup appears") - for _ in range(20): - assert exists(join(magic_folder, "its_a_file")) - assert not exists(join(magic_folder, "its_a_file.backup")) - time.sleep(1) - - -@pytest_twisted.inlineCallbacks -def test_alice_adds_files_while_bob_is_offline(reactor, request, temp_dir, magic_folder): - """ - Alice can add new files to a magic folder while Bob is offline. When Bob - comes back online his copy is updated to reflect the new files. - """ - alice_magic_dir, bob_magic_dir = magic_folder - alice_node_dir = join(temp_dir, "alice") - bob_node_dir = join(temp_dir, "bob") - - # Take Bob offline. - yield util.cli(request, reactor, bob_node_dir, "stop") - - # Create a couple files in Alice's local directory. - some_files = list( - (name * 3) + ".added-while-offline" - for name - in "xyz" - ) - for name in some_files: - with open(join(alice_magic_dir, name), "w") as f: - f.write(name + " some content") - - good = False - for i in range(15): - status = yield util.magic_folder_cli(request, reactor, alice_node_dir, "status") - good = status.count(".added-while-offline (36 B): good, version=0") == len(some_files) * 2 - if good: - # We saw each file as having a local good state and a remote good - # state. That means we're ready to involve Bob. - break - else: - time.sleep(1.0) - - assert good, ( - "Timed out waiting for good Alice state. Last status:\n{}".format(status) - ) - - # Start Bob up again - magic_text = 'Completed initial Magic Folder scan successfully' - yield util._run_node(reactor, bob_node_dir, request, magic_text) - - yield util.await_files_exist( - list( - join(bob_magic_dir, name) - for name - in some_files - ), - await_all=True, - ) - # Let it settle. It would be nicer to have a readable status output we - # could query. Parsing the current text format is more than I want to - # deal with right now. - time.sleep(1.0) - conflict_files = list(name + ".conflict" for name in some_files) - assert all( - list( - not exists(join(bob_magic_dir, name)) - for name - in conflict_files - ), - ) diff --git a/integration/test_tor.py b/integration/test_tor.py index 187754f08..633def8de 100644 --- a/integration/test_tor.py +++ b/integration/test_tor.py @@ -14,7 +14,7 @@ import pytest_twisted import util -# see "conftest.py" for the fixtures (e.g. "magic_folder") +# see "conftest.py" for the fixtures (e.g. "tor_network") @pytest_twisted.inlineCallbacks def test_onion_service_storage(reactor, request, temp_dir, flog_gatherer, tor_network, tor_introducer_furl): diff --git a/integration/util.py b/integration/util.py index aef9ba544..bbcf5efc6 100644 --- a/integration/util.py +++ b/integration/util.py @@ -498,7 +498,3 @@ def await_client_ready(tahoe, timeout=10, liveness=60*2): tahoe, ) ) - - -def magic_folder_cli(request, reactor, node_dir, *argv): - return cli(request, reactor, node_dir, "magic-folder", *argv) From ef73e18f3df31fb2f274ecdb9e8336f857d9734e Mon Sep 17 00:00:00 2001 From: Jean-Paul Calderone Date: Thu, 13 Feb 2020 10:43:50 -0500 Subject: [PATCH 06/25] Remove the greater part of the implementation --- src/allmydata/client.py | 48 - src/allmydata/frontends/magic_folder.py | 2111 -------------- src/allmydata/magicfolderdb.py | 204 -- src/allmydata/magicpath.py | 32 - src/allmydata/scripts/magic_folder_cli.py | 610 ----- src/allmydata/scripts/runner.py | 7 +- src/allmydata/test/cli/test_magic_folder.py | 814 ------ src/allmydata/test/test_client.py | 99 - src/allmydata/test/test_inotify.py | 171 -- src/allmydata/test/test_magic_folder.py | 2425 ----------------- src/allmydata/test/test_magicpath.py | 28 - .../test/test_python2_regressions.py | 3 - src/allmydata/test/test_websocket_logs.py | 2 +- src/allmydata/test/web/test_web.py | 98 - src/allmydata/util/eliotutil.py | 83 - src/allmydata/util/fake_inotify.py | 109 - src/allmydata/watchdog/__init__.py | 0 src/allmydata/watchdog/_watchdog_541.py | 16 - src/allmydata/watchdog/inotify.py | 212 -- src/allmydata/web/magic_folder.py | 52 - src/allmydata/web/root.py | 29 +- src/allmydata/web/static/css/new-tahoe.css | 5 - src/allmydata/web/statistics.xhtml | 12 - src/allmydata/web/status.py | 30 - src/allmydata/web/welcome.xhtml | 7 - src/allmydata/windows/inotify.py | 379 --- 26 files changed, 3 insertions(+), 7583 deletions(-) delete mode 100644 src/allmydata/frontends/magic_folder.py delete mode 100644 src/allmydata/magicfolderdb.py delete mode 100644 src/allmydata/magicpath.py delete mode 100644 src/allmydata/scripts/magic_folder_cli.py delete mode 100644 src/allmydata/test/cli/test_magic_folder.py delete mode 100644 src/allmydata/test/test_inotify.py delete mode 100644 src/allmydata/test/test_magic_folder.py delete mode 100644 src/allmydata/test/test_magicpath.py delete mode 100644 src/allmydata/util/fake_inotify.py delete mode 100644 src/allmydata/watchdog/__init__.py delete mode 100644 src/allmydata/watchdog/_watchdog_541.py delete mode 100644 src/allmydata/watchdog/inotify.py delete mode 100644 src/allmydata/web/magic_folder.py delete mode 100644 src/allmydata/windows/inotify.py diff --git a/src/allmydata/client.py b/src/allmydata/client.py index e45666049..5e8bf0d6d 100644 --- a/src/allmydata/client.py +++ b/src/allmydata/client.py @@ -85,9 +85,6 @@ _client_config = configutil.ValidConfiguration( "stats_gatherer.furl", "storage.plugins", ), - "drop_upload": ( # deprecated already? - "enabled", - ), "ftpd": ( "accounts.file", "accounts.url", @@ -121,12 +118,6 @@ _client_config = configutil.ValidConfiguration( "helper": ( "enabled", ), - "magic_folder": ( - "download.umask", - "enabled", - "local.directory", - "poll_interval", - ), }, is_valid_section=_is_valid_section, # Anything in a valid section is a valid item, for now. @@ -681,7 +672,6 @@ class _Client(node.Node, pollmixin.PollMixin): """ node.Node.__init__(self, config, main_tub, control_tub, i2p_provider, tor_provider) - self._magic_folders = dict() self.started_timestamp = time.time() self.logSource = "Client" self.encoding_params = self.DEFAULT_ENCODING_PARAMETERS.copy() @@ -707,7 +697,6 @@ class _Client(node.Node, pollmixin.PollMixin): self.init_helper() self.init_ftp_server() self.init_sftp_server() - self.init_magic_folder() # If the node sees an exit_trigger file, it will poll every second to see # whether the file still exists, and what its mtime is. If the file does not @@ -968,9 +957,6 @@ class _Client(node.Node, pollmixin.PollMixin): This returns a local authentication token, which is just some random data in "api_auth_token" which must be echoed to API calls. - - Currently only the URI '/magic' for magic-folder status; other - endpoints are invited to include this as well, as appropriate. """ return self.config.get_private_config('api_auth_token') @@ -1088,40 +1074,6 @@ class _Client(node.Node, pollmixin.PollMixin): sftp_portstr, pubkey_file, privkey_file) s.setServiceParent(self) - def init_magic_folder(self): - #print "init_magic_folder" - if self.config.get_config("drop_upload", "enabled", False, boolean=True): - raise node.OldConfigOptionError( - "The [drop_upload] section must be renamed to [magic_folder].\n" - "See docs/frontends/magic-folder.rst for more information." - ) - - if self.config.get_config("magic_folder", "enabled", False, boolean=True): - from allmydata.frontends import magic_folder - - try: - magic_folders = magic_folder.load_magic_folders(self.config._basedir) - except Exception as e: - log.msg("Error loading magic-folder config: {}".format(e)) - raise - - # start processing the upload queue when we've connected to - # enough servers - threshold = min(self.encoding_params["k"], - self.encoding_params["happy"] + 1) - - for (name, mf_config) in magic_folders.items(): - self.log("Starting magic_folder '{}'".format(name)) - s = magic_folder.MagicFolder.from_config(self, name, mf_config) - self._magic_folders[name] = s - s.setServiceParent(self) - - connected_d = self.storage_broker.when_connected_enough(threshold) - def connected_enough(ign, mf): - mf.ready() # returns a Deferred we ignore - return None - connected_d.addCallback(connected_enough, s) - def _check_exit_trigger(self, exit_trigger_file): if os.path.exists(exit_trigger_file): mtime = os.stat(exit_trigger_file)[stat.ST_MTIME] diff --git a/src/allmydata/frontends/magic_folder.py b/src/allmydata/frontends/magic_folder.py deleted file mode 100644 index bb2edce99..000000000 --- a/src/allmydata/frontends/magic_folder.py +++ /dev/null @@ -1,2111 +0,0 @@ -import six -import sys, os -import os.path -from errno import EEXIST -from collections import deque -from datetime import datetime -import time -import ConfigParser - -from twisted.python.log import msg as twmsg -from twisted.python.filepath import FilePath -from twisted.python.monkey import MonkeyPatcher -from twisted.internet import defer, reactor, task -from twisted.internet.error import AlreadyCancelled -from twisted.python.failure import Failure -from twisted.python import runtime -from twisted.application import service - -from zope.interface import Interface, Attribute, implementer - -from eliot import ( - Field, - Message, - start_action, - ActionType, - MessageType, - write_failure, - write_traceback, - log_call, -) -from eliot.twisted import ( - DeferredContext, -) - -from allmydata.util import ( - fileutil, - configutil, - yamlutil, - eliotutil, -) -from allmydata.interfaces import IDirectoryNode -from allmydata.util import log -from allmydata.util.fileutil import ( - precondition_abspath, - get_pathinfo, - ConflictError, - abspath_expanduser_unicode, -) -from allmydata.util.assertutil import precondition, _assert -from allmydata.util.deferredutil import HookMixin -from allmydata.util.progress import PercentProgress -from allmydata.util.encodingutil import listdir_filepath, to_filepath, \ - extend_filepath, unicode_from_filepath, unicode_segments_from, \ - quote_filepath, quote_local_unicode_path, FilenameEncodingError -from allmydata.util.time_format import format_time -from allmydata.immutable.upload import FileName, Data -from allmydata import magicfolderdb, magicpath - -if six.PY3: - long = int - - -# Mask off all non-owner permissions for magic-folders files by default. -_DEFAULT_DOWNLOAD_UMASK = 0o077 - -IN_EXCL_UNLINK = long(0x04000000) - - -class ConfigurationError(Exception): - """ - There was something wrong with some magic-folder configuration. - """ - - -def _get_inotify_module(): - try: - if sys.platform == "win32": - from allmydata.windows import inotify - elif runtime.platform.supportsINotify(): - from twisted.internet import inotify - elif not sys.platform.startswith("linux"): - from allmydata.watchdog import inotify - else: - raise NotImplementedError("filesystem notification needed for Magic Folder is not supported.\n" - "This currently requires Linux, Windows, or macOS.") - return inotify - except (ImportError, AttributeError) as e: - log.msg(e) - if sys.platform == "win32": - raise NotImplementedError("filesystem notification needed for Magic Folder is not supported.\n" - "Windows support requires at least Vista, and has only been tested on Windows 7.") - raise - - -def get_inotify_module(): - # Until Twisted #9579 is fixed, the Docker check just screws things up. - # Disable it. - monkey = MonkeyPatcher() - monkey.addPatch(runtime.platform, "isDocker", lambda: False) - return monkey.runWithPatches(_get_inotify_module) - - -def is_new_file(pathinfo, db_entry): - if db_entry is None: - return True - - if not pathinfo.exists and db_entry.size is None: - return False - - return ((pathinfo.size, pathinfo.ctime_ns, pathinfo.mtime_ns) != - (db_entry.size, db_entry.ctime_ns, db_entry.mtime_ns)) - - -def _upgrade_magic_folder_config(basedir): - """ - Helper that upgrades from single-magic-folder-only configs to - multiple magic-folder configuration style (in YAML) - """ - config_fname = os.path.join(basedir, "tahoe.cfg") - config = configutil.get_config(config_fname) - - collective_fname = os.path.join(basedir, "private", "collective_dircap") - upload_fname = os.path.join(basedir, "private", "magic_folder_dircap") - magic_folders = { - u"default": { - u"directory": config.get("magic_folder", "local.directory").decode("utf-8"), - u"collective_dircap": fileutil.read(collective_fname), - u"upload_dircap": fileutil.read(upload_fname), - u"poll_interval": int(config.get("magic_folder", "poll_interval")), - }, - } - fileutil.move_into_place( - source=os.path.join(basedir, "private", "magicfolderdb.sqlite"), - dest=os.path.join(basedir, "private", "magicfolder_default.sqlite"), - ) - save_magic_folders(basedir, magic_folders) - config.remove_option("magic_folder", "local.directory") - config.remove_option("magic_folder", "poll_interval") - configutil.write_config(os.path.join(basedir, 'tahoe.cfg'), config) - fileutil.remove_if_possible(collective_fname) - fileutil.remove_if_possible(upload_fname) - - -def maybe_upgrade_magic_folders(node_directory): - """ - If the given node directory is not already using the new-style - magic-folder config it will be upgraded to do so. (This should - only be done if the user is running a command that needs to modify - the config) - """ - yaml_fname = os.path.join(node_directory, u"private", u"magic_folders.yaml") - if os.path.exists(yaml_fname): - # we already have new-style magic folders - return - - config_fname = os.path.join(node_directory, "tahoe.cfg") - config = configutil.get_config(config_fname) - - # we have no YAML config; if we have config in tahoe.cfg then we - # can upgrade it to the YAML-based configuration - if config.has_option("magic_folder", "local.directory"): - _upgrade_magic_folder_config(node_directory) - - -def load_magic_folders(node_directory): - """ - Loads existing magic-folder configuration and returns it as a dict - mapping name -> dict of config. This will NOT upgrade from - old-style to new-style config (but WILL read old-style config and - return in the same way as if it was new-style). - - :param node_directory: path where node data is stored - :returns: dict mapping magic-folder-name to its config (also a dict) - """ - yaml_fname = os.path.join(node_directory, u"private", u"magic_folders.yaml") - folders = dict() - - config_fname = os.path.join(node_directory, "tahoe.cfg") - config = configutil.get_config(config_fname) - - if not os.path.exists(yaml_fname): - # there will still be a magic_folder section in a "new" - # config, but it won't have local.directory nor poll_interval - # in it. - if config.has_option("magic_folder", "local.directory"): - up_fname = os.path.join(node_directory, "private", "magic_folder_dircap") - coll_fname = os.path.join(node_directory, "private", "collective_dircap") - directory = config.get("magic_folder", "local.directory").decode('utf8') - try: - interval = int(config.get("magic_folder", "poll_interval")) - except ConfigParser.NoOptionError: - interval = 60 - - if config.has_option("magic_folder", "download.umask"): - umask = int(config.get("magic_folder", "download.umask"), 8) - else: - umask = _DEFAULT_DOWNLOAD_UMASK - - folders[u"default"] = { - u"directory": directory, - u"upload_dircap": fileutil.read(up_fname), - u"collective_dircap": fileutil.read(coll_fname), - u"poll_interval": interval, - u"umask": umask, - } - else: - # without any YAML file AND no local.directory option it's - # an error if magic-folder is "enabled" because we don't - # actually have enough config for any magic-folders at all - if config.has_section("magic_folder") \ - and config.getboolean("magic_folder", "enabled") \ - and not folders: - raise Exception( - "[magic_folder] is enabled but has no YAML file and no " - "'local.directory' option." - ) - - elif os.path.exists(yaml_fname): # yaml config-file exists - if config.has_option("magic_folder", "local.directory"): - raise Exception( - "magic-folder config has both old-style configuration" - " and new-style configuration; please remove the " - "'local.directory' key from tahoe.cfg or remove " - "'magic_folders.yaml' from {}".format(node_directory) - ) - with open(yaml_fname, "r") as f: - magic_folders = yamlutil.safe_load(f.read()) - if not isinstance(magic_folders, dict): - raise Exception( - "'{}' should contain a dict".format(yaml_fname) - ) - - folders = magic_folders['magic-folders'] - if not isinstance(folders, dict): - raise Exception( - "'magic-folders' in '{}' should be a dict".format(yaml_fname) - ) - - # check configuration - folders = dict( - (name, fix_magic_folder_config(yaml_fname, name, config)) - for (name, config) - in folders.items() - ) - return folders - - -def fix_magic_folder_config(yaml_fname, name, config): - """ - Check the given folder configuration for validity. - - If it refers to a local directory which does not exist, create that - directory with the configured permissions. - - :param unicode yaml_fname: The configuration file from which the - configuration was read. - - :param unicode name: The name of the magic-folder this particular - configuration blob is associated with. - - :param config: The configuration for a single magic-folder. This is - expected to be a ``dict`` with certain keys and values of certain - types but these properties will be checked. - - :raise ConfigurationError: If the given configuration object does not - conform to some magic-folder configuration requirement. - """ - if not isinstance(config, dict): - raise ConfigurationError( - "Each item in '{}' must itself be a dict".format(yaml_fname) - ) - - for k in ['collective_dircap', 'upload_dircap', 'directory', 'poll_interval']: - if k not in config: - raise ConfigurationError( - "Config for magic folder '{}' is missing '{}'".format( - name, k - ) - ) - - if not isinstance( - config.setdefault(u"umask", _DEFAULT_DOWNLOAD_UMASK), - int, - ): - raise Exception("magic-folder download umask must be an integer") - - # make sure directory for magic folder exists - dir_fp = to_filepath(config['directory']) - umask = config.setdefault('umask', 0o077) - - try: - os.mkdir(dir_fp.path, 0o777 & (~ umask)) - except OSError as e: - if EEXIST != e.errno: - # Report some unknown problem. - raise ConfigurationError( - "magic-folder {} configured path {} could not be created: " - "{}".format( - name, - dir_fp.path, - str(e), - ), - ) - elif not dir_fp.isdir(): - # Tell the user there's a collision. - raise ConfigurationError( - "magic-folder {} configured path {} exists and is not a " - "directory".format( - name, dir_fp.path, - ), - ) - - result_config = config.copy() - for k in ['collective_dircap', 'upload_dircap']: - if isinstance(config[k], unicode): - result_config[k] = config[k].encode('ascii') - return result_config - - - -def save_magic_folders(node_directory, folders): - fileutil.write_atomically( - os.path.join(node_directory, u"private", u"magic_folders.yaml"), - yamlutil.safe_dump({u"magic-folders": folders}), - ) - - config = configutil.get_config(os.path.join(node_directory, u"tahoe.cfg")) - configutil.set_config(config, "magic_folder", "enabled", "True") - configutil.write_config(os.path.join(node_directory, u"tahoe.cfg"), config) - - -class MagicFolder(service.MultiService): - - @classmethod - def from_config(cls, client_node, name, config): - """ - Create a ``MagicFolder`` from a client node and magic-folder - configuration. - - :param _Client client_node: The client node the magic-folder is - attached to. - - :param dict config: Magic-folder configuration like that in the list - returned by ``load_magic_folders``. - """ - db_filename = client_node.config.get_private_path("magicfolder_{}.sqlite".format(name)) - local_dir_config = config['directory'] - try: - poll_interval = int(config["poll_interval"]) - except ValueError: - raise ValueError("'poll_interval' option must be an int") - - return cls( - client=client_node, - upload_dircap=config["upload_dircap"], - collective_dircap=config["collective_dircap"], - # XXX surely a better way for this local_path_u business - local_path_u=abspath_expanduser_unicode( - local_dir_config, - base=client_node.config.get_config_path(), - ), - dbfile=abspath_expanduser_unicode(db_filename), - umask=config["umask"], - name=name, - downloader_delay=poll_interval, - ) - - def __init__(self, client, upload_dircap, collective_dircap, local_path_u, dbfile, umask, - name, uploader_delay=1.0, clock=None, downloader_delay=60): - precondition_abspath(local_path_u) - if not os.path.exists(local_path_u): - raise ValueError("'{}' does not exist".format(local_path_u)) - if not os.path.isdir(local_path_u): - raise ValueError("'{}' is not a directory".format(local_path_u)) - # this is used by 'service' things and must be unique in this Service hierarchy - self.name = 'magic-folder-{}'.format(name) - - service.MultiService.__init__(self) - - clock = clock or reactor - db = magicfolderdb.get_magicfolderdb(dbfile, create_version=(magicfolderdb.SCHEMA_v1, 1)) - if db is None: - raise Exception('ERROR: Unable to load magic folder db.') - - # for tests - self._client = client - self._db = db - - upload_dirnode = self._client.create_node_from_uri(upload_dircap) - collective_dirnode = self._client.create_node_from_uri(collective_dircap) - - self.uploader = Uploader(client, local_path_u, db, upload_dirnode, uploader_delay, clock) - self.downloader = Downloader(client, local_path_u, db, collective_dirnode, - upload_dirnode.get_readonly_uri(), clock, self.uploader.is_pending, umask, - self.set_public_status, poll_interval=downloader_delay) - self._public_status = (False, ['Magic folder has not yet started']) - - def get_public_status(self): - """ - For the web UI, basically. - """ - return self._public_status - - def set_public_status(self, status, *messages): - self._public_status = (status, messages) - - def startService(self): - service.MultiService.startService(self) - return self.uploader.start_monitoring() - - def stopService(self): - with MAGIC_FOLDER_STOP(nickname=self.name).context(): - d = DeferredContext(self._finish()) - d.addBoth( - lambda ign: service.MultiService.stopService(self) - ) - return d.addActionFinish() - - def ready(self): - """ready is used to signal us to start - processing the upload and download items... - """ - self.uploader.start_uploading() # synchronous, returns None - return self.downloader.start_downloading() - - def _finish(self): - d0 = self.downloader.stop() - d1 = self.uploader.stop() - return defer.DeferredList(list( - DeferredContext(d).addErrback(write_failure).result - for d in [d0, d1] - )) - - -_NICKNAME = Field.for_types( - u"nickname", - [unicode, bytes], - u"A Magic-Folder participant nickname.", -) - -_DIRECTION = Field.for_types( - u"direction", - [unicode], - u"A synchronization direction: uploader or downloader.", - eliotutil.validateSetMembership({u"uploader", u"downloader"}), -) - -PROCESSING_LOOP = ActionType( - u"magic-folder:processing-loop", - [_NICKNAME, _DIRECTION], - [], - u"A Magic-Folder is processing uploads or downloads.", -) - -ITERATION = ActionType( - u"magic-folder:iteration", - [_NICKNAME, _DIRECTION], - [], - u"A step towards synchronization in one direction.", -) - -_COUNT = Field.for_types( - u"count", - [int, long], - u"The number of items in the processing queue.", -) - -PROCESS_QUEUE = ActionType( - u"magic-folder:process-queue", - [_COUNT], - [], - u"A Magic-Folder is working through an item queue.", -) - -SCAN_REMOTE_COLLECTIVE = ActionType( - u"magic-folder:scan-remote-collective", - [], - [], - u"The remote collective is being scanned for peer DMDs.", -) - -_DMDS = Field( - u"dmds", - # The children of the collective directory are the participant DMDs. The - # keys in this dict give us the aliases of the participants. - lambda collective_directory_listing: collective_directory_listing.keys(), - u"The (D)istributed (M)utable (D)irectories belonging to each participant are being scanned for changes.", -) - -COLLECTIVE_SCAN = MessageType( - u"magic-folder:downloader:get-latest-file:collective-scan", - [_DMDS], - u"Participants in the collective are being scanned.", -) - - -SCAN_REMOTE_DMD = ActionType( - u"magic-folder:scan-remote-dmd", - [_NICKNAME], - [], - u"A peer DMD is being scanned for changes.", -) - -REMOTE_VERSION = Field.for_types( - u"remote_version", - [int, long], - u"The version of a path found in a peer DMD.", -) - -REMOTE_URI = Field.for_types( - u"remote_uri", - [bytes], - u"The filecap of a path found in a peer DMD.", -) - -REMOTE_DMD_ENTRY = MessageType( - u"magic-folder:remote-dmd-entry", - [eliotutil.RELPATH, magicfolderdb.PATHENTRY, REMOTE_VERSION, REMOTE_URI], - u"A single entry found by scanning a peer DMD.", -) - -ADD_TO_DOWNLOAD_QUEUE = MessageType( - u"magic-folder:add-to-download-queue", - [eliotutil.RELPATH], - u"An entry was found to be changed and is being queued for download.", -) - -MAGIC_FOLDER_STOP = ActionType( - u"magic-folder:stop", - [_NICKNAME], - [], - u"A Magic-Folder is being stopped.", -) - -MAYBE_UPLOAD = MessageType( - u"magic-folder:maybe-upload", - [eliotutil.RELPATH], - u"A decision is being made about whether to upload a file.", -) - -PENDING = Field( - u"pending", - lambda s: list(s), - u"The paths which are pending processing.", - eliotutil.validateInstanceOf(set), -) - -REMOVE_FROM_PENDING = ActionType( - u"magic-folder:remove-from-pending", - [eliotutil.RELPATH, PENDING], - [], - u"An item being processed is being removed from the pending set.", -) - -PATH = Field( - u"path", - lambda fp: fp.asTextMode().path, - u"A local filesystem path.", - eliotutil.validateInstanceOf(FilePath), -) - -NOTIFIED_OBJECT_DISAPPEARED = MessageType( - u"magic-folder:notified-object-disappeared", - [PATH], - u"A path which generated a notification was not found on the filesystem. This is normal.", -) - -PROPAGATE_DIRECTORY_DELETION = ActionType( - u"magic-folder:propagate-directory-deletion", - [], - [], - u"Children of a deleted directory are being queued for upload processing.", -) - -NO_DATABASE_ENTRY = MessageType( - u"magic-folder:no-database-entry", - [], - u"There is no local database entry for a particular relative path in the magic folder.", -) - -NOT_UPLOADING = MessageType( - u"magic-folder:not-uploading", - [], - u"An item being processed is not going to be uploaded.", -) - -SYMLINK = MessageType( - u"magic-folder:symlink", - [PATH], - u"An item being processed was a symlink and is being skipped", -) - -CREATED_DIRECTORY = Field.for_types( - u"created_directory", - [unicode], - u"The relative path of a newly created directory in a magic-folder.", -) - -PROCESS_DIRECTORY = ActionType( - u"magic-folder:process-directory", - [], - [CREATED_DIRECTORY], - u"An item being processed was a directory.", -) - -DIRECTORY_PATHENTRY = MessageType( - u"magic-folder:directory-dbentry", - [magicfolderdb.PATHENTRY], - u"Local database state relating to an item possibly being uploaded.", -) - -NOT_NEW_DIRECTORY = MessageType( - u"magic-folder:not-new-directory", - [], - u"A directory item being processed was found to not be new.", -) - -NOT_NEW_FILE = MessageType( - u"magic-folder:not-new-file", - [], - u"A file item being processed was found to not be new (or changed).", -) - -SPECIAL_FILE = MessageType( - u"magic-folder:special-file", - [], - u"An item being processed was found to be of a special type which is not supported.", -) - -_COUNTER_NAME = Field.for_types( - u"counter_name", - # Should really only be unicode - [unicode, bytes], - u"The name of a counter.", -) - -_DELTA = Field.for_types( - u"delta", - [int, long], - u"An amount of a specific change in a counter.", -) - -_VALUE = Field.for_types( - u"value", - [int, long], - u"The new value of a counter after a change.", -) - -COUNT_CHANGED = MessageType( - u"magic-folder:count", - [_COUNTER_NAME, _DELTA, _VALUE], - u"The value of a counter has changed.", -) - -START_MONITORING = ActionType( - u"magic-folder:start-monitoring", - [_NICKNAME, _DIRECTION], - [], - u"Uploader is beginning to monitor the filesystem for uploadable changes.", -) - -STOP_MONITORING = ActionType( - u"magic-folder:stop-monitoring", - [_NICKNAME, _DIRECTION], - [], - u"Uploader is terminating filesystem monitoring operation.", -) - -START_UPLOADING = ActionType( - u"magic-folder:start-uploading", - [_NICKNAME, _DIRECTION], - [], - u"Uploader is performing startup-time inspection of known files.", -) - -_IGNORED = Field.for_types( - u"ignored", - [bool], - u"A file proposed for queueing for processing is instead being ignored by policy.", -) - -_ALREADY_PENDING = Field.for_types( - u"already_pending", - [bool], - u"A file proposed for queueing for processing is already in the queue.", -) - -_SIZE = Field.for_types( - u"size", - [int, long, type(None)], - u"The size of a file accepted into the processing queue.", -) - -ADD_PENDING = ActionType( - u"magic-folder:add-pending", - [eliotutil.RELPATH], - [_IGNORED, _ALREADY_PENDING, _SIZE], - u"Uploader is adding a path to the processing queue.", -) - -FULL_SCAN = ActionType( - u"magic-folder:full-scan", - [_NICKNAME, _DIRECTION], - [], - u"A complete brute-force scan of the local directory is being performed.", -) - -SCAN = ActionType( - u"magic-folder:scan", - [eliotutil.RELPATH], - [], - u"A brute-force scan of a subset of the local directory is being performed.", -) - -NOTIFIED = ActionType( - u"magic-folder:notified", - [PATH, _NICKNAME, _DIRECTION], - [], - u"Magic-Folder received a notification of a local filesystem change for a certain path.", -) - -_NON_DIR_CREATED = Field.for_types( - u"non_dir_created", - [bool], - u"A creation event was for a non-directory and requires no further inspection.", -) - - -REACT_TO_INOTIFY = ActionType( - u"magic-folder:react-to-inotify", - [eliotutil.INOTIFY_EVENTS], - [_IGNORED, _NON_DIR_CREATED, _ALREADY_PENDING], - u"Magic-Folder is processing a notification from inotify(7) (or a clone) about a filesystem event.", -) - -_ABSPATH = Field.for_types( - u"abspath", - [unicode], - u"The absolute path of a file being written in a local directory.", -) - -_IS_CONFLICT = Field.for_types( - u"is_conflict", - [bool], - u"An indication of whether a file being written in a local directory is in a conflicted state.", -) - -_NOW = Field.for_types( - u"now", - [int, long, float], - u"The time at which a file is being written in a local directory.", -) - -_MTIME = Field.for_types( - u"mtime", - [int, long, float, type(None)], - u"A modification time to put into the metadata of a file being written in a local directory.", -) - -WRITE_DOWNLOADED_FILE = ActionType( - u"magic-folder:write-downloaded-file", - [_ABSPATH, _SIZE, _IS_CONFLICT, _NOW, _MTIME], - [], - u"A downloaded file is being written to the filesystem.", -) - -ALREADY_GONE = MessageType( - u"magic-folder:rename:already-gone", - [], - u"A deleted file could not be rewritten to a backup path because it no longer exists.", -) - -_REASON = Field( - u"reason", - lambda e: str(e), - u"An exception which may describe the form of the conflict.", - eliotutil.validateInstanceOf(Exception), -) - -OVERWRITE_BECOMES_CONFLICT = MessageType( - u"magic-folder:overwrite-becomes-conflict", - [_REASON], - u"An attempt to overwrite an existing file failed because that file is now conflicted.", -) - -_FILES = Field( - u"files", - lambda file_set: list(file_set), - u"All of the relative paths belonging to a Magic-Folder that are locally known.", -) - -ALL_FILES = MessageType( - u"magic-folder:all-files", - [_FILES], - u"A record of the rough state of the local database at the time of downloader start up.", -) - -_ITEMS = Field( - u"items", - lambda deque: list(dict(relpath=item.relpath_u, kind=item.kind) for item in deque), - u"Items in a processing queue.", -) - -ITEM_QUEUE = MessageType( - u"magic-folder:item-queue", - [_ITEMS], - u"A report of the items in the processing queue at this point.", -) - -_BATCH = Field( - u"batch", - # Just report the paths for now. Perhaps something from the values would - # also be useful, though? Consider it. - lambda batch: batch.keys(), - u"A batch of scanned items.", - eliotutil.validateInstanceOf(dict), -) - -SCAN_BATCH = MessageType( - u"magic-folder:scan-batch", - [_BATCH], - u"Items in a batch of files which were scanned from the DMD.", -) - -START_DOWNLOADING = ActionType( - u"magic-folder:start-downloading", - [_NICKNAME, _DIRECTION], - [], - u"A Magic-Folder downloader is initializing and beginning to manage downloads.", -) - -PERFORM_SCAN = ActionType( - u"magic-folder:perform-scan", - [], - [], - u"Remote storage is being scanned for changes which need to be synchronized.", -) - -_STATUS = Field.for_types( - u"status", - # Should just be unicode... - [unicode, bytes], - u"The status of an item in a processing queue.", -) - -QUEUED_ITEM_STATUS_CHANGE = MessageType( - u"magic-folder:item:status-change", - [eliotutil.RELPATH, _STATUS], - u"A queued item changed status.", -) - -_CONFLICT_REASON = Field.for_types( - u"conflict_reason", - [unicode, type(None)], - u"A human-readable explanation of why a file was in conflict.", - eliotutil.validateSetMembership({ - u"dbentry mismatch metadata", - u"dbentry newer version", - u"last_downloaded_uri mismatch", - u"file appeared", - None, - }), -) - -CHECKING_CONFLICTS = ActionType( - u"magic-folder:item:checking-conflicts", - [], - [_IS_CONFLICT, _CONFLICT_REASON], - u"A potential download item is being checked to determine if it is in a conflicted state.", -) - -REMOTE_DIRECTORY_CREATED = MessageType( - u"magic-folder:remote-directory-created", - [], - u"The downloader found a new directory in the DMD.", -) - -REMOTE_DIRECTORY_DELETED = MessageType( - u"magic-folder:remote-directory-deleted", - [], - u"The downloader found a directory has been deleted from the DMD.", -) - -class QueueMixin(HookMixin): - """ - A parent class for Uploader and Downloader that handles putting - IQueuedItem instances into a work queue and processing - them. Tracks some history of recent items processed (for the - "status" API). - - Subclasses implement _scan_delay, _perform_scan and _process - - :ivar unicode _name: Either "uploader" or "downloader". - - :ivar _deque: IQueuedItem instances to process - - :ivar _process_history: the last 20 items we processed - - :ivar _in_progress: current batch of items which are currently - being processed; chunks of work are removed from _deque and - worked on. As each finishes, it is added to _process_history - (with oldest items falling off the end). - """ - - def __init__(self, client, local_path_u, db, name, clock): - self._client = client - self._local_path_u = local_path_u - self._local_filepath = to_filepath(local_path_u) - self._db = db - self._name = name - self._clock = clock - self._log_fields = dict( - nickname=self._client.nickname, - direction=self._name, - ) - self._hooks = { - 'processed': None, - 'started': None, - 'iteration': None, - 'inotify': None, - 'item_processed': None, - } - self.started_d = self.set_hook('started') - - # we should have gotten nice errors already while loading the - # config, but just to be safe: - assert self._local_filepath.exists() - assert self._local_filepath.isdir() - - self._deque = deque() - # do we also want to bound on "maximum age"? - self._process_history = deque(maxlen=20) - self._in_progress = [] - - def get_status(self): - """ - Returns an iterable of instances that implement IQueuedItem - """ - for item in self._deque: - yield item - for item in self._in_progress: - yield item - for item in self._process_history: - yield item - - def _get_filepath(self, relpath_u): - return extend_filepath(self._local_filepath, relpath_u.split(u"/")) - - def stop(self): - """ - Don't process queued items anymore. - - :return Deferred: A ``Deferred`` that fires when processing has - completely stopped. - """ - d = self._processing - self._processing_loop.stop() - self._processing = None - self._processing_loop = None - return d - - def _begin_processing(self): - """ - Start a loop that looks for work to do and then does it. - """ - action = PROCESSING_LOOP(**self._log_fields) - - # Note that we don't put the processing iterations into the logging - # action because we expect this loop to run for the whole lifetime of - # the process. The tooling for dealing with incomplete action trees - # is still somewhat lacking. Putting the iteractions into the overall - # loop action would hamper reading those logs for now. - self._processing_loop = task.LoopingCall(self._processing_iteration) - self._processing_loop.clock = self._clock - self._processing = self._processing_loop.start(self._scan_delay(), now=True) - - with action.context(): - # We do make sure errors appear in the loop action though. - d = DeferredContext(self._processing) - d.addActionFinish() - - def _processing_iteration(self): - """ - One iteration runs self._process_deque which calls _perform_scan() and - then completely drains the _deque (processing each item). - """ - action = ITERATION(**self._log_fields) - with action.context(): - d = DeferredContext(defer.Deferred()) - - # During startup we scanned the collective for items to download. - # If we found work to do, we do not need to perform another scan - # here. More importantly, the logic for determining which items - # to download is *not correct* in the case where two scans are - # performed with no intermediate emptying of the work queue. - # Therefore, skip the scan any time there is queued work. The - # only time we expect there to be any, though, is on the first - # time through this loop. - if not self._deque: - # adds items to our deque - d.addCallback(lambda ignored: self._perform_scan()) - - # process anything in our queue - d.addCallback(lambda ignored: self._process_deque()) - - # Let the tests know we've made it this far. - d.addCallback(lambda ignored: self._call_hook(None, 'iteration')) - - # Get it out of the Eliot context - result = d.addActionFinish() - - # Kick it off - result.callback(None) - - # Give it back to LoopingCall so it can wait on us. - return result - - def _scan_delay(self): - raise NotImplementedError - - def _perform_scan(self): - return - - @eliotutil.inline_callbacks - def _process_deque(self): - # process everything currently in the queue. we're turning it - # into a list so that if any new items get added while we're - # processing, they'll not run until next time) - to_process = list(self._deque) - self._deque.clear() - self._count('objects_queued', -len(to_process)) - - # we want to include all these in the next status request, so - # we must put them 'somewhere' before the next yield (and it's - # not in _process_history because that gets trimmed and we - # don't want anything to disappear until after it is - # completed) - self._in_progress.extend(to_process) - - with PROCESS_QUEUE(count=len(to_process)): - for item in to_process: - self._process_history.appendleft(item) - self._in_progress.remove(item) - try: - proc = yield self._process(item) - if not proc: - self._process_history.remove(item) - self._call_hook(item, 'item_processed') - except: - write_traceback() - item.set_status('failed', self._clock.seconds()) - proc = Failure() - - self._call_hook(proc, 'processed') - - def _get_relpath(self, filepath): - segments = unicode_segments_from(filepath, self._local_filepath) - return u"/".join(segments) - - def _count(self, counter_name, delta=1): - ctr = 'magic_folder.%s.%s' % (self._name, counter_name) - self._client.stats_provider.count(ctr, delta) - COUNT_CHANGED.log( - counter_name=counter_name, - delta=delta, - value=self._client.stats_provider.counters[ctr], - ) - -# this isn't in interfaces.py because it's very specific to QueueMixin -class IQueuedItem(Interface): - relpath_u = Attribute("The path this item represents") - progress = Attribute("A PercentProgress instance") - - def set_status(self, status, current_time=None): - """ - """ - - def status_time(self, state): - """ - Get the time of particular state change, or None - """ - - def status_history(self): - """ - All status changes, sorted latest -> oldest - """ - - -@implementer(IQueuedItem) -class QueuedItem(object): - kind = None - - def __init__(self, relpath_u, progress, size): - self.relpath_u = relpath_u - self.progress = progress - self._status_history = dict() - self.size = size - - def set_status(self, status, current_time=None): - if current_time is None: - current_time = time.time() - self._status_history[status] = current_time - QUEUED_ITEM_STATUS_CHANGE.log( - relpath=self.relpath_u, - status=status, - ) - - def status_time(self, state): - """ - Returns None if there's no status-update for 'state', else returns - the timestamp when that state was reached. - """ - return self._status_history.get(state, None) - - def status_history(self): - """ - Returns a list of 2-tuples of (state, timestamp) sorted by timestamp - """ - hist = self._status_history.items() - hist.sort(lambda a, b: cmp(a[1], b[1])) - return hist - - def __eq__(self, other): - return ( - other.relpath_u == self.relpath_u, - other.status_history() == self.status_history(), - ) - - -class UploadItem(QueuedItem): - """ - Represents a single item the _deque of the Uploader - """ - kind = u"upload" - - -_ITEM = Field( - u"item", - lambda i: { - u"relpath": i.relpath_u, - u"size": i.size, - }, - u"An item to be uploaded or downloaded.", - eliotutil.validateInstanceOf(QueuedItem), -) - -PROCESS_ITEM = ActionType( - u"magic-folder:process-item", - [_ITEM], - [], - u"A path which was found wanting of an update is receiving an update.", -) - -DOWNLOAD_BEST_VERSION = ActionType( - u"magic-folder:download-best-version", - [], - [], - u"The content of a file in the Magic Folder is being downloaded.", -) - -class Uploader(QueueMixin): - - def __init__(self, client, local_path_u, db, upload_dirnode, pending_delay, clock): - QueueMixin.__init__(self, client, local_path_u, db, u'uploader', clock) - - self.is_ready = False - - if not IDirectoryNode.providedBy(upload_dirnode): - raise AssertionError("'upload_dircap' does not refer to a directory") - if upload_dirnode.is_unknown() or upload_dirnode.is_readonly(): - raise AssertionError("'upload_dircap' is not a writecap to a directory") - - self._upload_dirnode = upload_dirnode - self._inotify = get_inotify_module() - self._notifier = self._inotify.INotify() - - self._pending = set() # of unicode relpaths - self._pending_delay = pending_delay - self._periodic_full_scan_duration = 10 * 60 # perform a full scan every 10 minutes - self._periodic_callid = None - - if hasattr(self._notifier, 'set_pending_delay'): - self._notifier.set_pending_delay(pending_delay) - - # TODO: what about IN_MOVE_SELF and IN_UNMOUNT? - # - self.mask = ( self._inotify.IN_CREATE - | self._inotify.IN_CLOSE_WRITE - | self._inotify.IN_MOVED_TO - | self._inotify.IN_MOVED_FROM - | self._inotify.IN_DELETE - | self._inotify.IN_ONLYDIR - | IN_EXCL_UNLINK - ) - - def _add_watch(self, filepath): - self._notifier.watch( - filepath, - mask=self.mask, - callbacks=[self._notify], - recursive=True, - ) - - def start_monitoring(self): - action = START_MONITORING(**self._log_fields) - with action.context(): - d = DeferredContext(defer.succeed(None)) - - d.addCallback(lambda ign: self._add_watch(self._local_filepath)) - d.addCallback(lambda ign: self._notifier.startReading()) - d.addCallback(lambda ign: self._count('dirs_monitored')) - d.addBoth(self._call_hook, 'started') - return d.addActionFinish() - - def stop(self): - action = STOP_MONITORING(**self._log_fields) - with action.context(): - self._notifier.stopReading() - self._count('dirs_monitored', -1) - if self._periodic_callid: - try: - self._periodic_callid.cancel() - except AlreadyCancelled: - pass - - if hasattr(self._notifier, 'wait_until_stopped'): - d = DeferredContext(self._notifier.wait_until_stopped()) - else: - d = DeferredContext(defer.succeed(None)) - - d.addCallback(lambda ignored: QueueMixin.stop(self)) - return d.addActionFinish() - - def start_uploading(self): - action = START_UPLOADING(**self._log_fields) - with action: - self.is_ready = True - - all_relpaths = self._db.get_all_relpaths() - - for relpath_u in all_relpaths: - self._add_pending(relpath_u) - - self._full_scan() - self._begin_processing() - - def _scan_delay(self): - return self._pending_delay - - def _full_scan(self): - with FULL_SCAN(**self._log_fields): - self._periodic_callid = self._clock.callLater(self._periodic_full_scan_duration, self._full_scan) - self._scan(u"") - - def _add_pending(self, relpath_u): - with ADD_PENDING(relpath=relpath_u) as action: - if magicpath.should_ignore_file(relpath_u): - action.add_success_fields(ignored=True, already_pending=False, size=None) - return - if self.is_pending(relpath_u): - action.add_success_fields(ignored=False, already_pending=True, size=None) - return - - self._pending.add(relpath_u) - fp = self._get_filepath(relpath_u) - pathinfo = get_pathinfo(unicode_from_filepath(fp)) - progress = PercentProgress() - action.add_success_fields(ignored=False, already_pending=False, size=pathinfo.size) - item = UploadItem(relpath_u, progress, pathinfo.size) - item.set_status('queued', self._clock.seconds()) - self._deque.append(item) - self._count('objects_queued') - - def _scan(self, reldir_u): - # Scan a directory by (synchronously) adding the paths of all its children to self._pending. - # Note that this doesn't add them to the deque -- that will - with SCAN(relpath=reldir_u): - fp = self._get_filepath(reldir_u) - try: - children = listdir_filepath(fp) - except EnvironmentError: - raise Exception("WARNING: magic folder: permission denied on directory %s" - % quote_filepath(fp)) - except FilenameEncodingError: - raise Exception("WARNING: magic folder: could not list directory %s due to a filename encoding error" - % quote_filepath(fp)) - - for child in children: - _assert(isinstance(child, unicode), child=child) - self._add_pending("%s/%s" % (reldir_u, child) if reldir_u != u"" else child) - - def is_pending(self, relpath_u): - return relpath_u in self._pending - - def _notify(self, opaque, path, events_mask): - with NOTIFIED(path=path, **self._log_fields): - try: - return self._real_notify(opaque, path, events_mask) - except Exception: - write_traceback() - - def _real_notify(self, opaque, path, events_mask): - action = REACT_TO_INOTIFY( - # We could think about logging opaque here but ... it's opaque. - # All can do is id() or repr() it and neither of those actually - # produces very illuminating results. We drop opaque on the - # floor, anyway. - inotify_events=events_mask, - ) - success_fields = dict(non_dir_created=False, already_pending=False, ignored=False) - - with action: - relpath_u = self._get_relpath(path) - - # We filter out IN_CREATE events not associated with a directory. - # Acting on IN_CREATE for files could cause us to read and upload - # a possibly-incomplete file before the application has closed it. - # There should always be an IN_CLOSE_WRITE after an IN_CREATE, I think. - # It isn't possible to avoid watching for IN_CREATE at all, because - # it is the only event notified for a directory creation. - - if ((events_mask & self._inotify.IN_CREATE) != 0 and - (events_mask & self._inotify.IN_ISDIR) == 0): - success_fields[u"non_dir_created"] = True - elif relpath_u in self._pending: - success_fields[u"already_pending"] = True - elif magicpath.should_ignore_file(relpath_u): - success_fields[u"ignored"] = True - else: - self._add_pending(relpath_u) - - # Always fire the inotify hook. If an accident of timing causes a - # second inotify event for a particular path before the first has - # been processed, the expectation is still that any code that was - # waiting for the second inotify event should be notified. - self._call_hook(path, 'inotify') - action.add_success_fields(**success_fields) - - def _process(self, item): - """ - Possibly upload a single QueuedItem. If this returns False, the item is - removed from _process_history. - """ - # Uploader - with PROCESS_ITEM(item=item).context(): - relpath_u = item.relpath_u - precondition(isinstance(relpath_u, unicode), relpath_u) - precondition(not relpath_u.endswith(u'/'), relpath_u) - encoded_path_u = magicpath.path2magic(relpath_u) - - d = DeferredContext(defer.succeed(False)) - if relpath_u is None: - item.set_status('invalid_path', self._clock.seconds()) - return d.addActionFinish() - item.set_status('started', self._clock.seconds()) - - try: - # Take this item out of the pending set before we do any - # I/O-based processing related to it. If a further change - # takes place after we remove it from this set, we want it to - # end up in the set again. If we haven't gotten around to - # doing the I/O-based processing yet then the worst that will - # happen is we'll do a little redundant processing. - # - # If we did it the other way around, the sequence of events - # might be something like: we do some I/O, someone else does - # some I/O, a notification gets discarded because the path is - # still in the pending set, _then_ we remove it from the - # pending set. In such a circumstance, we've missed some I/O - # that we should have responded to. - with REMOVE_FROM_PENDING(relpath=relpath_u, pending=self._pending): - self._pending.remove(relpath_u) - except KeyError: - pass - - fp = self._get_filepath(relpath_u) - pathinfo = get_pathinfo(unicode_from_filepath(fp)) - - db_entry_is_dir = False - db_entry = self._db.get_db_entry(relpath_u) - if db_entry is None: - # Maybe it was a directory! - db_entry = self._db.get_db_entry(relpath_u + u"/") - if db_entry is None: - NO_DATABASE_ENTRY.log() - else: - db_entry_is_dir = True - - def _maybe_upload(ign, now=None): - MAYBE_UPLOAD.log(relpath=relpath_u) - if now is None: - now = time.time() - - if not pathinfo.exists: - # FIXME merge this with the 'isfile' case. - NOTIFIED_OBJECT_DISAPPEARED.log(path=fp) - self._count('objects_disappeared') - - if db_entry is None: - # If it exists neither on the filesystem nor in the - # database, it's neither a creation nor a deletion and - # there's nothing more to do. - return False - - if pathinfo.isdir or db_entry_is_dir: - with PROPAGATE_DIRECTORY_DELETION(): - for localpath in self._db.get_direct_children(relpath_u): - self._add_pending(localpath.relpath_u) - - last_downloaded_timestamp = now # is this correct? - - if is_new_file(pathinfo, db_entry): - new_version = db_entry.version + 1 - else: - NOT_UPLOADING.log() - self._count('objects_not_uploaded') - return False - - # look out! there's another place we set a "metadata" - # object like this (for new, not deleted files) - metadata = { - 'version': new_version, - 'deleted': True, - 'last_downloaded_timestamp': last_downloaded_timestamp, - 'user_mtime': pathinfo.ctime_ns / 1000000000.0, # why are we using ns in PathInfo?? - } - - # from the Fire Dragons part of the spec: - # Later, in response to a local filesystem change at a given path, the - # Magic Folder client reads the last-downloaded record associated with - # that path (if any) from the database and then uploads the current - # file. When it links the uploaded file into its client DMD, it - # includes the ``last_downloaded_uri`` field in the metadata of the - # directory entry, overwriting any existing field of that name. If - # there was no last-downloaded record associated with the path, this - # field is omitted. - # Note that ``last_downloaded_uri`` field does *not* record the URI of - # the uploaded file (which would be redundant); it records the URI of - # the last download before the local change that caused the upload. - # The field will be absent if the file has never been downloaded by - # this client (i.e. if it was created on this client and no change - # by any other client has been detected). - - # XXX currently not actually true: it will record the - # LAST THING we wrote to (or saw on) disk (not - # necessarily downloaded?) - - if db_entry.last_downloaded_uri is not None: - metadata['last_downloaded_uri'] = db_entry.last_downloaded_uri - if db_entry.last_uploaded_uri is not None: - metadata['last_uploaded_uri'] = db_entry.last_uploaded_uri - - if db_entry_is_dir: - real_encoded_path_u = encoded_path_u + magicpath.path2magic(u"/") - real_relpath_u = relpath_u + u"/" - else: - real_encoded_path_u = encoded_path_u - real_relpath_u = relpath_u - - empty_uploadable = Data("", self._client.convergence) - d2 = DeferredContext(self._upload_dirnode.add_file( - real_encoded_path_u, - empty_uploadable, - metadata=metadata, - overwrite=True, - progress=item.progress, - )) - - def _add_db_entry(filenode): - filecap = filenode.get_uri() - # if we're uploading a file, we want to set - # last_downloaded_uri to the filecap so that we don't - # immediately re-download it when we start up next - last_downloaded_uri = metadata.get('last_downloaded_uri', filecap) - self._db.did_upload_version( - real_relpath_u, - new_version, - filecap, - last_downloaded_uri, - last_downloaded_timestamp, - pathinfo, - ) - self._count('files_uploaded') - d2.addCallback(_add_db_entry) - d2.addCallback(lambda ign: True) - return d2.result - elif pathinfo.islink: - SYMLINK.log(path=fp) - return False - elif pathinfo.isdir: - if not getattr(self._notifier, 'recursive_includes_new_subdirectories', False): - self._add_watch(fp) - - DIRECTORY_PATHENTRY.log(pathentry=db_entry) - if not is_new_file(pathinfo, db_entry): - NOT_NEW_DIRECTORY.log() - return False - - uploadable = Data("", self._client.convergence) - with PROCESS_DIRECTORY().context() as action: - upload_d = DeferredContext(self._upload_dirnode.add_file( - encoded_path_u + magicpath.path2magic(u"/"), - uploadable, - metadata={"version": 0}, - overwrite=True, - progress=item.progress, - )) - def _dir_succeeded(dirnode): - action.add_success_fields(created_directory=relpath_u) - self._count('directories_created') - self._db.did_upload_version( - relpath_u + u"/", - version=0, - last_uploaded_uri=dirnode.get_uri(), - last_downloaded_uri=None, - last_downloaded_timestamp=now, - pathinfo=pathinfo, - ) - upload_d.addCallback(_dir_succeeded) - upload_d.addCallback(lambda ign: self._scan(relpath_u)) - upload_d.addCallback(lambda ign: True) - return upload_d.addActionFinish() - elif pathinfo.isfile: - last_downloaded_timestamp = now - - if db_entry is None: - new_version = 0 - elif is_new_file(pathinfo, db_entry): - new_version = db_entry.version + 1 - else: - NOT_NEW_FILE.log() - self._count('objects_not_uploaded') - return False - - metadata = { - 'version': new_version, - 'last_downloaded_timestamp': last_downloaded_timestamp, - 'user_mtime': pathinfo.mtime_ns / 1000000000.0, # why are we using ns in PathInfo?? - } - if db_entry is not None: - if db_entry.last_downloaded_uri is not None: - metadata['last_downloaded_uri'] = db_entry.last_downloaded_uri - if db_entry.last_uploaded_uri is not None: - metadata['last_uploaded_uri'] = db_entry.last_uploaded_uri - - uploadable = FileName(unicode_from_filepath(fp), self._client.convergence) - d2 = DeferredContext(self._upload_dirnode.add_file( - encoded_path_u, uploadable, - metadata=metadata, - overwrite=True, - progress=item.progress, - )) - - def _add_db_entry(filenode): - filecap = filenode.get_uri() - # if we're uploading a file, we want to set - # last_downloaded_uri to the filecap so that we don't - # immediately re-download it when we start up next - last_downloaded_uri = filecap - self._db.did_upload_version( - relpath_u, - new_version, - filecap, - last_downloaded_uri, - last_downloaded_timestamp, - pathinfo - ) - self._count('files_uploaded') - return True - d2.addCallback(_add_db_entry) - return d2.result - else: - SPECIAL_FILE.log() - return False - - d.addCallback(_maybe_upload) - - def _succeeded(res): - if res: - self._count('objects_succeeded') - # TODO: maybe we want the status to be 'ignored' if res is False - item.set_status('success', self._clock.seconds()) - return res - def _failed(f): - self._count('objects_failed') - item.set_status('failure', self._clock.seconds()) - return f - d.addCallbacks(_succeeded, _failed) - return d.addActionFinish() - - def _get_metadata(self, encoded_path_u): - try: - d = self._upload_dirnode.get_metadata_for(encoded_path_u) - except KeyError: - return Failure() - return d - - def _get_filenode(self, encoded_path_u): - try: - d = self._upload_dirnode.get(encoded_path_u) - except KeyError: - return Failure() - return d - - -class WriteFileMixin(object): - FUDGE_SECONDS = 10.0 - - def _get_conflicted_filename(self, abspath_u): - return abspath_u + u".conflict" - - def _write_downloaded_file(self, local_path_u, abspath_u, file_contents, - is_conflict=False, now=None, mtime=None): - if now is None: - now = time.time() - action = WRITE_DOWNLOADED_FILE( - abspath=abspath_u, - size=len(file_contents), - is_conflict=is_conflict, - now=now, - mtime=mtime, - ) - with action: - return self._write_downloaded_file_logged( - local_path_u, - abspath_u, - file_contents, - is_conflict, - now, - mtime, - ) - - def _write_downloaded_file_logged(self, local_path_u, abspath_u, - file_contents, is_conflict, now, mtime): - # 1. Write a temporary file, say .foo.tmp. - # 2. is_conflict determines whether this is an overwrite or a conflict. - # 3. Set the mtime of the replacement file to be T seconds before the - # current local time, or mtime whichever is oldest - # 4. Perform a file replacement with backup filename foo.backup, - # replaced file foo, and replacement file .foo.tmp. If any step of - # this operation fails, reclassify as a conflict and stop. - # - # Returns the path of the destination file. - precondition_abspath(abspath_u) - replacement_path_u = abspath_u + u".tmp" # FIXME more unique - - initial_path_u = os.path.dirname(abspath_u) - fileutil.make_dirs_with_absolute_mode(local_path_u, initial_path_u, (~ self._umask) & 0o777) - fileutil.write(replacement_path_u, file_contents) - os.chmod(replacement_path_u, (~ self._umask) & 0o666) - - # FUDGE_SECONDS is used to determine if another process has - # written to the same file concurrently. This is described in - # the Earth Dragon section of our design document ("T" in the - # spec is FUDGE_SECONDS here): - # docs/proposed/magic-folder/remote-to-local-sync.rst - fudge_time = now - self.FUDGE_SECONDS - modified_time = min(fudge_time, mtime) if mtime else fudge_time - os.utime(replacement_path_u, (now, modified_time)) - if is_conflict: - return self._rename_conflicted_file(abspath_u, replacement_path_u) - else: - try: - fileutil.replace_file(abspath_u, replacement_path_u) - return abspath_u - except fileutil.ConflictError as e: - OVERWRITE_BECOMES_CONFLICT.log(reason=e) - return self._rename_conflicted_file(abspath_u, replacement_path_u) - - @log_call( - action_type=u"magic-folder:rename-conflicted", - include_args=["abspath_u", "replacement_path_u"], - ) - def _rename_conflicted_file(self, abspath_u, replacement_path_u): - conflict_path_u = self._get_conflicted_filename(abspath_u) - fileutil.rename_no_overwrite(replacement_path_u, conflict_path_u) - return conflict_path_u - - @log_call( - action_type=u"magic-folder:rename-deleted", - include_args=["abspath_u"], - ) - def _rename_deleted_file(self, abspath_u): - try: - fileutil.rename_no_overwrite(abspath_u, abspath_u + u'.backup') - except OSError: - ALREADY_GONE.log() - return abspath_u - - -def _is_empty_filecap(client, cap): - """ - Internal helper. - - :param cap: a capability URI - - :returns: True if "cap" represents an empty file - """ - node = client.create_node_from_uri( - None, - cap.encode('ascii'), - ) - return (not node.get_size()) - - -class DownloadItem(QueuedItem): - """ - Represents a single item in the _deque of the Downloader - """ - kind = u"download" - - def __init__(self, relpath_u, progress, filenode, metadata, size): - super(DownloadItem, self).__init__(relpath_u, progress, size) - self.file_node = filenode - self.metadata = metadata - - -class Downloader(QueueMixin, WriteFileMixin): - - def __init__(self, client, local_path_u, db, collective_dirnode, - upload_readonly_dircap, clock, is_upload_pending, umask, - status_reporter, poll_interval=60): - QueueMixin.__init__(self, client, local_path_u, db, u'downloader', clock) - - if not IDirectoryNode.providedBy(collective_dirnode): - raise AssertionError("'collective_dircap' does not refer to a directory") - if collective_dirnode.is_unknown() or not collective_dirnode.is_readonly(): - raise AssertionError("'collective_dircap' is not a readonly cap to a directory") - - self._collective_dirnode = collective_dirnode - self._upload_readonly_dircap = upload_readonly_dircap - self._is_upload_pending = is_upload_pending - self._umask = umask - self._status_reporter = status_reporter - self._poll_interval = poll_interval - - @eliotutil.inline_callbacks - def start_downloading(self): - action = START_DOWNLOADING(**self._log_fields) - with action: - ALL_FILES.log(files=self._db.get_all_relpaths()) - - while True: - try: - yield self._scan_remote_collective(scan_self=True) - # The integration tests watch for this log message (in the - # Twisted log) to decide when it is safe to proceed. - # Clearly, we need better programmatic interrogation of - # magic-folder state. - twmsg("Completed initial Magic Folder scan successfully ({})".format(self)) - self._begin_processing() - return - except Exception: - self._status_reporter( - False, "Initial scan has failed", - "Last tried at %s" % self.nice_current_time(), - ) - write_traceback() - yield task.deferLater(self._clock, self._scan_delay(), lambda: None) - - def nice_current_time(self): - return format_time(datetime.fromtimestamp(self._clock.seconds()).timetuple()) - - def _should_download(self, relpath_u, remote_version, remote_uri): - """ - _should_download returns a bool indicating whether or not a remote object should be downloaded. - We check the remote metadata version against our magic-folder db version number; - latest version wins. - """ - if magicpath.should_ignore_file(relpath_u): - return False - db_entry = self._db.get_db_entry(relpath_u) - if db_entry is None: - return True - if db_entry.version < remote_version: - return True - if db_entry.last_downloaded_uri is None and _is_empty_filecap(self._client, remote_uri): - pass - elif db_entry.last_downloaded_uri != remote_uri: - return True - return False - - def _get_local_latest(self, relpath_u): - """ - _get_local_latest takes a unicode path string checks to see if this file object - exists in our magic-folder db; if not then return None - else check for an entry in our magic-folder db and return it. - """ - if not self._get_filepath(relpath_u).exists(): - return None - return self._db.get_db_entry(relpath_u) - - def _get_collective_latest_file(self, filename): - """ - _get_collective_latest_file takes a file path pointing to a file managed by - magic-folder and returns a deferred that fires with the two tuple containing a - file node and metadata for the latest version of the file located in the - magic-folder collective directory. - """ - action = start_action( - action_type=u"magic-folder:downloader:get-latest-file", - name=filename, - ) - with action.context(): - collective_dirmap_d = DeferredContext(self._collective_dirnode.list()) - def scan_collective(result): - COLLECTIVE_SCAN.log(dmds=result) - list_of_deferreds = [] - for dir_name in result: - # XXX make sure it's a directory - d = DeferredContext(defer.succeed(None)) - d.addCallback(lambda x, dir_name=dir_name: result[dir_name][0].get_child_and_metadata(filename)) - list_of_deferreds.append(d) - deferList = defer.DeferredList(list_of_deferreds, consumeErrors=True) - return deferList - collective_dirmap_d.addCallback(scan_collective) - def highest_version(deferredList): - max_version = 0 - metadata = None - node = None - for success, result in deferredList: - if success: - Message.log( - message_type=u"magic-folder:downloader:get-latest-file:version", - version=result[1]['version'], - ) - if node is None or result[1]['version'] > max_version: - node, metadata = result - max_version = result[1]['version'] - else: - Message.log( - message_type="magic-folder:downloader:get-latest-file:failed", - ) - return node, metadata - collective_dirmap_d.addCallback(highest_version) - return collective_dirmap_d.addActionFinish() - - def _scan_remote_dmd(self, nickname, dirnode, scan_batch): - """ - Read the contents of a single DMD into the given batch. - - :param unicode nickname: The nickname for the participant owning the - DMD to scan. - - :param IDirectoryNode dirnode: The node representing the chosen - participant's DMD. - - :param dict scan_batch: A dictionary into which to collect the results - of the scan. This is mutated to add the results in-place. Keys - are the unicode relative paths of contents of the DMD. Values are - a list of two-tuples. The first element of each two-tuple is the - ``IFilesystemNode`` for the content. The second element is a - ``dict`` of metadata. - - :return Deferred: A ``Deferred`` which fires when the scan is - complete. - """ - with SCAN_REMOTE_DMD(nickname=nickname).context(): - d = DeferredContext(dirnode.list()) - def scan_listing(listing_map): - for encoded_relpath_u, (file_node, metadata) in listing_map.iteritems(): - relpath_u = magicpath.magic2path(encoded_relpath_u) - local_dbentry = self._get_local_latest(relpath_u) - - # XXX FIXME this is *awefully* similar to - # _should_download code in function etc -- can we - # share? - remote_version = metadata.get('version', None) - remote_uri = file_node.get_readonly_uri() - REMOTE_DMD_ENTRY.log( - relpath=relpath_u, - pathentry=local_dbentry, - remote_version=remote_version, - remote_uri=remote_uri, - ) - - if (local_dbentry is None or remote_version is None or - local_dbentry.version < remote_version or - (local_dbentry.version == remote_version and local_dbentry.last_downloaded_uri != remote_uri)): - ADD_TO_DOWNLOAD_QUEUE.log(relpath=relpath_u) - - # The scan_batch is shared across the scan of multiple - # DMDs. It is expected the DMDs will most often be mostly - # synchronized with each other. The common case, then, is - # that there is already an entry for relpath_u. So try to - # make that the fast path: assume there is a value already - # and extend it. If there's not, we'll do an extra lookup - # to initialize it. - try: - scan_batch[relpath_u] += [(file_node, metadata)] - except KeyError: - scan_batch[relpath_u] = [(file_node, metadata)] - self._status_reporter( - True, 'Magic folder is working', - 'Last scan: %s' % self.nice_current_time(), - ) - - d.addCallback(scan_listing) - return d.addActionFinish() - - @eliotutil.log_call_deferred(SCAN_REMOTE_COLLECTIVE.action_type) - def _scan_remote_collective(self, scan_self=False): - precondition(not self._deque, "Items in _deque invalidate should_download logic") - scan_batch = {} # path -> [(filenode, metadata)] - d = DeferredContext(self._collective_dirnode.list()) - def scan_collective(dirmap): - d2 = DeferredContext(defer.succeed(None)) - for dir_name in dirmap: - (dirnode, metadata) = dirmap[dir_name] - if scan_self or dirnode.get_readonly_uri() != self._upload_readonly_dircap: - d2.addCallback(lambda ign, dir_name=dir_name, dirnode=dirnode: - self._scan_remote_dmd(dir_name, dirnode, scan_batch)) - # XXX what should we do to make this failure more visible to users? - d2.addErrback(write_traceback) - return d2.result - d.addCallback(scan_collective) - - @log_call( - action_type=u"magic-folder:filter-batch-to-deque", - include_args=[], - include_result=False, - ) - def _filter_batch_to_deque(ign): - ITEM_QUEUE.log(items=self._deque) - SCAN_BATCH.log(batch=scan_batch) - for relpath_u, versions in scan_batch.iteritems(): - file_node, metadata = max(versions, key=lambda x: x[1]['version']) - - if self._should_download(relpath_u, metadata['version'], file_node.get_readonly_uri()): - to_dl = DownloadItem( - relpath_u, - PercentProgress(file_node.get_size()), - file_node, - metadata, - file_node.get_size(), - ) - to_dl.set_status('queued', self._clock.seconds()) - self._deque.append(to_dl) - self._count("objects_queued") - else: - self._call_hook(None, 'processed', async=True) # await this maybe-Deferred?? - - d.addCallback(_filter_batch_to_deque) - return d.result - - def _scan_delay(self): - return self._poll_interval - - @eliotutil.log_call_deferred(PERFORM_SCAN.action_type) - @eliotutil.inline_callbacks - def _perform_scan(self): - try: - yield self._scan_remote_collective() - self._status_reporter( - True, 'Magic folder is working', - 'Last scan: %s' % self.nice_current_time(), - ) - except Exception as e: - write_traceback() - self._status_reporter( - False, 'Remote scan has failed: %s' % str(e), - 'Last attempted at %s' % self.nice_current_time(), - ) - - def _process(self, item): - """ - Possibly upload a single QueuedItem. If this returns False, the item is - removed from _process_history. - """ - # Downloader - now = self._clock.seconds() - - item.set_status('started', now) - fp = self._get_filepath(item.relpath_u) - abspath_u = unicode_from_filepath(fp) - conflict_path_u = self._get_conflicted_filename(abspath_u) - last_uploaded_uri = item.metadata.get('last_uploaded_uri', None) - - with PROCESS_ITEM(item=item): - d = DeferredContext(defer.succeed(False)) - - def do_update_db(written_abspath_u): - filecap = item.file_node.get_uri() - if not item.file_node.get_size(): - filecap = None # ^ is an empty file - last_downloaded_uri = filecap - last_downloaded_timestamp = now - written_pathinfo = get_pathinfo(written_abspath_u) - - if not written_pathinfo.exists and not item.metadata.get('deleted', False): - raise Exception("downloaded object %s disappeared" % quote_local_unicode_path(written_abspath_u)) - - self._db.did_upload_version( - item.relpath_u, - item.metadata['version'], - last_uploaded_uri, - last_downloaded_uri, - last_downloaded_timestamp, - written_pathinfo, - ) - self._count('objects_downloaded') - item.set_status('success', self._clock.seconds()) - return True - - def failed(f): - item.set_status('failure', self._clock.seconds()) - self._count('objects_failed') - return f - - if os.path.isfile(conflict_path_u): - def fail(res): - raise ConflictError("download failed: already conflicted: %r" % (item.relpath_u,)) - d.addCallback(fail) - else: - - # Let ``last_downloaded_uri`` be the field of that name obtained from - # the directory entry metadata for ``foo`` in Bob's DMD (this field - # may be absent). Then the algorithm is: - - # * 2a. Attempt to "stat" ``foo`` to get its *current statinfo* (size - # in bytes, ``mtime``, and ``ctime``). If Alice has no local copy - # of ``foo``, classify as an overwrite. - - current_statinfo = get_pathinfo(abspath_u) - - is_conflict = False - db_entry = self._db.get_db_entry(item.relpath_u) - dmd_last_downloaded_uri = item.metadata.get('last_downloaded_uri', None) - - # * 2b. Read the following information for the path ``foo`` from the - # local magic folder db: - # * the *last-seen statinfo*, if any (this is the size in - # bytes, ``mtime``, and ``ctime`` stored in the ``local_files`` - # table when the file was last uploaded); - # * the ``last_uploaded_uri`` field of the ``local_files`` table - # for this file, which is the URI under which the file was last - # uploaded. - - with CHECKING_CONFLICTS() as action: - conflict_reason = None - if db_entry: - # * 2c. If any of the following are true, then classify as a conflict: - # * i. there are pending notifications of changes to ``foo``; - # * ii. the last-seen statinfo is either absent (i.e. there is - # no entry in the database for this path), or different from the - # current statinfo; - - if current_statinfo.exists: - if (db_entry.mtime_ns != current_statinfo.mtime_ns or \ - db_entry.ctime_ns != current_statinfo.ctime_ns or \ - db_entry.size != current_statinfo.size): - is_conflict = True - conflict_reason = u"dbentry mismatch metadata" - - if db_entry.last_downloaded_uri is None \ - or db_entry.last_uploaded_uri is None \ - or dmd_last_downloaded_uri is None: - # we've never downloaded anything before for this - # file, but the other side might have created a new - # file "at the same time" - if db_entry.version >= item.metadata['version']: - is_conflict = True - conflict_reason = u"dbentry newer version" - elif dmd_last_downloaded_uri != db_entry.last_downloaded_uri: - is_conflict = True - conflict_reason = u"last_downloaded_uri mismatch" - - else: # no local db_entry .. but has the file appeared locally meantime? - if current_statinfo.exists: - is_conflict = True - conflict_reason = u"file appeared" - - action.add_success_fields( - is_conflict=is_conflict, - conflict_reason=conflict_reason, - ) - - if is_conflict: - self._count('objects_conflicted') - - if item.relpath_u.endswith(u"/"): - if item.metadata.get('deleted', False): - REMOTE_DIRECTORY_DELETED.log() - else: - REMOTE_DIRECTORY_CREATED.log() - d.addCallback(lambda ign: fileutil.make_dirs(abspath_u)) - d.addCallback(lambda ign: abspath_u) - else: - if item.metadata.get('deleted', False): - d.addCallback(lambda ign: self._rename_deleted_file(abspath_u)) - else: - @eliotutil.log_call_deferred(DOWNLOAD_BEST_VERSION.action_type) - def download_best_version(ignored): - d = DeferredContext(item.file_node.download_best_version(progress=item.progress)) - d.addCallback(lambda contents: self._write_downloaded_file( - self._local_path_u, abspath_u, contents, - is_conflict=is_conflict, - mtime=item.metadata.get('user_mtime', item.metadata.get('tahoe', {}).get('linkmotime')), - )) - return d.result - - d.addCallback(download_best_version) - - d.addCallback(do_update_db) - d.addErrback(failed) - - def trap_conflicts(f): - f.trap(ConflictError) - return False - d.addErrback(trap_conflicts) - return d.addActionFinish() diff --git a/src/allmydata/magicfolderdb.py b/src/allmydata/magicfolderdb.py deleted file mode 100644 index 9b09f6c95..000000000 --- a/src/allmydata/magicfolderdb.py +++ /dev/null @@ -1,204 +0,0 @@ -from __future__ import print_function - -import sys -from collections import namedtuple - -from allmydata.util.dbutil import get_db, DBError -from allmydata.util.eliotutil import ( - RELPATH, - VERSION, - LAST_UPLOADED_URI, - LAST_DOWNLOADED_URI, - LAST_DOWNLOADED_TIMESTAMP, - PATHINFO, - validateSetMembership, - validateInstanceOf, -) -from eliot import ( - Field, - ActionType, -) - -PathEntry = namedtuple('PathEntry', 'size mtime_ns ctime_ns version last_uploaded_uri ' - 'last_downloaded_uri last_downloaded_timestamp') - -PATHENTRY = Field( - u"pathentry", - lambda v: None if v is None else { - "size": v.size, - "mtime_ns": v.mtime_ns, - "ctime_ns": v.ctime_ns, - "version": v.version, - "last_uploaded_uri": v.last_uploaded_uri, - "last_downloaded_uri": v.last_downloaded_uri, - "last_downloaded_timestamp": v.last_downloaded_timestamp, - }, - u"The local database state of a file.", - validateInstanceOf((type(None), PathEntry)), -) - -_INSERT_OR_UPDATE = Field.for_types( - u"insert_or_update", - [unicode], - u"An indication of whether the record for this upload was new or an update to a previous entry.", - validateSetMembership({u"insert", u"update"}), -) - -UPDATE_ENTRY = ActionType( - u"magic-folder-db:update-entry", - [RELPATH, VERSION, LAST_UPLOADED_URI, LAST_DOWNLOADED_URI, LAST_DOWNLOADED_TIMESTAMP, PATHINFO], - [_INSERT_OR_UPDATE], - u"Record some metadata about a relative path in the magic-folder.", -) - - -# magic-folder db schema version 1 -SCHEMA_v1 = """ -CREATE TABLE version -( - version INTEGER -- contains one row, set to 1 -); - -CREATE TABLE local_files -( - path VARCHAR(1024) PRIMARY KEY, -- UTF-8 filename relative to local magic folder dir - size INTEGER, -- ST_SIZE, or NULL if the file has been deleted - mtime_ns INTEGER, -- ST_MTIME in nanoseconds - ctime_ns INTEGER, -- ST_CTIME in nanoseconds - version INTEGER, - last_uploaded_uri VARCHAR(256), -- URI:CHK:... - last_downloaded_uri VARCHAR(256), -- URI:CHK:... - last_downloaded_timestamp TIMESTAMP -); -""" - - -def get_magicfolderdb(dbfile, stderr=sys.stderr, - create_version=(SCHEMA_v1, 1), just_create=False): - # Open or create the given backupdb file. The parent directory must - # exist. - try: - (sqlite3, db) = get_db(dbfile, stderr, create_version, - just_create=just_create, dbname="magicfolderdb") - if create_version[1] in (1, 2): - return MagicFolderDB(sqlite3, db) - else: - print("invalid magicfolderdb schema version specified", file=stderr) - return None - except DBError as e: - print(e, file=stderr) - return None - -class LocalPath(object): - @classmethod - def fromrow(self, row): - p = LocalPath() - p.relpath_u = row[0] - p.entry = PathEntry(*row[1:]) - return p - - -class MagicFolderDB(object): - VERSION = 1 - - def __init__(self, sqlite_module, connection): - self.sqlite_module = sqlite_module - self.connection = connection - self.cursor = connection.cursor() - - def close(self): - self.connection.close() - - def get_db_entry(self, relpath_u): - """ - Retrieve the entry in the database for a given path, or return None - if there is no such entry. - """ - c = self.cursor - c.execute("SELECT size, mtime_ns, ctime_ns, version, last_uploaded_uri," - " last_downloaded_uri, last_downloaded_timestamp" - " FROM local_files" - " WHERE path=?", - (relpath_u,)) - row = self.cursor.fetchone() - if not row: - return None - else: - (size, mtime_ns, ctime_ns, version, last_uploaded_uri, - last_downloaded_uri, last_downloaded_timestamp) = row - return PathEntry(size=size, mtime_ns=mtime_ns, ctime_ns=ctime_ns, version=version, - last_uploaded_uri=last_uploaded_uri, - last_downloaded_uri=last_downloaded_uri, - last_downloaded_timestamp=last_downloaded_timestamp) - - def get_direct_children(self, relpath_u): - """ - Given the relative path to a directory, return ``LocalPath`` instances - representing all direct children of that directory. - """ - # It would be great to not be interpolating data into query - # statements. However, query parameters are not supported in the - # position where we need them. - sqlitesafe_relpath_u = relpath_u.replace(u"'", u"''") - statement = ( - """ - SELECT - path, size, mtime_ns, ctime_ns, version, last_uploaded_uri, - last_downloaded_uri, last_downloaded_timestamp - FROM - local_files - WHERE - -- The "_" used here ensures there is at least one character - -- after the /. This prevents matching the path itself. - path LIKE '{path}/_%' AND - - -- The "_" used here serves a similar purpose. This allows - -- matching directory children but avoids matching their - -- children. - path NOT LIKE '{path}/_%/_%' - """ - ).format(path=sqlitesafe_relpath_u) - - self.cursor.execute(statement) - rows = self.cursor.fetchall() - return list( - LocalPath.fromrow(row) - for row - in rows - ) - - def get_all_relpaths(self): - """ - Retrieve a set of all relpaths of files that have had an entry in magic folder db - (i.e. that have been downloaded at least once). - """ - self.cursor.execute("SELECT path FROM local_files") - rows = self.cursor.fetchall() - return set([r[0] for r in rows]) - - def did_upload_version(self, relpath_u, version, last_uploaded_uri, last_downloaded_uri, last_downloaded_timestamp, pathinfo): - action = UPDATE_ENTRY( - relpath=relpath_u, - version=version, - last_uploaded_uri=last_uploaded_uri, - last_downloaded_uri=last_downloaded_uri, - last_downloaded_timestamp=last_downloaded_timestamp, - pathinfo=pathinfo, - ) - with action: - try: - self.cursor.execute("INSERT INTO local_files VALUES (?,?,?,?,?,?,?,?)", - (relpath_u, pathinfo.size, pathinfo.mtime_ns, pathinfo.ctime_ns, - version, last_uploaded_uri, last_downloaded_uri, - last_downloaded_timestamp)) - action.add_success_fields(insert_or_update=u"insert") - except (self.sqlite_module.IntegrityError, self.sqlite_module.OperationalError): - self.cursor.execute("UPDATE local_files" - " SET size=?, mtime_ns=?, ctime_ns=?, version=?, last_uploaded_uri=?," - " last_downloaded_uri=?, last_downloaded_timestamp=?" - " WHERE path=?", - (pathinfo.size, pathinfo.mtime_ns, pathinfo.ctime_ns, version, - last_uploaded_uri, last_downloaded_uri, last_downloaded_timestamp, - relpath_u)) - action.add_success_fields(insert_or_update=u"update") - self.connection.commit() diff --git a/src/allmydata/magicpath.py b/src/allmydata/magicpath.py deleted file mode 100644 index 1f46330d4..000000000 --- a/src/allmydata/magicpath.py +++ /dev/null @@ -1,32 +0,0 @@ -import re -import os.path - -from allmydata.util.assertutil import precondition, _assert - -def path2magic(path): - return re.sub(u'[/@]', lambda m: {u'/': u'@_', u'@': u'@@'}[m.group(0)], path) - -def magic2path(path): - return re.sub(u'@[_@]', lambda m: {u'@_': u'/', u'@@': u'@'}[m.group(0)], path) - - -IGNORE_SUFFIXES = [u'.backup', u'.tmp', u'.conflict'] -IGNORE_PREFIXES = [u'.'] - -def should_ignore_file(path_u): - precondition(isinstance(path_u, unicode), path_u=path_u) - - for suffix in IGNORE_SUFFIXES: - if path_u.endswith(suffix): - return True - - while path_u != u"": - oldpath_u = path_u - path_u, tail_u = os.path.split(path_u) - if tail_u.startswith(u"."): - return True - if path_u == oldpath_u: - return True # the path was absolute - _assert(len(path_u) < len(oldpath_u), path_u=path_u, oldpath_u=oldpath_u) - - return False diff --git a/src/allmydata/scripts/magic_folder_cli.py b/src/allmydata/scripts/magic_folder_cli.py deleted file mode 100644 index 4165dec49..000000000 --- a/src/allmydata/scripts/magic_folder_cli.py +++ /dev/null @@ -1,610 +0,0 @@ -from __future__ import print_function - -import os -import urllib -from types import NoneType -from six.moves import cStringIO as StringIO -from datetime import datetime -import json - - -from twisted.python import usage - -from allmydata.util.assertutil import precondition - -from .common import BaseOptions, BasedirOptions, get_aliases -from .cli import MakeDirectoryOptions, LnOptions, CreateAliasOptions -import tahoe_mv -from allmydata.util.encodingutil import argv_to_abspath, argv_to_unicode, to_str, \ - quote_local_unicode_path -from allmydata.scripts.common_http import do_http, BadResponse -from allmydata.util import fileutil -from allmydata import uri -from allmydata.util.abbreviate import abbreviate_space, abbreviate_time -from allmydata.frontends.magic_folder import load_magic_folders -from allmydata.frontends.magic_folder import save_magic_folders -from allmydata.frontends.magic_folder import maybe_upgrade_magic_folders - - -INVITE_SEPARATOR = "+" - -class CreateOptions(BasedirOptions): - nickname = None # NOTE: *not* the "name of this magic-folder" - local_dir = None - synopsis = "MAGIC_ALIAS: [NICKNAME LOCAL_DIR]" - optParameters = [ - ("poll-interval", "p", "60", "How often to ask for updates"), - ("name", "n", "default", "The name of this magic-folder"), - ] - description = ( - "Create a new magic-folder. If you specify NICKNAME and " - "LOCAL_DIR, this client will also be invited and join " - "using the given nickname. A new alias (see 'tahoe list-aliases') " - "will be added with the master folder's writecap." - ) - - def parseArgs(self, alias, nickname=None, local_dir=None): - BasedirOptions.parseArgs(self) - alias = argv_to_unicode(alias) - if not alias.endswith(u':'): - raise usage.UsageError("An alias must end with a ':' character.") - self.alias = alias[:-1] - self.nickname = None if nickname is None else argv_to_unicode(nickname) - try: - if int(self['poll-interval']) <= 0: - raise ValueError("should be positive") - except ValueError: - raise usage.UsageError( - "--poll-interval must be a positive integer" - ) - - # Expand the path relative to the current directory of the CLI command, not the node. - self.local_dir = None if local_dir is None else argv_to_abspath(local_dir, long_path=False) - - if self.nickname and not self.local_dir: - raise usage.UsageError("If NICKNAME is specified then LOCAL_DIR must also be specified.") - node_url_file = os.path.join(self['node-directory'], u"node.url") - self['node-url'] = fileutil.read(node_url_file).strip() - -def _delegate_options(source_options, target_options): - target_options.aliases = get_aliases(source_options['node-directory']) - target_options["node-url"] = source_options["node-url"] - target_options["node-directory"] = source_options["node-directory"] - target_options["name"] = source_options["name"] - target_options.stdin = StringIO("") - target_options.stdout = StringIO() - target_options.stderr = StringIO() - return target_options - -def create(options): - precondition(isinstance(options.alias, unicode), alias=options.alias) - precondition(isinstance(options.nickname, (unicode, NoneType)), nickname=options.nickname) - precondition(isinstance(options.local_dir, (unicode, NoneType)), local_dir=options.local_dir) - - # make sure we don't already have a magic-folder with this name before we create the alias - maybe_upgrade_magic_folders(options["node-directory"]) - folders = load_magic_folders(options["node-directory"]) - if options['name'] in folders: - print("Already have a magic-folder named '{}'".format(options['name']), file=options.stderr) - return 1 - - # create an alias; this basically just remembers the cap for the - # master directory - from allmydata.scripts import tahoe_add_alias - create_alias_options = _delegate_options(options, CreateAliasOptions()) - create_alias_options.alias = options.alias - - rc = tahoe_add_alias.create_alias(create_alias_options) - if rc != 0: - print(create_alias_options.stderr.getvalue(), file=options.stderr) - return rc - print(create_alias_options.stdout.getvalue(), file=options.stdout) - - if options.nickname is not None: - print(u"Inviting myself as client '{}':".format(options.nickname), file=options.stdout) - invite_options = _delegate_options(options, InviteOptions()) - invite_options.alias = options.alias - invite_options.nickname = options.nickname - invite_options['name'] = options['name'] - rc = invite(invite_options) - if rc != 0: - print(u"magic-folder: failed to invite after create\n", file=options.stderr) - print(invite_options.stderr.getvalue(), file=options.stderr) - return rc - invite_code = invite_options.stdout.getvalue().strip() - print(u" created invite code", file=options.stdout) - join_options = _delegate_options(options, JoinOptions()) - join_options['poll-interval'] = options['poll-interval'] - join_options.nickname = options.nickname - join_options.local_dir = options.local_dir - join_options.invite_code = invite_code - rc = join(join_options) - if rc != 0: - print(u"magic-folder: failed to join after create\n", file=options.stderr) - print(join_options.stderr.getvalue(), file=options.stderr) - return rc - print(u" joined new magic-folder", file=options.stdout) - print( - u"Successfully created magic-folder '{}' with alias '{}:' " - u"and client '{}'\nYou must re-start your node before the " - u"magic-folder will be active." - .format(options['name'], options.alias, options.nickname), file=options.stdout) - return 0 - - -class ListOptions(BasedirOptions): - description = ( - "List all magic-folders this client has joined" - ) - optFlags = [ - ("json", "", "Produce JSON output") - ] - - -def list_(options): - folders = load_magic_folders(options["node-directory"]) - if options["json"]: - _list_json(options, folders) - return 0 - _list_human(options, folders) - return 0 - - -def _list_json(options, folders): - """ - List our magic-folders using JSON - """ - info = dict() - for name, details in folders.items(): - info[name] = { - u"directory": details["directory"], - } - print(json.dumps(info), file=options.stdout) - return 0 - - -def _list_human(options, folders): - """ - List our magic-folders for a human user - """ - if folders: - print("This client has the following magic-folders:", file=options.stdout) - biggest = max([len(nm) for nm in folders.keys()]) - fmt = " {:>%d}: {}" % (biggest, ) - for name, details in folders.items(): - print(fmt.format(name, details["directory"]), file=options.stdout) - else: - print("No magic-folders", file=options.stdout) - - -class InviteOptions(BasedirOptions): - nickname = None - synopsis = "MAGIC_ALIAS: NICKNAME" - stdin = StringIO("") - optParameters = [ - ("name", "n", "default", "The name of this magic-folder"), - ] - description = ( - "Invite a new participant to a given magic-folder. The resulting " - "invite-code that is printed is secret information and MUST be " - "transmitted securely to the invitee." - ) - - def parseArgs(self, alias, nickname=None): - BasedirOptions.parseArgs(self) - alias = argv_to_unicode(alias) - if not alias.endswith(u':'): - raise usage.UsageError("An alias must end with a ':' character.") - self.alias = alias[:-1] - self.nickname = argv_to_unicode(nickname) - node_url_file = os.path.join(self['node-directory'], u"node.url") - self['node-url'] = open(node_url_file, "r").read().strip() - aliases = get_aliases(self['node-directory']) - self.aliases = aliases - - -def invite(options): - precondition(isinstance(options.alias, unicode), alias=options.alias) - precondition(isinstance(options.nickname, unicode), nickname=options.nickname) - - from allmydata.scripts import tahoe_mkdir - mkdir_options = _delegate_options(options, MakeDirectoryOptions()) - mkdir_options.where = None - - rc = tahoe_mkdir.mkdir(mkdir_options) - if rc != 0: - print("magic-folder: failed to mkdir\n", file=options.stderr) - return rc - - # FIXME this assumes caps are ASCII. - dmd_write_cap = mkdir_options.stdout.getvalue().strip() - dmd_readonly_cap = uri.from_string(dmd_write_cap).get_readonly().to_string() - if dmd_readonly_cap is None: - print("magic-folder: failed to diminish dmd write cap\n", file=options.stderr) - return 1 - - magic_write_cap = get_aliases(options["node-directory"])[options.alias] - magic_readonly_cap = uri.from_string(magic_write_cap).get_readonly().to_string() - - # tahoe ln CLIENT_READCAP COLLECTIVE_WRITECAP/NICKNAME - ln_options = _delegate_options(options, LnOptions()) - ln_options.from_file = unicode(dmd_readonly_cap, 'utf-8') - ln_options.to_file = u"%s/%s" % (unicode(magic_write_cap, 'utf-8'), options.nickname) - rc = tahoe_mv.mv(ln_options, mode="link") - if rc != 0: - print("magic-folder: failed to create link\n", file=options.stderr) - print(ln_options.stderr.getvalue(), file=options.stderr) - return rc - - # FIXME: this assumes caps are ASCII. - print("%s%s%s" % (magic_readonly_cap, INVITE_SEPARATOR, dmd_write_cap), file=options.stdout) - return 0 - -class JoinOptions(BasedirOptions): - synopsis = "INVITE_CODE LOCAL_DIR" - dmd_write_cap = "" - magic_readonly_cap = "" - optParameters = [ - ("poll-interval", "p", "60", "How often to ask for updates"), - ("name", "n", "default", "Name of the magic-folder"), - ] - - def parseArgs(self, invite_code, local_dir): - BasedirOptions.parseArgs(self) - - try: - if int(self['poll-interval']) <= 0: - raise ValueError("should be positive") - except ValueError: - raise usage.UsageError( - "--poll-interval must be a positive integer" - ) - # Expand the path relative to the current directory of the CLI command, not the node. - self.local_dir = None if local_dir is None else argv_to_abspath(local_dir, long_path=False) - self.invite_code = to_str(argv_to_unicode(invite_code)) - -def join(options): - fields = options.invite_code.split(INVITE_SEPARATOR) - if len(fields) != 2: - raise usage.UsageError("Invalid invite code.") - magic_readonly_cap, dmd_write_cap = fields - - maybe_upgrade_magic_folders(options["node-directory"]) - existing_folders = load_magic_folders(options["node-directory"]) - - if options['name'] in existing_folders: - print("This client already has a magic-folder named '{}'".format(options['name']), file=options.stderr) - return 1 - - db_fname = os.path.join( - options["node-directory"], - u"private", - u"magicfolder_{}.sqlite".format(options['name']), - ) - if os.path.exists(db_fname): - print("Database '{}' already exists; not overwriting".format(db_fname), file=options.stderr) - return 1 - - folder = { - u"directory": options.local_dir.encode('utf-8'), - u"collective_dircap": magic_readonly_cap, - u"upload_dircap": dmd_write_cap, - u"poll_interval": options["poll-interval"], - } - existing_folders[options["name"]] = folder - - save_magic_folders(options["node-directory"], existing_folders) - return 0 - - -class LeaveOptions(BasedirOptions): - synopsis = "Remove a magic-folder and forget all state" - optParameters = [ - ("name", "n", "default", "Name of magic-folder to leave"), - ] - - -def leave(options): - from ConfigParser import SafeConfigParser - - existing_folders = load_magic_folders(options["node-directory"]) - - if not existing_folders: - print("No magic-folders at all", file=options.stderr) - return 1 - - if options["name"] not in existing_folders: - print("No such magic-folder '{}'".format(options["name"]), file=options.stderr) - return 1 - - privdir = os.path.join(options["node-directory"], u"private") - db_fname = os.path.join(privdir, u"magicfolder_{}.sqlite".format(options["name"])) - - # delete from YAML file and re-write it - del existing_folders[options["name"]] - save_magic_folders(options["node-directory"], existing_folders) - - # delete the database file - try: - fileutil.remove(db_fname) - except Exception as e: - print("Warning: unable to remove %s due to %s: %s" - % (quote_local_unicode_path(db_fname), e.__class__.__name__, str(e)), file=options.stderr) - - # if this was the last magic-folder, disable them entirely - if not existing_folders: - parser = SafeConfigParser() - parser.read(os.path.join(options["node-directory"], u"tahoe.cfg")) - parser.remove_section("magic_folder") - with open(os.path.join(options["node-directory"], u"tahoe.cfg"), "w") as f: - parser.write(f) - - return 0 - - -class StatusOptions(BasedirOptions): - synopsis = "" - stdin = StringIO("") - optParameters = [ - ("name", "n", "default", "Name for the magic-folder to show status"), - ] - - def parseArgs(self): - BasedirOptions.parseArgs(self) - node_url_file = os.path.join(self['node-directory'], u"node.url") - with open(node_url_file, "r") as f: - self['node-url'] = f.read().strip() - - -def _get_json_for_fragment(options, fragment, method='GET', post_args=None): - nodeurl = options['node-url'] - if nodeurl.endswith('/'): - nodeurl = nodeurl[:-1] - - url = u'%s/%s' % (nodeurl, fragment) - if method == 'POST': - if post_args is None: - raise ValueError("Must pass post_args= for POST method") - body = urllib.urlencode(post_args) - else: - body = '' - if post_args is not None: - raise ValueError("post_args= only valid for POST method") - resp = do_http(method, url, body=body) - if isinstance(resp, BadResponse): - # specifically NOT using format_http_error() here because the - # URL is pretty sensitive (we're doing /uri/). - raise RuntimeError( - "Failed to get json from '%s': %s" % (nodeurl, resp.error) - ) - - data = resp.read() - parsed = json.loads(data) - if parsed is None: - raise RuntimeError("No data from '%s'" % (nodeurl,)) - return parsed - - -def _get_json_for_cap(options, cap): - return _get_json_for_fragment( - options, - 'uri/%s?t=json' % urllib.quote(cap), - ) - -def _print_item_status(item, now, longest): - paddedname = (' ' * (longest - len(item['path']))) + item['path'] - if 'failure_at' in item: - ts = datetime.fromtimestamp(item['started_at']) - prog = 'Failed %s (%s)' % (abbreviate_time(now - ts), ts) - elif item['percent_done'] < 100.0: - if 'started_at' not in item: - prog = 'not yet started' - else: - so_far = now - datetime.fromtimestamp(item['started_at']) - if so_far.seconds > 0.0: - rate = item['percent_done'] / so_far.seconds - if rate != 0: - time_left = (100.0 - item['percent_done']) / rate - prog = '%2.1f%% done, around %s left' % ( - item['percent_done'], - abbreviate_time(time_left), - ) - else: - time_left = None - prog = '%2.1f%% done' % (item['percent_done'],) - else: - prog = 'just started' - else: - prog = '' - for verb in ['finished', 'started', 'queued']: - keyname = verb + '_at' - if keyname in item: - when = datetime.fromtimestamp(item[keyname]) - prog = '%s %s' % (verb, abbreviate_time(now - when)) - break - - print(" %s: %s" % (paddedname, prog)) - - -def status(options): - nodedir = options["node-directory"] - stdout, stderr = options.stdout, options.stderr - magic_folders = load_magic_folders(os.path.join(options["node-directory"])) - - with open(os.path.join(nodedir, u'private', u'api_auth_token'), 'rb') as f: - token = f.read() - - print("Magic-folder status for '{}':".format(options["name"]), file=stdout) - - if options["name"] not in magic_folders: - raise Exception( - "No such magic-folder '{}'".format(options["name"]) - ) - - dmd_cap = magic_folders[options["name"]]["upload_dircap"] - collective_readcap = magic_folders[options["name"]]["collective_dircap"] - - # do *all* our data-retrievals first in case there's an error - try: - dmd_data = _get_json_for_cap(options, dmd_cap) - remote_data = _get_json_for_cap(options, collective_readcap) - magic_data = _get_json_for_fragment( - options, - 'magic_folder?t=json', - method='POST', - post_args=dict( - t='json', - name=options["name"], - token=token, - ) - ) - except Exception as e: - print("failed to retrieve data: %s" % str(e), file=stderr) - return 2 - - for d in [dmd_data, remote_data, magic_data]: - if isinstance(d, dict) and 'error' in d: - print("Error from server: %s" % d['error'], file=stderr) - print("This means we can't retrieve the remote shared directory.", file=stderr) - return 3 - - captype, dmd = dmd_data - if captype != 'dirnode': - print("magic_folder_dircap isn't a directory capability", file=stderr) - return 2 - - now = datetime.now() - - print("Local files:", file=stdout) - for (name, child) in dmd['children'].items(): - captype, meta = child - status = 'good' - size = meta['size'] - created = datetime.fromtimestamp(meta['metadata']['tahoe']['linkcrtime']) - version = meta['metadata']['version'] - nice_size = abbreviate_space(size) - nice_created = abbreviate_time(now - created) - if captype != 'filenode': - print("%20s: error, should be a filecap" % name, file=stdout) - continue - print(" %s (%s): %s, version=%s, created %s" % (name, nice_size, status, version, nice_created), file=stdout) - - print(file=stdout) - print("Remote files:", file=stdout) - - captype, collective = remote_data - for (name, data) in collective['children'].items(): - if data[0] != 'dirnode': - print("Error: '%s': expected a dirnode, not '%s'" % (name, data[0]), file=stdout) - print(" %s's remote:" % name, file=stdout) - dmd = _get_json_for_cap(options, data[1]['ro_uri']) - if isinstance(dmd, dict) and 'error' in dmd: - print(" Error: could not retrieve directory", file=stdout) - continue - if dmd[0] != 'dirnode': - print("Error: should be a dirnode", file=stdout) - continue - for (n, d) in dmd[1]['children'].items(): - if d[0] != 'filenode': - print("Error: expected '%s' to be a filenode." % (n,), file=stdout) - - meta = d[1] - status = 'good' - size = meta['size'] - created = datetime.fromtimestamp(meta['metadata']['tahoe']['linkcrtime']) - version = meta['metadata']['version'] - nice_size = abbreviate_space(size) - nice_created = abbreviate_time(now - created) - print(" %s (%s): %s, version=%s, created %s" % (n, nice_size, status, version, nice_created), file=stdout) - - if len(magic_data): - uploads = [item for item in magic_data if item['kind'] == 'upload'] - downloads = [item for item in magic_data if item['kind'] == 'download'] - longest = max([len(item['path']) for item in magic_data]) - - # maybe gate this with --show-completed option or something? - uploads = [item for item in uploads if item['status'] != 'success'] - downloads = [item for item in downloads if item['status'] != 'success'] - - if len(uploads): - print() - print("Uploads:", file=stdout) - for item in uploads: - _print_item_status(item, now, longest) - - if len(downloads): - print() - print("Downloads:", file=stdout) - for item in downloads: - _print_item_status(item, now, longest) - - for item in magic_data: - if item['status'] == 'failure': - print("Failed:", item, file=stdout) - - return 0 - - -class MagicFolderCommand(BaseOptions): - subCommands = [ - ["create", None, CreateOptions, "Create a Magic Folder."], - ["invite", None, InviteOptions, "Invite someone to a Magic Folder."], - ["join", None, JoinOptions, "Join a Magic Folder."], - ["leave", None, LeaveOptions, "Leave a Magic Folder."], - ["status", None, StatusOptions, "Display status of uploads/downloads."], - ["list", None, ListOptions, "List Magic Folders configured in this client."], - ] - optFlags = [ - ["debug", "d", "Print full stack-traces"], - ] - description = ( - "A magic-folder has an owner who controls the writecap " - "containing a list of nicknames and readcaps. The owner can invite " - "new participants. Every participant has the writecap for their " - "own folder (the corresponding readcap is in the master folder). " - "All clients download files from all other participants using the " - "readcaps contained in the master magic-folder directory." - ) - - def postOptions(self): - if not hasattr(self, 'subOptions'): - raise usage.UsageError("must specify a subcommand") - def getSynopsis(self): - return "Usage: tahoe [global-options] magic-folder" - def getUsage(self, width=None): - t = BaseOptions.getUsage(self, width) - t += ( - "Please run e.g. 'tahoe magic-folder create --help' for more " - "details on each subcommand.\n" - ) - return t - -subDispatch = { - "create": create, - "invite": invite, - "join": join, - "leave": leave, - "status": status, - "list": list_, -} - -def do_magic_folder(options): - so = options.subOptions - so.stdout = options.stdout - so.stderr = options.stderr - f = subDispatch[options.subCommand] - try: - return f(so) - except Exception as e: - print("Error: %s" % (e,), file=options.stderr) - if options['debug']: - raise - -subCommands = [ - ["magic-folder", None, MagicFolderCommand, - "Magic Folder subcommands: use 'tahoe magic-folder' for a list."], -] - -dispatch = { - "magic-folder": do_magic_folder, -} diff --git a/src/allmydata/scripts/runner.py b/src/allmydata/scripts/runner.py index 60322c426..705d3193b 100644 --- a/src/allmydata/scripts/runner.py +++ b/src/allmydata/scripts/runner.py @@ -9,7 +9,7 @@ from twisted.internet import defer, task, threads from allmydata.version_checks import get_package_versions_string from allmydata.scripts.common import get_default_nodedir from allmydata.scripts import debug, create_node, cli, \ - stats_gatherer, admin, magic_folder_cli, tahoe_daemonize, tahoe_start, \ + stats_gatherer, admin, tahoe_daemonize, tahoe_start, \ tahoe_stop, tahoe_restart, tahoe_run, tahoe_invite from allmydata.util.encodingutil import quote_output, quote_local_unicode_path, get_io_encoding from allmydata.util.eliotutil import ( @@ -61,7 +61,6 @@ class Options(usage.Options): + process_control_commands + debug.subCommands + cli.subCommands - + magic_folder_cli.subCommands + tahoe_invite.subCommands ) @@ -154,10 +153,6 @@ def dispatch(config, # these are blocking, and must be run in a thread f0 = cli.dispatch[command] f = lambda so: threads.deferToThread(f0, so) - elif command in magic_folder_cli.dispatch: - # same - f0 = magic_folder_cli.dispatch[command] - f = lambda so: threads.deferToThread(f0, so) elif command in tahoe_invite.dispatch: f = tahoe_invite.dispatch[command] else: diff --git a/src/allmydata/test/cli/test_magic_folder.py b/src/allmydata/test/cli/test_magic_folder.py deleted file mode 100644 index b8d26b893..000000000 --- a/src/allmydata/test/cli/test_magic_folder.py +++ /dev/null @@ -1,814 +0,0 @@ -import json -import shutil -import os.path -import mock -import re -import time -from datetime import datetime - -from eliot import ( - log_call, - start_action, -) -from eliot.twisted import ( - DeferredContext, -) - -from twisted.trial import unittest -from twisted.internet import defer -from twisted.internet import reactor -from twisted.python import usage - -from allmydata.util.assertutil import precondition -from allmydata.util import fileutil -from allmydata.scripts.common import get_aliases -from ..no_network import GridTestMixin -from ..common_util import parse_cli -from .common import CLITestMixin -from allmydata.test.common_util import NonASCIIPathMixin -from allmydata.scripts import magic_folder_cli -from allmydata.util.fileutil import abspath_expanduser_unicode -from allmydata.util.encodingutil import unicode_to_argv -from allmydata.frontends.magic_folder import MagicFolder -from allmydata import uri -from ...util.eliotutil import ( - log_call_deferred, -) - -class MagicFolderCLITestMixin(CLITestMixin, GridTestMixin, NonASCIIPathMixin): - def setUp(self): - GridTestMixin.setUp(self) - self.alice_nickname = self.unicode_or_fallback(u"Alice\u00F8", u"Alice", io_as_well=True) - self.bob_nickname = self.unicode_or_fallback(u"Bob\u00F8", u"Bob", io_as_well=True) - - def do_create_magic_folder(self, client_num): - with start_action(action_type=u"create-magic-folder", client_num=client_num).context(): - d = DeferredContext( - self.do_cli( - "magic-folder", "--debug", "create", "magic:", - client_num=client_num, - ) - ) - def _done(args): - (rc, stdout, stderr) = args - self.failUnlessEqual(rc, 0, stdout + stderr) - self.assertIn("Alias 'magic' created", stdout) -# self.failUnlessIn("joined new magic-folder", stdout) -# self.failUnlessIn("Successfully created magic-folder", stdout) - self.failUnlessEqual(stderr, "") - aliases = get_aliases(self.get_clientdir(i=client_num)) - self.assertIn("magic", aliases) - self.failUnless(aliases["magic"].startswith("URI:DIR2:")) - d.addCallback(_done) - return d.addActionFinish() - - def do_invite(self, client_num, nickname): - nickname_arg = unicode_to_argv(nickname) - action = start_action( - action_type=u"invite-to-magic-folder", - client_num=client_num, - nickname=nickname, - ) - with action.context(): - d = DeferredContext( - self.do_cli( - "magic-folder", - "invite", - "magic:", - nickname_arg, - client_num=client_num, - ) - ) - def _done(args): - (rc, stdout, stderr) = args - self.failUnlessEqual(rc, 0) - return (rc, stdout, stderr) - d.addCallback(_done) - return d.addActionFinish() - - def do_list(self, client_num, json=False): - args = ("magic-folder", "list",) - if json: - args = args + ("--json",) - d = self.do_cli(*args, client_num=client_num) - def _done(args): - (rc, stdout, stderr) = args - return (rc, stdout, stderr) - d.addCallback(_done) - return d - - def do_status(self, client_num, name=None): - args = ("magic-folder", "status",) - if name is not None: - args = args + ("--name", name) - d = self.do_cli(*args, client_num=client_num) - def _done(args): - (rc, stdout, stderr) = args - return (rc, stdout, stderr) - d.addCallback(_done) - return d - - def do_join(self, client_num, local_dir, invite_code): - action = start_action( - action_type=u"join-magic-folder", - client_num=client_num, - local_dir=local_dir, - invite_code=invite_code, - ) - with action.context(): - precondition(isinstance(local_dir, unicode), local_dir=local_dir) - precondition(isinstance(invite_code, str), invite_code=invite_code) - local_dir_arg = unicode_to_argv(local_dir) - d = DeferredContext( - self.do_cli( - "magic-folder", - "join", - invite_code, - local_dir_arg, - client_num=client_num, - ) - ) - def _done(args): - (rc, stdout, stderr) = args - self.failUnlessEqual(rc, 0) - self.failUnlessEqual(stdout, "") - self.failUnlessEqual(stderr, "") - return (rc, stdout, stderr) - d.addCallback(_done) - return d.addActionFinish() - - def do_leave(self, client_num): - d = self.do_cli("magic-folder", "leave", client_num=client_num) - def _done(args): - (rc, stdout, stderr) = args - self.failUnlessEqual(rc, 0) - return (rc, stdout, stderr) - d.addCallback(_done) - return d - - def check_joined_config(self, client_num, upload_dircap): - """Tests that our collective directory has the readonly cap of - our upload directory. - """ - action = start_action(action_type=u"check-joined-config") - with action.context(): - collective_readonly_cap = self.get_caps_from_files(client_num)[0] - d = DeferredContext( - self.do_cli( - "ls", "--json", - collective_readonly_cap, - client_num=client_num, - ) - ) - def _done(args): - (rc, stdout, stderr) = args - self.failUnlessEqual(rc, 0) - return (rc, stdout, stderr) - d.addCallback(_done) - def test_joined_magic_folder(args): - (rc, stdout, stderr) = args - readonly_cap = unicode(uri.from_string(upload_dircap).get_readonly().to_string(), 'utf-8') - s = re.search(readonly_cap, stdout) - self.failUnless(s is not None) - return None - d.addCallback(test_joined_magic_folder) - return d.addActionFinish() - - def get_caps_from_files(self, client_num): - from allmydata.frontends.magic_folder import load_magic_folders - folders = load_magic_folders(self.get_clientdir(i=client_num)) - mf = folders["default"] - return mf['collective_dircap'], mf['upload_dircap'] - - @log_call - def check_config(self, client_num, local_dir): - client_config = fileutil.read(os.path.join(self.get_clientdir(i=client_num), "tahoe.cfg")) - mf_yaml = fileutil.read(os.path.join(self.get_clientdir(i=client_num), "private", "magic_folders.yaml")) - local_dir_utf8 = local_dir.encode('utf-8') - magic_folder_config = "[magic_folder]\nenabled = True" - self.assertIn(magic_folder_config, client_config) - self.assertIn(local_dir_utf8, mf_yaml) - - def create_invite_join_magic_folder(self, nickname, local_dir): - nickname_arg = unicode_to_argv(nickname) - local_dir_arg = unicode_to_argv(local_dir) - # the --debug means we get real exceptions on failures - d = self.do_cli("magic-folder", "--debug", "create", "magic:", nickname_arg, local_dir_arg) - def _done(args): - (rc, stdout, stderr) = args - self.failUnlessEqual(rc, 0, stdout + stderr) - - client = self.get_client() - self.collective_dircap, self.upload_dircap = self.get_caps_from_files(0) - self.collective_dirnode = client.create_node_from_uri(self.collective_dircap) - self.upload_dirnode = client.create_node_from_uri(self.upload_dircap) - d.addCallback(_done) - d.addCallback(lambda ign: self.check_joined_config(0, self.upload_dircap)) - d.addCallback(lambda ign: self.check_config(0, local_dir)) - return d - - # XXX should probably just be "tearDown"... - @log_call_deferred(action_type=u"test:cli:magic-folder:cleanup") - def cleanup(self, res): - d = DeferredContext(defer.succeed(None)) - def _clean(ign): - return self.magicfolder.disownServiceParent() - - d.addCallback(_clean) - d.addCallback(lambda ign: res) - return d.result - - def init_magicfolder(self, client_num, upload_dircap, collective_dircap, local_magic_dir, clock): - dbfile = abspath_expanduser_unicode(u"magicfolder_default.sqlite", base=self.get_clientdir(i=client_num)) - magicfolder = MagicFolder( - client=self.get_client(client_num), - upload_dircap=upload_dircap, - collective_dircap=collective_dircap, - local_path_u=local_magic_dir, - dbfile=dbfile, - umask=0o077, - name='default', - clock=clock, - uploader_delay=0.2, - downloader_delay=0, - ) - - magicfolder.setServiceParent(self.get_client(client_num)) - magicfolder.ready() - return magicfolder - - def setup_alice_and_bob(self, alice_clock=reactor, bob_clock=reactor): - self.set_up_grid(num_clients=2, oneshare=True) - - self.alice_magicfolder = None - self.bob_magicfolder = None - - alice_magic_dir = abspath_expanduser_unicode(u"Alice-magic", base=self.basedir) - self.mkdir_nonascii(alice_magic_dir) - bob_magic_dir = abspath_expanduser_unicode(u"Bob-magic", base=self.basedir) - self.mkdir_nonascii(bob_magic_dir) - - # Alice creates a Magic Folder, invites herself and joins. - d = self.do_create_magic_folder(0) - d.addCallback(lambda ign: self.do_invite(0, self.alice_nickname)) - def get_invite_code(result): - self.invite_code = result[1].strip() - d.addCallback(get_invite_code) - d.addCallback(lambda ign: self.do_join(0, alice_magic_dir, self.invite_code)) - def get_alice_caps(ign): - self.alice_collective_dircap, self.alice_upload_dircap = self.get_caps_from_files(0) - d.addCallback(get_alice_caps) - d.addCallback(lambda ign: self.check_joined_config(0, self.alice_upload_dircap)) - d.addCallback(lambda ign: self.check_config(0, alice_magic_dir)) - def get_Alice_magicfolder(result): - self.alice_magicfolder = self.init_magicfolder(0, self.alice_upload_dircap, - self.alice_collective_dircap, - alice_magic_dir, alice_clock) - return result - d.addCallback(get_Alice_magicfolder) - - # Alice invites Bob. Bob joins. - d.addCallback(lambda ign: self.do_invite(0, self.bob_nickname)) - def get_invite_code(result): - self.invite_code = result[1].strip() - d.addCallback(get_invite_code) - d.addCallback(lambda ign: self.do_join(1, bob_magic_dir, self.invite_code)) - def get_bob_caps(ign): - self.bob_collective_dircap, self.bob_upload_dircap = self.get_caps_from_files(1) - d.addCallback(get_bob_caps) - d.addCallback(lambda ign: self.check_joined_config(1, self.bob_upload_dircap)) - d.addCallback(lambda ign: self.check_config(1, bob_magic_dir)) - def get_Bob_magicfolder(result): - self.bob_magicfolder = self.init_magicfolder(1, self.bob_upload_dircap, - self.bob_collective_dircap, - bob_magic_dir, bob_clock) - return result - d.addCallback(get_Bob_magicfolder) - return d - - -class ListMagicFolder(MagicFolderCLITestMixin, unittest.TestCase): - - @defer.inlineCallbacks - def setUp(self): - yield super(ListMagicFolder, self).setUp() - self.basedir="mf_list" - self.set_up_grid(oneshare=True) - self.local_dir = os.path.join(self.basedir, "magic") - os.mkdir(self.local_dir) - self.abs_local_dir_u = abspath_expanduser_unicode(unicode(self.local_dir), long_path=False) - - yield self.do_create_magic_folder(0) - (rc, stdout, stderr) = yield self.do_invite(0, self.alice_nickname) - invite_code = stdout.strip() - yield self.do_join(0, unicode(self.local_dir), invite_code) - - @defer.inlineCallbacks - def tearDown(self): - yield super(ListMagicFolder, self).tearDown() - shutil.rmtree(self.basedir) - - @defer.inlineCallbacks - def test_list(self): - rc, stdout, stderr = yield self.do_list(0) - self.failUnlessEqual(rc, 0) - self.assertIn("default:", stdout) - - @defer.inlineCallbacks - def test_list_none(self): - yield self.do_leave(0) - rc, stdout, stderr = yield self.do_list(0) - self.failUnlessEqual(rc, 0) - self.assertIn("No magic-folders", stdout) - - @defer.inlineCallbacks - def test_list_json(self): - rc, stdout, stderr = yield self.do_list(0, json=True) - self.failUnlessEqual(rc, 0) - res = json.loads(stdout) - self.assertEqual( - dict(default=dict(directory=self.abs_local_dir_u)), - res, - ) - - -class StatusMagicFolder(MagicFolderCLITestMixin, unittest.TestCase): - - @defer.inlineCallbacks - def setUp(self): - yield super(StatusMagicFolder, self).setUp() - self.basedir="mf_list" - self.set_up_grid(oneshare=True) - self.local_dir = os.path.join(self.basedir, "magic") - os.mkdir(self.local_dir) - self.abs_local_dir_u = abspath_expanduser_unicode(unicode(self.local_dir), long_path=False) - - yield self.do_create_magic_folder(0) - (rc, stdout, stderr) = yield self.do_invite(0, self.alice_nickname) - invite_code = stdout.strip() - yield self.do_join(0, unicode(self.local_dir), invite_code) - - @defer.inlineCallbacks - def tearDown(self): - yield super(StatusMagicFolder, self).tearDown() - shutil.rmtree(self.basedir) - - @defer.inlineCallbacks - def test_status(self): - now = datetime.now() - then = now.replace(year=now.year - 5) - five_year_interval = (now - then).total_seconds() - - def json_for_cap(options, cap): - if cap.startswith('URI:DIR2:'): - return ( - 'dirnode', - { - "children": { - "foo": ('filenode', { - "size": 1234, - "metadata": { - "tahoe": { - "linkcrtime": (time.time() - five_year_interval), - }, - "version": 1, - }, - "ro_uri": "read-only URI", - }) - } - } - ) - else: - return ('dirnode', {"children": {}}) - jc = mock.patch( - "allmydata.scripts.magic_folder_cli._get_json_for_cap", - side_effect=json_for_cap, - ) - - def json_for_frag(options, fragment, method='GET', post_args=None): - return {} - jf = mock.patch( - "allmydata.scripts.magic_folder_cli._get_json_for_fragment", - side_effect=json_for_frag, - ) - - with jc, jf: - rc, stdout, stderr = yield self.do_status(0) - self.failUnlessEqual(rc, 0) - self.assertIn("default", stdout) - - self.assertIn( - "foo (1.23 kB): good, version=1, created 5 years ago", - stdout, - ) - - @defer.inlineCallbacks - def test_status_child_not_dirnode(self): - def json_for_cap(options, cap): - if cap.startswith('URI:DIR2'): - return ( - 'dirnode', - { - "children": { - "foo": ('filenode', { - "size": 1234, - "metadata": { - "tahoe": { - "linkcrtime": 0.0, - }, - "version": 1, - }, - "ro_uri": "read-only URI", - }) - } - } - ) - elif cap == "read-only URI": - return { - "error": "bad stuff", - } - else: - return ('dirnode', {"children": {}}) - jc = mock.patch( - "allmydata.scripts.magic_folder_cli._get_json_for_cap", - side_effect=json_for_cap, - ) - - def json_for_frag(options, fragment, method='GET', post_args=None): - return {} - jf = mock.patch( - "allmydata.scripts.magic_folder_cli._get_json_for_fragment", - side_effect=json_for_frag, - ) - - with jc, jf: - rc, stdout, stderr = yield self.do_status(0) - self.failUnlessEqual(rc, 0) - - self.assertIn( - "expected a dirnode", - stdout + stderr, - ) - - @defer.inlineCallbacks - def test_status_error_not_dircap(self): - def json_for_cap(options, cap): - if cap.startswith('URI:DIR2:'): - return ( - 'filenode', - {} - ) - else: - return ('dirnode', {"children": {}}) - jc = mock.patch( - "allmydata.scripts.magic_folder_cli._get_json_for_cap", - side_effect=json_for_cap, - ) - - def json_for_frag(options, fragment, method='GET', post_args=None): - return {} - jf = mock.patch( - "allmydata.scripts.magic_folder_cli._get_json_for_fragment", - side_effect=json_for_frag, - ) - - with jc, jf: - rc, stdout, stderr = yield self.do_status(0) - self.failUnlessEqual(rc, 2) - self.assertIn( - "magic_folder_dircap isn't a directory capability", - stdout + stderr, - ) - - @defer.inlineCallbacks - def test_status_nothing(self): - rc, stdout, stderr = yield self.do_status(0, name="blam") - self.assertIn("No such magic-folder 'blam'", stderr) - - -class CreateMagicFolder(MagicFolderCLITestMixin, unittest.TestCase): - def test_create_and_then_invite_join(self): - self.basedir = "cli/MagicFolder/create-and-then-invite-join" - self.set_up_grid(oneshare=True) - local_dir = os.path.join(self.basedir, "magic") - os.mkdir(local_dir) - abs_local_dir_u = abspath_expanduser_unicode(unicode(local_dir), long_path=False) - - d = self.do_create_magic_folder(0) - d.addCallback(lambda ign: self.do_invite(0, self.alice_nickname)) - def get_invite_code_and_join(args): - (rc, stdout, stderr) = args - invite_code = stdout.strip() - return self.do_join(0, unicode(local_dir), invite_code) - d.addCallback(get_invite_code_and_join) - def get_caps(ign): - self.collective_dircap, self.upload_dircap = self.get_caps_from_files(0) - d.addCallback(get_caps) - d.addCallback(lambda ign: self.check_joined_config(0, self.upload_dircap)) - d.addCallback(lambda ign: self.check_config(0, abs_local_dir_u)) - return d - - def test_create_error(self): - self.basedir = "cli/MagicFolder/create-error" - self.set_up_grid(oneshare=True) - - d = self.do_cli("magic-folder", "create", "m a g i c:", client_num=0) - def _done(args): - (rc, stdout, stderr) = args - self.failIfEqual(rc, 0) - self.failUnlessIn("Alias names cannot contain spaces.", stderr) - d.addCallback(_done) - return d - - @defer.inlineCallbacks - def test_create_duplicate_name(self): - self.basedir = "cli/MagicFolder/create-dup" - self.set_up_grid(oneshare=True) - - rc, stdout, stderr = yield self.do_cli( - "magic-folder", "create", "magic:", "--name", "foo", - client_num=0, - ) - self.assertEqual(rc, 0) - - rc, stdout, stderr = yield self.do_cli( - "magic-folder", "create", "magic:", "--name", "foo", - client_num=0, - ) - self.assertEqual(rc, 1) - self.assertIn( - "Already have a magic-folder named 'default'", - stderr - ) - - @defer.inlineCallbacks - def test_leave_wrong_folder(self): - self.basedir = "cli/MagicFolder/leave_wrong_folders" - yield self.set_up_grid(oneshare=True) - magic_dir = os.path.join(self.basedir, 'magic') - os.mkdir(magic_dir) - - rc, stdout, stderr = yield self.do_cli( - "magic-folder", "create", "--name", "foo", "magic:", "my_name", magic_dir, - client_num=0, - ) - self.assertEqual(rc, 0) - - rc, stdout, stderr = yield self.do_cli( - "magic-folder", "leave", "--name", "bar", - client_num=0, - ) - self.assertNotEqual(rc, 0) - self.assertIn( - "No such magic-folder 'bar'", - stdout + stderr, - ) - - @defer.inlineCallbacks - def test_leave_no_folder(self): - self.basedir = "cli/MagicFolder/leave_no_folders" - yield self.set_up_grid(oneshare=True) - magic_dir = os.path.join(self.basedir, 'magic') - os.mkdir(magic_dir) - - rc, stdout, stderr = yield self.do_cli( - "magic-folder", "create", "--name", "foo", "magic:", "my_name", magic_dir, - client_num=0, - ) - self.assertEqual(rc, 0) - - rc, stdout, stderr = yield self.do_cli( - "magic-folder", "leave", "--name", "foo", - client_num=0, - ) - self.assertEqual(rc, 0) - - rc, stdout, stderr = yield self.do_cli( - "magic-folder", "leave", "--name", "foo", - client_num=0, - ) - self.assertEqual(rc, 1) - self.assertIn( - "No magic-folders at all", - stderr, - ) - - @defer.inlineCallbacks - def test_leave_no_folders_at_all(self): - self.basedir = "cli/MagicFolder/leave_no_folders_at_all" - yield self.set_up_grid(oneshare=True) - - rc, stdout, stderr = yield self.do_cli( - "magic-folder", "leave", - client_num=0, - ) - self.assertEqual(rc, 1) - self.assertIn( - "No magic-folders at all", - stderr, - ) - - def test_create_invite_join(self): - self.basedir = "cli/MagicFolder/create-invite-join" - self.set_up_grid(oneshare=True) - local_dir = os.path.join(self.basedir, "magic") - abs_local_dir_u = abspath_expanduser_unicode(unicode(local_dir), long_path=False) - - d = self.do_cli("magic-folder", "create", "magic:", "Alice", local_dir) - def _done(args): - (rc, stdout, stderr) = args - self.failUnlessEqual(rc, 0) - self.collective_dircap, self.upload_dircap = self.get_caps_from_files(0) - d.addCallback(_done) - d.addCallback(lambda ign: self.check_joined_config(0, self.upload_dircap)) - d.addCallback(lambda ign: self.check_config(0, abs_local_dir_u)) - return d - - def test_help_synopsis(self): - self.basedir = "cli/MagicFolder/help_synopsis" - os.makedirs(self.basedir) - - o = magic_folder_cli.CreateOptions() - o.parent = magic_folder_cli.MagicFolderCommand() - o.parent.getSynopsis() - - def test_create_invite_join_failure(self): - self.basedir = "cli/MagicFolder/create-invite-join-failure" - os.makedirs(self.basedir) - - o = magic_folder_cli.CreateOptions() - o.parent = magic_folder_cli.MagicFolderCommand() - o.parent['node-directory'] = self.basedir - try: - o.parseArgs("magic:", "Alice", "-foo") - except usage.UsageError as e: - self.failUnlessIn("cannot start with '-'", str(e)) - else: - self.fail("expected UsageError") - - def test_join_failure(self): - self.basedir = "cli/MagicFolder/create-join-failure" - os.makedirs(self.basedir) - - o = magic_folder_cli.JoinOptions() - o.parent = magic_folder_cli.MagicFolderCommand() - o.parent['node-directory'] = self.basedir - try: - o.parseArgs("URI:invite+URI:code", "-foo") - except usage.UsageError as e: - self.failUnlessIn("cannot start with '-'", str(e)) - else: - self.fail("expected UsageError") - - def test_join_twice_failure(self): - self.basedir = "cli/MagicFolder/create-join-twice-failure" - os.makedirs(self.basedir) - self.set_up_grid(oneshare=True) - local_dir = os.path.join(self.basedir, "magic") - abs_local_dir_u = abspath_expanduser_unicode(unicode(local_dir), long_path=False) - - d = self.do_create_magic_folder(0) - d.addCallback(lambda ign: self.do_invite(0, self.alice_nickname)) - def get_invite_code_and_join(args): - (rc, stdout, stderr) = args - self.invite_code = stdout.strip() - return self.do_join(0, unicode(local_dir), self.invite_code) - d.addCallback(get_invite_code_and_join) - def get_caps(ign): - self.collective_dircap, self.upload_dircap = self.get_caps_from_files(0) - d.addCallback(get_caps) - d.addCallback(lambda ign: self.check_joined_config(0, self.upload_dircap)) - d.addCallback(lambda ign: self.check_config(0, abs_local_dir_u)) - def join_again(ignore): - return self.do_cli("magic-folder", "join", self.invite_code, local_dir, client_num=0) - d.addCallback(join_again) - def get_results(result): - (rc, out, err) = result - self.failUnlessEqual(out, "") - self.failUnlessIn("This client already has a magic-folder", err) - self.failIfEqual(rc, 0) - d.addCallback(get_results) - return d - - def test_join_leave_join(self): - self.basedir = "cli/MagicFolder/create-join-leave-join" - os.makedirs(self.basedir) - self.set_up_grid(oneshare=True) - local_dir = os.path.join(self.basedir, "magic") - abs_local_dir_u = abspath_expanduser_unicode(unicode(local_dir), long_path=False) - - self.invite_code = None - d = self.do_create_magic_folder(0) - d.addCallback(lambda ign: self.do_invite(0, self.alice_nickname)) - def get_invite_code_and_join(args): - (rc, stdout, stderr) = args - self.failUnlessEqual(rc, 0) - self.invite_code = stdout.strip() - return self.do_join(0, unicode(local_dir), self.invite_code) - d.addCallback(get_invite_code_and_join) - def get_caps(ign): - self.collective_dircap, self.upload_dircap = self.get_caps_from_files(0) - d.addCallback(get_caps) - d.addCallback(lambda ign: self.check_joined_config(0, self.upload_dircap)) - d.addCallback(lambda ign: self.check_config(0, abs_local_dir_u)) - d.addCallback(lambda ign: self.do_leave(0)) - - d.addCallback(lambda ign: self.do_join(0, unicode(local_dir), self.invite_code)) - def get_caps(ign): - self.collective_dircap, self.upload_dircap = self.get_caps_from_files(0) - d.addCallback(get_caps) - d.addCallback(lambda ign: self.check_joined_config(0, self.upload_dircap)) - d.addCallback(lambda ign: self.check_config(0, abs_local_dir_u)) - - return d - - def test_join_failures(self): - self.basedir = "cli/MagicFolder/create-join-failures" - os.makedirs(self.basedir) - self.set_up_grid(oneshare=True) - local_dir = os.path.join(self.basedir, "magic") - os.mkdir(local_dir) - abs_local_dir_u = abspath_expanduser_unicode(unicode(local_dir), long_path=False) - - self.invite_code = None - d = self.do_create_magic_folder(0) - d.addCallback(lambda ign: self.do_invite(0, self.alice_nickname)) - def get_invite_code_and_join(args): - (rc, stdout, stderr) = args - self.failUnlessEqual(rc, 0) - self.invite_code = stdout.strip() - return self.do_join(0, unicode(local_dir), self.invite_code) - d.addCallback(get_invite_code_and_join) - def get_caps(ign): - self.collective_dircap, self.upload_dircap = self.get_caps_from_files(0) - d.addCallback(get_caps) - d.addCallback(lambda ign: self.check_joined_config(0, self.upload_dircap)) - d.addCallback(lambda ign: self.check_config(0, abs_local_dir_u)) - - def check_success(result): - (rc, out, err) = result - self.failUnlessEqual(rc, 0, out + err) - def check_failure(result): - (rc, out, err) = result - self.failIfEqual(rc, 0) - - def leave(ign): - return self.do_cli("magic-folder", "leave", client_num=0) - d.addCallback(leave) - d.addCallback(check_success) - - magic_folder_db_file = os.path.join(self.get_clientdir(i=0), u"private", u"magicfolder_default.sqlite") - - def check_join_if_file(my_file): - fileutil.write(my_file, "my file data") - d2 = self.do_cli("magic-folder", "join", self.invite_code, local_dir, client_num=0) - d2.addCallback(check_failure) - return d2 - - for my_file in [magic_folder_db_file]: - d.addCallback(lambda ign, my_file: check_join_if_file(my_file), my_file) - d.addCallback(leave) - # we didn't successfully join, so leaving should be an error - d.addCallback(check_failure) - - return d - -class CreateErrors(unittest.TestCase): - def test_poll_interval(self): - e = self.assertRaises(usage.UsageError, parse_cli, - "magic-folder", "create", "--poll-interval=frog", - "alias:") - self.assertEqual(str(e), "--poll-interval must be a positive integer") - - e = self.assertRaises(usage.UsageError, parse_cli, - "magic-folder", "create", "--poll-interval=-4", - "alias:") - self.assertEqual(str(e), "--poll-interval must be a positive integer") - - def test_alias(self): - e = self.assertRaises(usage.UsageError, parse_cli, - "magic-folder", "create", "no-colon") - self.assertEqual(str(e), "An alias must end with a ':' character.") - - def test_nickname(self): - e = self.assertRaises(usage.UsageError, parse_cli, - "magic-folder", "create", "alias:", "nickname") - self.assertEqual(str(e), "If NICKNAME is specified then LOCAL_DIR must also be specified.") - -class InviteErrors(unittest.TestCase): - def test_alias(self): - e = self.assertRaises(usage.UsageError, parse_cli, - "magic-folder", "invite", "no-colon") - self.assertEqual(str(e), "An alias must end with a ':' character.") - -class JoinErrors(unittest.TestCase): - def test_poll_interval(self): - e = self.assertRaises(usage.UsageError, parse_cli, - "magic-folder", "join", "--poll-interval=frog", - "code", "localdir") - self.assertEqual(str(e), "--poll-interval must be a positive integer") - - e = self.assertRaises(usage.UsageError, parse_cli, - "magic-folder", "join", "--poll-interval=-2", - "code", "localdir") - self.assertEqual(str(e), "--poll-interval must be a positive integer") diff --git a/src/allmydata/test/test_client.py b/src/allmydata/test/test_client.py index 824db82fe..85d508093 100644 --- a/src/allmydata/test/test_client.py +++ b/src/allmydata/test/test_client.py @@ -37,7 +37,6 @@ from testtools.twistedsupport import ( ) import allmydata -import allmydata.frontends.magic_folder import allmydata.util.log from allmydata.node import OldConfigError, OldConfigOptionError, UnescapedHashError, _Config, create_node_dir @@ -658,104 +657,6 @@ class Basic(testutil.ReallyEqualMixin, testutil.NonASCIIPathMixin, unittest.Test yield _check("helper.furl = None", None) yield _check("helper.furl = pb://blah\n", "pb://blah") - @defer.inlineCallbacks - def test_create_magic_folder_service(self): - """ - providing magic-folder options actually creates a MagicFolder service - """ - boom = False - class Boom(Exception): - pass - - class MockMagicFolder(allmydata.frontends.magic_folder.MagicFolder): - name = 'magic-folder' - - def __init__(self, client, upload_dircap, collective_dircap, local_path_u, dbfile, umask, name, - inotify=None, uploader_delay=1.0, clock=None, downloader_delay=3): - if boom: - raise Boom() - - service.MultiService.__init__(self) - self.client = client - self._umask = umask - self.upload_dircap = upload_dircap - self.collective_dircap = collective_dircap - self.local_dir = local_path_u - self.dbfile = dbfile - self.inotify = inotify - - def startService(self): - self.running = True - - def stopService(self): - self.running = False - - def ready(self): - pass - - self.patch(allmydata.frontends.magic_folder, 'MagicFolder', MockMagicFolder) - - upload_dircap = "URI:DIR2:blah" - local_dir_u = self.unicode_or_fallback(u"loc\u0101l_dir", u"local_dir") - local_dir_utf8 = local_dir_u.encode('utf-8') - config = (BASECONFIG + - "[storage]\n" + - "enabled = false\n" + - "[magic_folder]\n" + - "enabled = true\n") - - basedir1 = "test_client.Basic.test_create_magic_folder_service1" - os.mkdir(basedir1) - os.mkdir(local_dir_u) - - # which config-entry should be missing? - fileutil.write(os.path.join(basedir1, "tahoe.cfg"), - config + "local.directory = " + local_dir_utf8 + "\n") - with self.assertRaises(IOError): - yield client.create_client(basedir1) - - # local.directory entry missing .. but that won't be an error - # now, it'll just assume there are not magic folders - # .. hrm...should we make that an error (if enabled=true but - # there's not yaml AND no local.directory?) - fileutil.write(os.path.join(basedir1, "tahoe.cfg"), config) - fileutil.write(os.path.join(basedir1, "private", "magic_folder_dircap"), "URI:DIR2:blah") - fileutil.write(os.path.join(basedir1, "private", "collective_dircap"), "URI:DIR2:meow") - - fileutil.write(os.path.join(basedir1, "tahoe.cfg"), - config.replace("[magic_folder]\n", "[drop_upload]\n")) - - with self.assertRaises(OldConfigOptionError): - yield client.create_client(basedir1) - - fileutil.write(os.path.join(basedir1, "tahoe.cfg"), - config + "local.directory = " + local_dir_utf8 + "\n") - c1 = yield client.create_client(basedir1) - magicfolder = c1.getServiceNamed('magic-folder') - self.failUnless(isinstance(magicfolder, MockMagicFolder), magicfolder) - self.failUnlessReallyEqual(magicfolder.client, c1) - self.failUnlessReallyEqual(magicfolder.upload_dircap, upload_dircap) - self.failUnlessReallyEqual(os.path.basename(magicfolder.local_dir), local_dir_u) - self.failUnless(magicfolder.inotify is None, magicfolder.inotify) - # It doesn't start until the client starts. - self.assertFalse(magicfolder.running) - - # See above. - boom = True - - basedir2 = "test_client.Basic.test_create_magic_folder_service2" - os.mkdir(basedir2) - os.mkdir(os.path.join(basedir2, "private")) - fileutil.write(os.path.join(basedir2, "tahoe.cfg"), - BASECONFIG + - "[magic_folder]\n" + - "enabled = true\n" + - "local.directory = " + local_dir_utf8 + "\n") - fileutil.write(os.path.join(basedir2, "private", "magic_folder_dircap"), "URI:DIR2:blah") - fileutil.write(os.path.join(basedir2, "private", "collective_dircap"), "URI:DIR2:meow") - with self.assertRaises(Boom): - yield client.create_client(basedir2) - def flush_but_dont_ignore(res): d = flushEventualQueue() diff --git a/src/allmydata/test/test_inotify.py b/src/allmydata/test/test_inotify.py deleted file mode 100644 index 9f618a34a..000000000 --- a/src/allmydata/test/test_inotify.py +++ /dev/null @@ -1,171 +0,0 @@ -# Copyright (c) Twisted Matrix Laboratories. -# See LICENSE for details. - -""" -Tests for the inotify-alike implementation L{allmydata.watchdog}. -""" - -# Note: See https://twistedmatrix.com/trac/ticket/8915 for a proposal -# to avoid all of this duplicated code from Twisted. - -from twisted.internet import defer, reactor -from twisted.python import filepath, runtime - -from allmydata.frontends.magic_folder import get_inotify_module -from .common import ( - AsyncTestCase, - skipIf, -) -inotify = get_inotify_module() - - -@skipIf(runtime.platformType == "win32", "inotify does not yet work on windows") -class INotifyTests(AsyncTestCase): - """ - Define all the tests for the basic functionality exposed by - L{inotify.INotify}. - """ - def setUp(self): - self.dirname = filepath.FilePath(self.mktemp()) - self.dirname.createDirectory() - self.inotify = inotify.INotify() - self.inotify.startReading() - self.addCleanup(self.inotify.stopReading) - return super(INotifyTests, self).setUp() - - - def _notificationTest(self, mask, operation, expectedPath=None): - """ - Test notification from some filesystem operation. - - @param mask: The event mask to use when setting up the watch. - - @param operation: A function which will be called with the - name of a file in the watched directory and which should - trigger the event. - - @param expectedPath: Optionally, the name of the path which is - expected to come back in the notification event; this will - also be passed to C{operation} (primarily useful when the - operation is being done to the directory itself, not a - file in it). - - @return: A L{Deferred} which fires successfully when the - expected event has been received or fails otherwise. - """ - if expectedPath is None: - expectedPath = self.dirname.child("foo.bar") - notified = defer.Deferred() - def cbNotified(result): - (watch, filename, events) = result - self.assertEqual(filename.asBytesMode(), expectedPath.asBytesMode()) - self.assertTrue(events & mask) - self.inotify.ignore(self.dirname) - notified.addCallback(cbNotified) - - def notify_event(*args): - notified.callback(args) - self.inotify.watch( - self.dirname, mask=mask, - callbacks=[notify_event]) - operation(expectedPath) - return notified - - - def test_modify(self): - """ - Writing to a file in a monitored directory sends an - C{inotify.IN_MODIFY} event to the callback. - """ - def operation(path): - with path.open("w") as fObj: - fObj.write(b'foo') - - return self._notificationTest(inotify.IN_MODIFY, operation) - - - def test_attrib(self): - """ - Changing the metadata of a file in a monitored directory - sends an C{inotify.IN_ATTRIB} event to the callback. - """ - def operation(path): - # Create the file. - path.touch() - # Modify the file's attributes. - path.touch() - - return self._notificationTest(inotify.IN_ATTRIB, operation) - - - def test_closeWrite(self): - """ - Closing a file which was open for writing in a monitored - directory sends an C{inotify.IN_CLOSE_WRITE} event to the - callback. - """ - def operation(path): - path.open("w").close() - - return self._notificationTest(inotify.IN_CLOSE_WRITE, operation) - - - def test_delete(self): - """ - Deleting a file in a monitored directory sends an - C{inotify.IN_DELETE} event to the callback. - """ - expectedPath = self.dirname.child("foo.bar") - expectedPath.touch() - notified = defer.Deferred() - def cbNotified(result): - (watch, filename, events) = result - self.assertEqual(filename.asBytesMode(), expectedPath.asBytesMode()) - self.assertTrue(events & inotify.IN_DELETE) - notified.addCallback(cbNotified) - self.inotify.watch( - self.dirname, mask=inotify.IN_DELETE, - callbacks=[lambda *args: notified.callback(args)]) - expectedPath.remove() - return notified - - - def test_humanReadableMask(self): - """ - L{inotify.humanReadableMask} translates all the possible event masks to a - human readable string. - """ - for mask, value in inotify._FLAG_TO_HUMAN: - self.assertEqual(inotify.humanReadableMask(mask)[0], value) - - checkMask = ( - inotify.IN_CLOSE_WRITE | inotify.IN_ACCESS | inotify.IN_OPEN) - self.assertEqual( - set(inotify.humanReadableMask(checkMask)), - set(['close_write', 'access', 'open'])) - - - def test_noAutoAddSubdirectory(self): - """ - L{inotify.INotify.watch} with autoAdd==False will stop inotify - from watching subdirectories created under the watched one. - """ - def _callback(wp, fp, mask): - # We are notified before we actually process new - # directories, so we need to defer this check. - def _(): - try: - self.assertFalse(self.inotify._isWatched(subdir)) - d.callback(None) - except Exception: - d.errback() - reactor.callLater(0, _) - - checkMask = inotify.IN_ISDIR | inotify.IN_CREATE - self.inotify.watch( - self.dirname, mask=checkMask, autoAdd=False, - callbacks=[_callback]) - subdir = self.dirname.child('test') - d = defer.Deferred() - subdir.createDirectory() - return d diff --git a/src/allmydata/test/test_magic_folder.py b/src/allmydata/test/test_magic_folder.py deleted file mode 100644 index c39027eea..000000000 --- a/src/allmydata/test/test_magic_folder.py +++ /dev/null @@ -1,2425 +0,0 @@ -from __future__ import print_function - -import os, sys, time -import stat, shutil, json -import mock -from os.path import join, exists, isdir -from errno import ENOENT - -from twisted.internet import defer, task, reactor -from twisted.python.runtime import platform -from twisted.python.filepath import FilePath - -from testtools.matchers import ( - Not, - Is, - ContainsDict, - Equals, -) - -from eliot import ( - Message, - start_action, - log_call, -) -from eliot.twisted import DeferredContext - -from allmydata.interfaces import ( - IDirectoryNode, - NoSharesError, -) -from allmydata.util.assertutil import precondition - -from allmydata.util import fake_inotify, fileutil, configutil, yamlutil -from allmydata.util.encodingutil import get_filesystem_encoding, to_filepath -from allmydata.util.consumer import download_to_data -from allmydata.test.no_network import GridTestMixin -from allmydata.test.common_util import ReallyEqualMixin -from .common import ( - ShouldFailMixin, - SyncTestCase, - AsyncTestCase, - skipIf, -) -from .cli.test_magic_folder import MagicFolderCLITestMixin - -from allmydata.frontends import magic_folder -from allmydata.frontends.magic_folder import ( - MagicFolder, WriteFileMixin, - ConfigurationError, -) -from allmydata import magicfolderdb, magicpath -from allmydata.util.fileutil import get_pathinfo -from allmydata.util.fileutil import abspath_expanduser_unicode -from allmydata.immutable.upload import Data -from allmydata.mutable.common import ( - UnrecoverableFileError, -) - -from ..util.eliotutil import ( - inline_callbacks, - log_call_deferred, -) - -_debug = False - -try: - magic_folder.get_inotify_module() -except NotImplementedError: - support_missing = True - support_message = ( - "Magic Folder support can only be tested for-real on an OS that " - "supports inotify or equivalent." - ) -else: - support_missing = False - support_message = None - -if platform.isMacOSX(): - def modified_mtime_barrier(path): - """ - macOS filesystem (HFS+) has one second resolution on filesystem - modification time metadata. Make sure that code running after this - function which modifies the file will produce a changed mtime on that - file. - """ - try: - mtime = path.getModificationTime() - except OSError as e: - if e.errno == ENOENT: - # If the file does not exist yet, there is no current mtime - # value that might match a future mtime value. We have - # nothing to do. - return - # Propagate any other errors as we don't know what's going on. - raise - if int(time.time()) == int(mtime): - # The current time matches the file's modification time, to the - # resolution of the filesystem metadata. Therefore, change the - # current time. - time.sleep(1) -else: - def modified_mtime_barrier(path): - """ - non-macOS platforms have sufficiently high-resolution file modification - time metadata that nothing in particular is required to ensure a - modified mtime as a result of a future write. - """ - - -class NewConfigUtilTests(SyncTestCase): - - def setUp(self): - # some tests look at the umask of created directories or files - # so we set an explicit one - old_umask = os.umask(0o022) - self.addCleanup(lambda: os.umask(old_umask)) - self.basedir = abspath_expanduser_unicode(unicode(self.mktemp())) - os.mkdir(self.basedir) - self.local_dir = abspath_expanduser_unicode(unicode(self.mktemp())) - os.mkdir(self.local_dir) - privdir = join(self.basedir, "private") - os.mkdir(privdir) - - self.poll_interval = 60 - self.collective_dircap = u"a" * 32 - self.magic_folder_dircap = u"b" * 32 - - self.folders = { - u"default": { - u"directory": self.local_dir, - u"upload_dircap": self.magic_folder_dircap, - u"collective_dircap": self.collective_dircap, - u"poll_interval": self.poll_interval, - } - } - - # we need a bit of tahoe.cfg - self.write_tahoe_config( - self.basedir, - u"[magic_folder]\n" - u"enabled = True\n", - ) - # ..and the yaml - self.write_magic_folder_config(self.basedir, self.folders) - return super(NewConfigUtilTests, self).setUp() - - def write_tahoe_config(self, basedir, tahoe_config): - with open(join(basedir, u"tahoe.cfg"), "w") as f: - f.write(tahoe_config) - - def write_magic_folder_config(self, basedir, folder_configuration): - yaml_fname = join(basedir, u"private", u"magic_folders.yaml") - with open(yaml_fname, "w") as f: - f.write(yamlutil.safe_dump({u"magic-folders": folder_configuration})) - - def test_load(self): - folders = magic_folder.load_magic_folders(self.basedir) - self.assertEqual(['default'], list(folders.keys())) - self.assertEqual(folders['default'][u'umask'], 0o077) - - def test_load_makes_directory(self): - """ - If the *directory* does not exist then it is created by - ``load_magic_folders``. - """ - os.rmdir(self.local_dir) - # Just pick some arbitrary bits. - # rwxr-xr-- - perm = stat.S_IRWXU | stat.S_IRGRP | stat.S_IXGRP | stat.S_IROTH - self.folders[u"default"][u"umask"] = (0o777 & ~perm) - self.write_magic_folder_config(self.basedir, self.folders) - - magic_folder.load_magic_folders(self.basedir) - - # It is created. - self.assertTrue( - isdir(self.local_dir), - "magic-folder local directory {} was not created".format( - self.local_dir, - ), - ) - # It has permissions determined by the configured umask. - if sys.platform != "win32": - self.assertEqual( - perm, - stat.S_IMODE(os.stat(self.local_dir).st_mode), - ) - else: - # Do directories even have permissions on Windows? - print("Not asserting directory-creation mode on windows") - - def test_directory_collision(self): - """ - If a non-directory already exists at the magic folder's configured local - directory path, ``load_magic_folders`` raises an exception. - """ - os.rmdir(self.local_dir) - open(self.local_dir, "w").close() - - with self.assertRaises(ConfigurationError) as ctx: - magic_folder.load_magic_folders(self.basedir) - self.assertIn( - "exists and is not a directory", - str(ctx.exception), - ) - - def test_directory_creation_error(self): - """ - If a directory at the magic folder's configured local directory path - cannot be created for some other reason, ``load_magic_folders`` raises - an exception. - """ - os.rmdir(self.local_dir) - open(self.local_dir, "w").close() - self.folders[u"default"][u"directory"] = self.local_dir + "/foo" - self.write_magic_folder_config(self.basedir, self.folders) - - with self.assertRaises(ConfigurationError) as ctx: - magic_folder.load_magic_folders(self.basedir) - self.assertIn( - "could not be created", - str(ctx.exception), - ) - - def test_both_styles_of_config(self): - os.unlink(join(self.basedir, u"private", u"magic_folders.yaml")) - with self.assertRaises(Exception) as ctx: - magic_folder.load_magic_folders(self.basedir) - self.assertIn( - "[magic_folder] is enabled but has no YAML file and no 'local.directory' option", - str(ctx.exception) - ) - - def test_wrong_obj(self): - yaml_fname = join(self.basedir, u"private", u"magic_folders.yaml") - with open(yaml_fname, "w") as f: - f.write('----\n') - - with self.assertRaises(Exception) as ctx: - magic_folder.load_magic_folders(self.basedir) - self.assertIn( - "should contain a dict", - str(ctx.exception) - ) - - def test_no_magic_folders(self): - yaml_fname = join(self.basedir, u"private", u"magic_folders.yaml") - with open(yaml_fname, "w") as f: - f.write('') - - with self.assertRaises(Exception) as ctx: - magic_folder.load_magic_folders(self.basedir) - self.assertIn( - "should contain a dict", - str(ctx.exception) - ) - - def test_magic_folders_not_dict(self): - yaml_fname = join(self.basedir, u"private", u"magic_folders.yaml") - with open(yaml_fname, "w") as f: - f.write('magic-folders: "foo"\n') - - with self.assertRaises(Exception) as ctx: - magic_folder.load_magic_folders(self.basedir) - self.assertIn( - "should be a dict", - str(ctx.exception) - ) - self.assertIn( - "'magic-folders'", - str(ctx.exception) - ) - - def test_wrong_umask_obj(self): - """ - If a umask is given for a magic-folder that is not an integer, an - exception is raised. - """ - self.folders[u"default"][u"umask"] = "0077" - yaml_fname = join(self.basedir, u"private", u"magic_folders.yaml") - with open(yaml_fname, "w") as f: - f.write(yamlutil.safe_dump({u"magic-folders": self.folders})) - - with self.assertRaises(Exception) as ctx: - magic_folder.load_magic_folders(self.basedir) - self.assertIn( - "umask must be an integer", - str(ctx.exception) - ) - - def test_wrong_sub_obj(self): - yaml_fname = join(self.basedir, u"private", u"magic_folders.yaml") - with open(yaml_fname, "w") as f: - f.write("magic-folders:\n default: foo\n") - - with self.assertRaises(Exception) as ctx: - magic_folder.load_magic_folders(self.basedir) - self.assertIn( - "must itself be a dict", - str(ctx.exception) - ) - - def test_missing_interval(self): - del self.folders[u"default"]["poll_interval"] - yaml_fname = join(self.basedir, u"private", u"magic_folders.yaml") - with open(yaml_fname, "w") as f: - f.write(yamlutil.safe_dump({u"magic-folders": self.folders})) - - with self.assertRaises(Exception) as ctx: - magic_folder.load_magic_folders(self.basedir) - self.assertIn( - "missing 'poll_interval'", - str(ctx.exception) - ) - - -class LegacyConfigUtilTests(SyncTestCase): - - def setUp(self): - # create a valid 'old style' magic-folder configuration - self.basedir = abspath_expanduser_unicode(unicode(self.mktemp())) - os.mkdir(self.basedir) - self.local_dir = abspath_expanduser_unicode(unicode(self.mktemp())) - os.mkdir(self.local_dir) - privdir = join(self.basedir, "private") - os.mkdir(privdir) - - # state tests might need to know - self.poll_interval = 60 - self.collective_dircap = u"a" * 32 - self.magic_folder_dircap = u"b" * 32 - - # write fake config structure - with open(join(self.basedir, u"tahoe.cfg"), "w") as f: - f.write( - u"[magic_folder]\n" - u"enabled = True\n" - u"local.directory = {}\n" - u"poll_interval = {}\n".format( - self.local_dir, - self.poll_interval, - ) - ) - with open(join(privdir, "collective_dircap"), "w") as f: - f.write("{}\n".format(self.collective_dircap)) - with open(join(privdir, "magic_folder_dircap"), "w") as f: - f.write("{}\n".format(self.magic_folder_dircap)) - with open(join(privdir, "magicfolderdb.sqlite"), "w") as f: - pass - return super(LegacyConfigUtilTests, self).setUp() - - def test_load_legacy_no_dir(self): - expected = self.local_dir + 'foo' - with open(join(self.basedir, u"tahoe.cfg"), "w") as f: - f.write( - u"[magic_folder]\n" - u"enabled = True\n" - u"local.directory = {}\n" - u"poll_interval = {}\n".format( - expected, - self.poll_interval, - ) - ) - - magic_folder.load_magic_folders(self.basedir) - - self.assertTrue( - isdir(expected), - "magic-folder local directory {} was not created".format( - expected, - ), - ) - - def test_load_legacy_not_a_dir(self): - with open(join(self.basedir, u"tahoe.cfg"), "w") as f: - f.write( - u"[magic_folder]\n" - u"enabled = True\n" - u"local.directory = {}\n" - u"poll_interval = {}\n".format( - self.local_dir + "foo", - self.poll_interval, - ) - ) - with open(self.local_dir + "foo", "w") as f: - f.write("not a directory") - - with self.assertRaises(ConfigurationError) as ctx: - magic_folder.load_magic_folders(self.basedir) - self.assertIn( - "is not a directory", - str(ctx.exception) - ) - - def test_load_legacy_and_new(self): - with open(join(self.basedir, u"private", u"magic_folders.yaml"), "w") as f: - f.write("---") - - with self.assertRaises(Exception) as ctx: - magic_folder.load_magic_folders(self.basedir) - self.assertIn( - "both old-style configuration and new-style", - str(ctx.exception) - ) - - def test_upgrade(self): - # test data is created in setUp; upgrade config - magic_folder._upgrade_magic_folder_config(self.basedir) - - # ensure old stuff is gone - self.assertFalse( - exists(join(self.basedir, "private", "collective_dircap")) - ) - self.assertFalse( - exists(join(self.basedir, "private", "magic_folder_dircap")) - ) - self.assertFalse( - exists(join(self.basedir, "private", "magicfolderdb.sqlite")) - ) - - # ensure we've got the new stuff - self.assertTrue( - exists(join(self.basedir, "private", "magicfolder_default.sqlite")) - ) - # what about config? - config = configutil.get_config(join(self.basedir, u"tahoe.cfg")) - self.assertFalse(config.has_option("magic_folder", "local.directory")) - - def test_load_legacy(self): - folders = magic_folder.load_magic_folders(self.basedir) - - self.assertEqual(['default'], list(folders.keys())) - self.assertTrue( - exists(join(self.basedir, "private", "collective_dircap")) - ) - self.assertTrue( - exists(join(self.basedir, "private", "magic_folder_dircap")) - ) - self.assertTrue( - exists(join(self.basedir, "private", "magicfolderdb.sqlite")) - ) - - def test_load_legacy_upgrade(self): - magic_folder.maybe_upgrade_magic_folders(self.basedir) - folders = magic_folder.load_magic_folders(self.basedir) - - self.assertEqual(['default'], list(folders.keys())) - # 'legacy' files should be gone - self.assertFalse( - exists(join(self.basedir, "private", "collective_dircap")) - ) - self.assertFalse( - exists(join(self.basedir, "private", "magic_folder_dircap")) - ) - self.assertFalse( - exists(join(self.basedir, "private", "magicfolderdb.sqlite")) - ) - - - -class MagicFolderDbTests(SyncTestCase): - - def setUp(self): - self.temp = abspath_expanduser_unicode(unicode(self.mktemp())) - os.mkdir(self.temp) - self.addCleanup(lambda: shutil.rmtree(self.temp)) - dbfile = abspath_expanduser_unicode(u"testdb.sqlite", base=self.temp) - self.db = magicfolderdb.get_magicfolderdb(dbfile, create_version=(magicfolderdb.SCHEMA_v1, 1)) - self.addCleanup(lambda: self.db.close()) - self.failUnless(self.db, "unable to create magicfolderdb from %r" % (dbfile,)) - self.failUnlessEqual(self.db.VERSION, 1) - return super(MagicFolderDbTests, self).setUp() - - def test_create(self): - self.db.did_upload_version( - relpath_u=u'fake_path', - version=0, - last_uploaded_uri=None, - last_downloaded_uri='URI:foo', - last_downloaded_timestamp=1234.5, - pathinfo=get_pathinfo(self.temp), # a directory, but should be fine for test - ) - - entry = self.db.get_db_entry(u'fake_path') - self.assertTrue(entry is not None) - self.assertEqual(entry.last_downloaded_uri, 'URI:foo') - - def test_update(self): - self.db.did_upload_version( - relpath_u=u'fake_path', - version=0, - last_uploaded_uri=None, - last_downloaded_uri='URI:foo', - last_downloaded_timestamp=1234.5, - pathinfo=get_pathinfo(self.temp), # a directory, but should be fine for test - ) - self.db.did_upload_version( - relpath_u=u'fake_path', - version=1, - last_uploaded_uri=None, - last_downloaded_uri='URI:bar', - last_downloaded_timestamp=1234.5, - pathinfo=get_pathinfo(self.temp), # a directory, but should be fine for test - ) - - entry = self.db.get_db_entry(u'fake_path') - self.assertTrue(entry is not None) - self.assertEqual(entry.last_downloaded_uri, 'URI:bar') - self.assertEqual(entry.version, 1) - - def test_same_content_different_path(self): - content_uri = 'URI:CHK:27d2yruqwk6zb2w7hkbbfxxbue:ipmszjysmn4vdeaxz7rtxtv3gwv6vrqcg2ktrdmn4oxqqucltxxq:2:4:1052835840' - self.db.did_upload_version( - relpath_u=u'path0', - version=0, - last_uploaded_uri=None, - last_downloaded_uri=content_uri, - last_downloaded_timestamp=1234.5, - pathinfo=get_pathinfo(self.temp), # a directory, but should be fine for test - ) - self.db.did_upload_version( - relpath_u=u'path1', - version=0, - last_uploaded_uri=None, - last_downloaded_uri=content_uri, - last_downloaded_timestamp=1234.5, - pathinfo=get_pathinfo(self.temp), # a directory, but should be fine for test - ) - - entry = self.db.get_db_entry(u'path0') - self.assertTrue(entry is not None) - self.assertEqual(entry.last_downloaded_uri, content_uri) - - entry = self.db.get_db_entry(u'path1') - self.assertTrue(entry is not None) - self.assertEqual(entry.last_downloaded_uri, content_uri) - - def test_get_direct_children(self): - """ - ``get_direct_children`` returns a list of ``PathEntry`` representing each - local file in the database which is a direct child of the given path. - """ - def add_file(relpath_u): - self.db.did_upload_version( - relpath_u=relpath_u, - version=0, - last_uploaded_uri=None, - last_downloaded_uri=None, - last_downloaded_timestamp=1234, - pathinfo=get_pathinfo(self.temp), - ) - paths = [ - u"some_random_file", - u"the_target_directory_is_elsewhere", - u"the_target_directory_is_not_this/", - u"the_target_directory_is_not_this/and_not_in_here", - u"the_target_directory/", - u"the_target_directory/foo", - u"the_target_directory/bar", - u"the_target_directory/baz", - u"the_target_directory/quux/", - u"the_target_directory/quux/exclude_grandchildren", - u"the_target_directory/quux/and_great_grandchildren/", - u"the_target_directory/quux/and_great_grandchildren/foo", - u"the_target_directory_is_over/stuff", - u"please_ignore_this_for_sure", - ] - for relpath_u in paths: - add_file(relpath_u) - - expected_paths = [ - u"the_target_directory/foo", - u"the_target_directory/bar", - u"the_target_directory/baz", - u"the_target_directory/quux/", - ] - - actual_paths = list( - localpath.relpath_u - for localpath - in self.db.get_direct_children(u"the_target_directory") - ) - self.assertEqual(expected_paths, actual_paths) - - -def iterate_downloader(magic): - return magic.downloader._processing_iteration() - - -def iterate_uploader(magic): - return magic.uploader._processing_iteration() - -@inline_callbacks -def iterate(magic): - yield iterate_uploader(magic) - yield iterate_downloader(magic) - - -@inline_callbacks -def notify_when_pending(uploader, filename): - with start_action(action_type=u"notify-when-pending", filename=filename): - relpath = uploader._get_relpath(FilePath(filename)) - while not uploader.is_pending(relpath): - Message.log(message_type=u"not-pending") - yield uploader.set_hook('inotify') - - -class FileOperationsHelper(object): - """ - This abstracts all file operations we might do in magic-folder unit-tests. - - This is so we can correctly wait for inotify events to 'actually' - propagate. For the mock tests this is easy, since we're sending - them sychronously. For the Real tests we have to wait for the - actual inotify thing. - """ - _timeout = 30.0 - - def __init__(self, uploader, inject_events=False): - self._uploader = uploader - self._inotify = fake_inotify # fixme? - self._fake_inotify = inject_events - - @log_call_deferred(action_type=u"fileops:move") - def move(self, from_path_u, to_path_u): - from_fname = from_path_u - to_fname = to_path_u - d = self._uploader.set_hook('inotify') - os.rename(from_fname, to_fname) - - self._maybe_notify(to_fname, self._inotify.IN_MOVED_TO) - # hmm? we weren't faking IN_MOVED_FROM previously .. but seems like we should have been? - # self._uploader._notifier.event(to_filepath(from_fname), self._inotify.IN_MOVED_FROM) - return d.addTimeout(self._timeout, reactor) - - @log_call_deferred(action_type=u"fileops:write") - def write(self, path_u, contents): - fname = path_u - if not os.path.exists(fname): - self._maybe_notify(fname, self._inotify.IN_CREATE) - - d = notify_when_pending(self._uploader, path_u) - - modified_mtime_barrier(FilePath(fname)) - with open(fname, "wb") as f: - f.write(contents) - - self._maybe_notify(fname, self._inotify.IN_CLOSE_WRITE) - return d.addTimeout(self._timeout, reactor) - - @log_call_deferred(action_type=u"fileops:mkdir") - def mkdir(self, path_u): - fname = path_u - d = self._uploader.set_hook('inotify') - os.mkdir(fname) - self._maybe_notify(fname, self._inotify.IN_CREATE | self._inotify.IN_ISDIR) - return d.addTimeout(self._timeout, reactor) - - @log_call_deferred(action_type=u"fileops:delete") - def delete(self, path_u): - fname = path_u - d = self._uploader.set_hook('inotify') - if os.path.isdir(fname): - remove = os.rmdir - else: - remove = os.unlink - remove(fname) - - self._maybe_notify(fname, self._inotify.IN_DELETE) - return d.addTimeout(self._timeout, reactor) - - def _maybe_notify(self, fname, mask): - if self._fake_inotify: - self._uploader._notifier.event(to_filepath(fname), mask) - - -class CheckerMixin(object): - """ - Factored out of one of the many test classes. - - *Ideally* these should just be bare helper methods, but many of - them already depended upon self.* state. One major problem is that - they're using self.magicfolder *but* some of the alice/bob tests - use this, too, and they just do "self.magicfolder = - self.bob_magicfolder" or whatever before calling them, which is - *horrible*. - """ - def _check_mkdir(self, name_u): - return self._check_file(name_u + u"/", "", directory=True) - - @defer.inlineCallbacks - def _check_file(self, name_u, data, temporary=False, directory=False): - precondition(not (temporary and directory), temporary=temporary, directory=directory) - - # print "%r._check_file(%r, %r, temporary=%r, directory=%r)" % (self, name_u, data, temporary, directory) - previously_uploaded = self._get_count('uploader.objects_succeeded') - previously_disappeared = self._get_count('uploader.objects_disappeared') - - path_u = abspath_expanduser_unicode(name_u, base=self.local_dir) - - if directory: - yield self.fileops.mkdir(path_u) - else: - # We don't use FilePath.setContent() here because it creates a temporary file that - # is renamed into place, which causes events that the test is not expecting. - yield self.fileops.write(path_u, data) - yield iterate(self.magicfolder) - if temporary: - yield iterate(self.magicfolder) - yield self.fileops.delete(path_u) - - yield iterate(self.magicfolder) - encoded_name_u = magicpath.path2magic(name_u) - - yield self.failUnlessReallyEqual(self._get_count('uploader.objects_failed'), 0) - if temporary: - yield self.failUnlessReallyEqual(self._get_count('uploader.objects_disappeared'), - previously_disappeared + 1) - else: - yield self.magicfolder.uploader._upload_dirnode.list() - x = yield self.magicfolder.uploader._upload_dirnode.get(encoded_name_u) - actual_data = yield download_to_data(x) - self.failUnlessReallyEqual(actual_data, data) - self.failUnlessReallyEqual(self._get_count('uploader.objects_succeeded'), - previously_uploaded + 1) - - self.failUnlessReallyEqual(self._get_count('uploader.objects_queued'), 0) - - @defer.inlineCallbacks - def _check_version_in_dmd(self, magicfolder, relpath_u, expected_version): - encoded_name_u = magicpath.path2magic(relpath_u) - result = yield magicfolder.downloader._get_collective_latest_file(encoded_name_u) - self.assertIsNot( - result, - None, - "collective_latest_file({}) is None".format(encoded_name_u), - ) - node, metadata = result - self.assertIsNot( - metadata, - None, - "collective_latest_file({}) metadata is None".format(encoded_name_u), - ) - self.failUnlessEqual(metadata['version'], expected_version) - - def _check_version_in_local_db(self, magicfolder, relpath_u, expected_version): - db_entry = magicfolder._db.get_db_entry(relpath_u) - if db_entry is not None: - #print "_check_version_in_local_db: %r has version %s" % (relpath_u, version) - self.failUnlessEqual(db_entry.version, expected_version) - - def _check_file_gone(self, magicfolder, relpath_u): - path = os.path.join(magicfolder.uploader._local_path_u, relpath_u) - self.assertTrue(not os.path.exists(path)) - - def _check_uploader_count(self, name, expected, magic=None): - if magic is None: - magic = self.alice_magicfolder - self.failUnlessReallyEqual( - self._get_count( - 'uploader.'+name, - client=magic._client, - ), - expected, - "Pending: {}\n" - "Deque: {}\n".format(magic.uploader._pending, magic.uploader._deque), - ) - - def _check_downloader_count(self, name, expected, magic=None): - self.failUnlessReallyEqual(self._get_count('downloader.'+name, client=(magic or self.bob_magicfolder)._client), - expected) - - def _get_count(self, name, client=None): - counters = (client or self.get_client()).stats_provider.get_stats()["counters"] - return counters.get('magic_folder.%s' % (name,), 0) - - - -class MagicFolderAliceBobTestMixin(MagicFolderCLITestMixin, ShouldFailMixin, ReallyEqualMixin, CheckerMixin): - inject_inotify = False - - def setUp(self): - MagicFolderCLITestMixin.setUp(self) - temp = self.mktemp() - self.basedir = abspath_expanduser_unicode(temp.decode(get_filesystem_encoding())) - # set_up_grid depends on self.basedir existing - with start_action(action_type=u"set_up_grid"): - self.set_up_grid(num_clients=2, oneshare=True) - - self.alice_clock = task.Clock() - self.bob_clock = task.Clock() - - # this is all just .setup_alice_and_bob(), essentially - self.alice_magicfolder = None - self.bob_magicfolder = None - - self.alice_magic_dir = abspath_expanduser_unicode(u"Alice-magic", base=self.basedir) - self.mkdir_nonascii(self.alice_magic_dir) - self.bob_magic_dir = abspath_expanduser_unicode(u"Bob-magic", base=self.basedir) - self.mkdir_nonascii(self.bob_magic_dir) - - # Alice creates a Magic Folder, invites herself and joins. - d = DeferredContext(self.do_create_magic_folder(0)) - d.addCallback(lambda ign: self.do_invite(0, self.alice_nickname)) - def get_invite_code(result): - self.invite_code = result[1].strip() - d.addCallback(get_invite_code) - d.addCallback(lambda ign: self.do_join(0, self.alice_magic_dir, self.invite_code)) - def get_alice_caps(ign): - self.alice_collective_dircap, self.alice_upload_dircap = self.get_caps_from_files(0) - d.addCallback(get_alice_caps) - d.addCallback(lambda ign: self.check_joined_config(0, self.alice_upload_dircap)) - d.addCallback(lambda ign: self.check_config(0, self.alice_magic_dir)) - def get_Alice_magicfolder(result): - self.alice_magicfolder = self.init_magicfolder(0, self.alice_upload_dircap, - self.alice_collective_dircap, - self.alice_magic_dir, self.alice_clock) - self.alice_fileops = FileOperationsHelper(self.alice_magicfolder.uploader, self.inject_inotify) - d0 = self.alice_magicfolder.uploader.set_hook('iteration') - d1 = self.alice_magicfolder.downloader.set_hook('iteration') - self.alice_clock.advance(self.alice_magicfolder.uploader._pending_delay + 1) - d0.addCallback(lambda ign: d1) - d0.addCallback(lambda ign: result) - return d0 - d.addCallback(get_Alice_magicfolder) - - # Alice invites Bob. Bob joins. - d.addCallback(lambda ign: self.do_invite(0, self.bob_nickname)) - def get_invite_code(result): - self.invite_code = result[1].strip() - d.addCallback(get_invite_code) - d.addCallback(lambda ign: self.do_join(1, self.bob_magic_dir, self.invite_code)) - def get_bob_caps(ign): - self.bob_collective_dircap, self.bob_upload_dircap = self.get_caps_from_files(1) - d.addCallback(get_bob_caps) - d.addCallback(lambda ign: self.check_joined_config(1, self.bob_upload_dircap)) - d.addCallback(lambda ign: self.check_config(1, self.bob_magic_dir)) - def get_Bob_magicfolder(result): - self.bob_magicfolder = self.init_magicfolder(1, self.bob_upload_dircap, - self.bob_collective_dircap, - self.bob_magic_dir, self.bob_clock) - self.bob_fileops = FileOperationsHelper(self.bob_magicfolder.uploader, self.inject_inotify) - d0 = self.bob_magicfolder.uploader.set_hook('iteration') - d1 = self.bob_magicfolder.downloader.set_hook('iteration') - self.bob_clock.advance(self.alice_magicfolder.uploader._pending_delay + 1) - d0.addCallback(lambda ign: d1) - d0.addCallback(lambda ign: result) - return d0 - d.addCallback(get_Bob_magicfolder) - return d.result - - @defer.inlineCallbacks - def tearDown(self): - yield GridTestMixin.tearDown(self) - - for mf in [self.alice_magicfolder, self.bob_magicfolder]: - mf.uploader._clock.advance(mf.uploader._pending_delay + 1) - mf.downloader._clock.advance(mf.downloader._poll_interval + 1) - - @inline_callbacks - def test_alice_delete_bob_restore(self): - alice_fname = os.path.join(self.alice_magic_dir, 'blam') - bob_fname = os.path.join(self.bob_magic_dir, 'blam') - - alice_proc = self.alice_magicfolder.uploader.set_hook('processed') - - with start_action(action_type=u"alice:create"): - yield self.alice_fileops.write(alice_fname, 'contents0\n') - yield iterate(self.alice_magicfolder) # for windows - - with start_action(action_type=u"alice:upload"): - yield iterate_uploader(self.alice_magicfolder) - yield alice_proc - - with start_action(action_type=u"alice:check-upload"): - yield self._check_version_in_dmd(self.alice_magicfolder, u"blam", 0) - yield self._check_version_in_local_db(self.alice_magicfolder, u"blam", 0) - - with start_action(action_type=u"bob:download"): - yield iterate_downloader(self.bob_magicfolder) - - with start_action(action_type=u"alice:recheck-upload"): - yield self._check_version_in_dmd(self.alice_magicfolder, u"blam", 0) - yield self._check_version_in_local_db(self.alice_magicfolder, u"blam", 0) - - with start_action(action_type=u"bob:check-download"): - yield self._check_version_in_dmd(self.bob_magicfolder, u"blam", 0) - yield self._check_version_in_local_db(self.bob_magicfolder, u"blam", 0) - yield self.failUnlessReallyEqual( - self._get_count('downloader.objects_failed', client=self.bob_magicfolder._client), - 0 - ) - yield self.failUnlessReallyEqual( - self._get_count('downloader.objects_downloaded', client=self.bob_magicfolder._client), - 1 - ) - - yield iterate(self.bob_magicfolder) # for windows - - - bob_proc = self.bob_magicfolder.uploader.set_hook('processed') - alice_proc = self.alice_magicfolder.downloader.set_hook('processed') - - with start_action(action_type=u"bob:delete"): - yield self.bob_fileops.delete(bob_fname) - yield iterate(self.bob_magicfolder) # for windows - - with start_action(action_type=u"bob:upload"): - yield iterate_uploader(self.bob_magicfolder) - yield bob_proc - - with start_action(action_type=u"alice:download"): - yield iterate_downloader(self.alice_magicfolder) - yield alice_proc - - # check versions - with start_action(action_type=u"bob:check-upload"): - node, metadata = yield self.alice_magicfolder.downloader._get_collective_latest_file(u'blam') - self.assertTrue(metadata['deleted']) - yield self._check_version_in_dmd(self.bob_magicfolder, u"blam", 1) - yield self._check_version_in_local_db(self.bob_magicfolder, u"blam", 1) - - with start_action(action_type=u"alice:check-download"): - yield self._check_version_in_dmd(self.alice_magicfolder, u"blam", 1) - yield self._check_version_in_local_db(self.alice_magicfolder, u"blam", 1) - - with start_action(action_type=u"alice:mysterious-iterate"): - # not *entirely* sure why we need to iterate Alice for the - # real test here. But, we do. - yield iterate(self.alice_magicfolder) - - # now alice restores it (alice should upload, bob download) - alice_proc = self.alice_magicfolder.uploader.set_hook('processed') - bob_proc = self.bob_magicfolder.downloader.set_hook('processed') - - with start_action(action_type=u"alice:rewrite"): - yield self.alice_fileops.write(alice_fname, 'new contents\n') - yield iterate(self.alice_magicfolder) # for windows - - with start_action(action_type=u"alice:reupload"): - yield iterate_uploader(self.alice_magicfolder) - yield alice_proc - - with start_action(action_type=u"bob:redownload"): - yield iterate_downloader(self.bob_magicfolder) - yield bob_proc - - # check versions - with start_action(action_type=u"bob:recheck-download"): - node, metadata = yield self.alice_magicfolder.downloader._get_collective_latest_file(u'blam') - self.assertTrue('deleted' not in metadata or not metadata['deleted']) - yield self._check_version_in_dmd(self.bob_magicfolder, u"blam", 2) - yield self._check_version_in_local_db(self.bob_magicfolder, u"blam", 2) - - with start_action(action_type=u"alice:final-check-upload"): - yield self._check_version_in_dmd(self.alice_magicfolder, u"blam", 2) - yield self._check_version_in_local_db(self.alice_magicfolder, u"blam", 2) - - @inline_callbacks - def test_alice_sees_bobs_delete_with_error(self): - # alice creates a file, bob deletes it -- and we also arrange - # for Alice's file to have "gone missing" as well. - alice_fname = os.path.join(self.alice_magic_dir, 'blam') - bob_fname = os.path.join(self.bob_magic_dir, 'blam') - - # alice creates a file, bob downloads it - alice_proc = self.alice_magicfolder.uploader.set_hook('processed') - bob_proc = self.bob_magicfolder.downloader.set_hook('processed') - - with start_action(action_type=u"alice:create"): - yield self.alice_fileops.write(alice_fname, 'contents0\n') - yield iterate(self.alice_magicfolder) # for windows - - with start_action(action_type=u"alice:upload"): - yield iterate_uploader(self.alice_magicfolder) - yield alice_proc # alice uploads - - with start_action(action_type=u"bob:download"): - yield iterate_downloader(self.bob_magicfolder) - yield bob_proc # bob downloads - - with start_action(action_type=u"mysterious:iterate"): - yield iterate(self.alice_magicfolder) # for windows - yield iterate(self.bob_magicfolder) # for windows - - # check the state (XXX I had to switch the versions to 0; is that really right? why?) - with start_action(action_type=u"alice:check"): - yield self._check_version_in_dmd(self.alice_magicfolder, u"blam", 0) - yield self._check_version_in_local_db(self.alice_magicfolder, u"blam", 0) - - with start_action(action_type=u"bob:check"): - yield self._check_version_in_dmd(self.bob_magicfolder, u"blam", 0) - yield self._check_version_in_local_db(self.bob_magicfolder, u"blam", 0) - self.failUnlessReallyEqual( - self._get_count('downloader.objects_failed', client=self.bob_magicfolder._client), - 0 - ) - self.failUnlessReallyEqual( - self._get_count('downloader.objects_downloaded', client=self.bob_magicfolder._client), - 1 - ) - - bob_proc = self.bob_magicfolder.uploader.set_hook('processed') - alice_proc = self.alice_magicfolder.downloader.set_hook('processed') - - with start_action(action_type=u"bob:delete"): - yield self.bob_fileops.delete(bob_fname) - - with start_action(action_type=u"alice:delete"): - # just after notifying bob, we also delete alice's, - # covering the 'except' flow in _rename_deleted_file() - yield self.alice_fileops.delete(alice_fname) - - with start_action(action_type=u"bob:upload-delete"): - yield iterate_uploader(self.bob_magicfolder) - yield bob_proc - - with start_action(action_type=u"alice:download-delete"): - yield iterate_downloader(self.alice_magicfolder) - yield alice_proc - - # check versions - with start_action(action_type=u"bob:check"): - node, metadata = yield self.alice_magicfolder.downloader._get_collective_latest_file(u'blam') - self.assertTrue(metadata['deleted']) - yield self._check_version_in_dmd(self.bob_magicfolder, u"blam", 1) - yield self._check_version_in_local_db(self.bob_magicfolder, u"blam", 1) - - with start_action(action_type=u"alice:check"): - yield self._check_version_in_dmd(self.alice_magicfolder, u"blam", 1) - yield self._check_version_in_local_db(self.alice_magicfolder, u"blam", 1) - - @inline_callbacks - def test_alice_create_bob_update(self): - alice_fname = os.path.join(self.alice_magic_dir, 'blam') - bob_fname = os.path.join(self.bob_magic_dir, 'blam') - - # alice creates a file, bob downloads it - yield self.alice_fileops.write(alice_fname, 'contents0\n') - - yield iterate(self.alice_magicfolder) - yield iterate(self.alice_magicfolder) - yield iterate(self.bob_magicfolder) - - # check the state (XXX ditto, had to switch to veresion 0; right?) - yield self._check_version_in_dmd(self.alice_magicfolder, u"blam", 0) - self._check_version_in_local_db(self.alice_magicfolder, u"blam", 0) - yield self._check_version_in_dmd(self.bob_magicfolder, u"blam", 0) - self._check_version_in_local_db(self.bob_magicfolder, u"blam", 0) - self.failUnlessReallyEqual( - self._get_count('downloader.objects_failed', client=self.bob_magicfolder._client), - 0 - ) - self.failUnlessReallyEqual( - self._get_count('downloader.objects_downloaded', client=self.bob_magicfolder._client), - 1 - ) - - yield iterate(self.bob_magicfolder) - # now bob updates it (bob should upload, alice download) - yield self.bob_fileops.write(bob_fname, 'bob wuz here\n') - - yield iterate(self.bob_magicfolder) - yield iterate(self.bob_magicfolder) - yield iterate(self.alice_magicfolder) - - # check the state - yield self._check_version_in_dmd(self.bob_magicfolder, u"blam", 1) - self._check_version_in_local_db(self.bob_magicfolder, u"blam", 1) - yield self._check_version_in_dmd(self.alice_magicfolder, u"blam", 1) - self._check_version_in_local_db(self.alice_magicfolder, u"blam", 1) - - @inline_callbacks - def test_download_retry(self): - alice_fname = os.path.join(self.alice_magic_dir, 'blam') - # bob_fname = os.path.join(self.bob_magic_dir, 'blam') - - # Alice creates a file - yield self.alice_fileops.write(alice_fname, ''.join(['contents-%04d\n' % i for i in range(1024)])) - yield iterate(self.alice_magicfolder) - # check alice created the file - yield self._check_version_in_dmd(self.alice_magicfolder, u"blam", 0) - self._check_version_in_local_db(self.alice_magicfolder, u"blam", 0) - - # now, we ONLY want to do the scan, not a full iteration of - # the process loop. So we do just the scan part "by hand" in - # Bob's downloader - with start_action(action_type=u"test:perform-scan"): - yield self.bob_magicfolder.downloader._perform_scan() - # while we're delving into internals, I guess we might as well - # confirm that we did queue up an item to download - self.assertEqual(1, len(self.bob_magicfolder.downloader._deque)) - - # break all the servers so the download fails. count=1 because we - # only want the download attempted by _process_deque to fail. After - # that, we want it to work again. - for server_id in self.g.get_all_serverids(): - self.g.break_server(server_id, count=1) - - # now let bob try to do the download. Reach in and call - # _process_deque directly because we are already half-way through a - # logical iteration thanks to the _perform_scan call above. - with start_action(action_type=u"test:process-deque"): - yield self.bob_magicfolder.downloader._process_deque() - - self.eliot_logger.flushTracebacks(UnrecoverableFileError) - logged = self.eliot_logger.flushTracebacks(NoSharesError) - self.assertEqual( - 1, - len(logged), - "Got other than expected single NoSharesError: {}".format(logged), - ) - - # ...however Bob shouldn't have downloaded anything - self._check_version_in_local_db(self.bob_magicfolder, u"blam", 0) - # bob should *not* have downloaded anything, as we failed all the servers - self.failUnlessReallyEqual( - self._get_count('downloader.objects_downloaded', client=self.bob_magicfolder._client), - 0 - ) - self.failUnlessReallyEqual( - self._get_count('downloader.objects_failed', client=self.bob_magicfolder._client), - 1 - ) - - with start_action(action_type=u"test:iterate"): - # now we let Bob try again - yield iterate(self.bob_magicfolder) - - # ...and he should have succeeded - self.failUnlessReallyEqual( - self._get_count('downloader.objects_downloaded', client=self.bob_magicfolder._client), - 1 - ) - yield self._check_version_in_dmd(self.bob_magicfolder, u"blam", 0) - - @inline_callbacks - def test_conflict_local_change_fresh(self): - alice_fname = os.path.join(self.alice_magic_dir, 'localchange0') - bob_fname = os.path.join(self.bob_magic_dir, 'localchange0') - - # alice creates a file, bob downloads it - alice_proc = self.alice_magicfolder.uploader.set_hook('processed') - bob_proc = self.bob_magicfolder.downloader.set_hook('processed') - - yield self.alice_fileops.write(alice_fname, 'contents0\n') - yield iterate(self.alice_magicfolder) # for windows - - # before bob downloads, we make a local file for bob by the - # same name - with open(bob_fname, 'w') as f: - f.write("not the right stuff") - - yield iterate_uploader(self.alice_magicfolder) - yield alice_proc # alice uploads - - yield iterate_downloader(self.bob_magicfolder) - yield bob_proc # bob downloads - - # ...so now bob should produce a conflict - self.assertTrue(os.path.exists(bob_fname + '.conflict')) - - @inline_callbacks - def test_conflict_local_change_existing(self): - alice_fname = os.path.join(self.alice_magic_dir, 'localchange1') - bob_fname = os.path.join(self.bob_magic_dir, 'localchange1') - - alice_proc = self.alice_magicfolder.uploader.set_hook('processed') - bob_proc = self.bob_magicfolder.downloader.set_hook('processed') - - with start_action(action_type=u"alice:create"): - yield self.alice_fileops.write(alice_fname, 'contents0\n') - yield iterate(self.alice_magicfolder) # for windows - - with start_action(action_type=u"alice:upload"): - yield iterate_uploader(self.alice_magicfolder) - yield alice_proc # alice uploads - self.assertEqual( - 1, - self._get_count( - 'uploader.files_uploaded', - client=self.alice_magicfolder._client, - ), - ) - - with start_action(action_type=u"bob:download"): - yield iterate_downloader(self.bob_magicfolder) - yield bob_proc # bob downloads - self.assertEqual( - 1, - self._get_count( - 'downloader.objects_downloaded', - client=self.bob_magicfolder._client, - ), - ) - - alice_proc = self.alice_magicfolder.uploader.set_hook('processed') - bob_proc = self.bob_magicfolder.downloader.set_hook('processed') - - with start_action(action_type=u"alice:rewrite"): - yield self.alice_fileops.write(alice_fname, 'contents1\n') - yield iterate(self.alice_magicfolder) # for windows - - with start_action(action_type=u"bob:rewrite"): - # before bob downloads, make a local change - with open(bob_fname, "w") as f: - f.write("bob's local change") - - with start_action(action_type=u"alice:reupload"): - yield iterate_uploader(self.alice_magicfolder) - yield alice_proc # alice uploads - self.assertEqual( - 2, - self._get_count( - 'uploader.files_uploaded', - client=self.alice_magicfolder._client, - ), - ) - - with start_action(action_type=u"bob:redownload-and-conflict"): - yield iterate_downloader(self.bob_magicfolder) - yield bob_proc # bob downloads - - self.assertEqual( - 2, - self._get_count( - 'downloader.objects_downloaded', - client=self.bob_magicfolder._client, - ), - ) - self.assertEqual( - 1, - self._get_count( - 'downloader.objects_conflicted', - client=self.bob_magicfolder._client, - ), - ) - - # ...so now bob should produce a conflict - self.assertTrue(os.path.exists(bob_fname + '.conflict')) - - @inline_callbacks - def test_alice_delete_and_restore(self): - alice_fname = os.path.join(self.alice_magic_dir, 'blam') - bob_fname = os.path.join(self.bob_magic_dir, 'blam') - - # alice creates a file, bob downloads it - alice_proc = self.alice_magicfolder.uploader.set_hook('processed') - bob_proc = self.bob_magicfolder.downloader.set_hook('processed') - - with start_action(action_type=u"alice:create"): - yield self.alice_fileops.write(alice_fname, 'contents0\n') - yield iterate(self.alice_magicfolder) # for windows - - with start_action(action_type=u"alice:upload"): - yield iterate_uploader(self.alice_magicfolder) - yield alice_proc # alice uploads - - with start_action(action_type=u"bob:download"): - yield iterate_downloader(self.bob_magicfolder) - yield bob_proc # bob downloads - - with start_action(action_type=u"alice:check"): - yield self._check_version_in_dmd(self.alice_magicfolder, u"blam", 0) - yield self._check_version_in_local_db(self.alice_magicfolder, u"blam", 0) - - with start_action(action_type=u"bob:check"): - yield self._check_version_in_dmd(self.bob_magicfolder, u"blam", 0) - yield self._check_version_in_local_db(self.bob_magicfolder, u"blam", 0) - yield self.failUnlessReallyEqual( - self._get_count('downloader.objects_failed', client=self.bob_magicfolder._client), - 0 - ) - yield self.failUnlessReallyEqual( - self._get_count('downloader.objects_downloaded', client=self.bob_magicfolder._client), - 1 - ) - self.failUnless(os.path.exists(bob_fname)) - self.failUnless(not os.path.exists(bob_fname + '.backup')) - self.failUnless(not os.path.exists(bob_fname + '.conflict')) - - alice_proc = self.alice_magicfolder.uploader.set_hook('processed') - bob_proc = self.bob_magicfolder.downloader.set_hook('processed') - - with start_action(action_type=u"alice:delete"): - yield self.alice_fileops.delete(alice_fname) - yield iterate_uploader(self.alice_magicfolder) - yield alice_proc - - with start_action(action_type=u"bob:redownload"): - yield iterate_downloader(self.bob_magicfolder) - yield bob_proc - - with start_action(action_type=u"bob:recheck"): - yield self._check_version_in_dmd(self.bob_magicfolder, u"blam", 1) - yield self._check_version_in_local_db(self.bob_magicfolder, u"blam", 1) - self.assertFalse(os.path.exists(bob_fname)) - self.assertTrue(os.path.exists(bob_fname + '.backup')) - self.assertFalse(os.path.exists(bob_fname + '.conflict')) - - with start_action(action_type=u"alice:recheck"): - yield self._check_version_in_dmd(self.alice_magicfolder, u"blam", 1) - yield self._check_version_in_local_db(self.alice_magicfolder, u"blam", 1) - - with start_action(action_type=u"alice:restore"): - os.unlink(bob_fname + '.backup') - alice_proc = self.alice_magicfolder.uploader.set_hook('processed') - bob_proc = self.bob_magicfolder.downloader.set_hook('processed') - yield self.alice_fileops.write(alice_fname, 'alice wuz here\n') - yield iterate(self.alice_magicfolder) # for windows - - with start_action(action_type=u"alice:reupload"): - yield iterate_uploader(self.alice_magicfolder) - yield iterate_downloader(self.alice_magicfolder) # why? - yield alice_proc - - with start_action(action_type=u"bob:final-redownload"): - yield iterate_downloader(self.bob_magicfolder) - yield iterate_uploader(self.bob_magicfolder) - yield bob_proc - - with start_action(action_type=u"bob:final-check"): - yield self._check_version_in_dmd(self.bob_magicfolder, u"blam", 2) - yield self._check_version_in_local_db(self.bob_magicfolder, u"blam", 2) - self.failUnless(os.path.exists(bob_fname)) - - with start_action(action_type=u"alice:final-check"): - yield self._check_version_in_dmd(self.alice_magicfolder, u"blam", 2) - yield self._check_version_in_local_db(self.alice_magicfolder, u"blam", 2) - - # XXX this should be shortened -- as in, any cases not covered by - # the other tests in here should get their own minimal test-case. - @skipIf(sys.platform == "win32", "Still inotify problems on Windows (FIXME)") - def test_alice_bob(self): - d = DeferredContext(defer.succeed(None)) - - # XXX FIXME just quickly porting this test via aliases -- the - # "real" solution is to break out any relevant test-cases as - # their own (smaller!) tests. - alice_clock = self.alice_magicfolder.uploader._clock - bob_clock = self.bob_magicfolder.uploader._clock - - def _wait_for_Alice(ign, downloaded_d): - if _debug: print("Now waiting for Alice to download\n") - alice_clock.advance(4) - return downloaded_d - - def _wait_for_Bob(ign, downloaded_d): - if _debug: print("Now waiting for Bob to download\n") - bob_clock.advance(4) - return downloaded_d - - def _wait_for(ign, something_to_do, alice=True): - if alice: - downloaded_d = self.bob_magicfolder.downloader.set_hook('processed') - uploaded_d = self.alice_magicfolder.uploader.set_hook('processed') - else: - downloaded_d = self.alice_magicfolder.downloader.set_hook('processed') - uploaded_d = self.bob_magicfolder.uploader.set_hook('processed') - - d = something_to_do() - - def advance(ign): - if alice: - if _debug: print("Waiting for Alice to upload 3\n") - alice_clock.advance(4) - uploaded_d.addCallback(_wait_for_Bob, downloaded_d) - else: - if _debug: print("Waiting for Bob to upload\n") - bob_clock.advance(4) - uploaded_d.addCallback(_wait_for_Alice, downloaded_d) - return uploaded_d - d.addCallback(advance) - return d - - @inline_callbacks - def Alice_to_write_a_file(): - if _debug: print("Alice writes a file\n\n\n\n\n") - self.file_path = abspath_expanduser_unicode(u"file1", base=self.alice_magicfolder.uploader._local_path_u) - yield self.alice_fileops.write(self.file_path, "meow, meow meow. meow? meow meow! meow.") - yield iterate(self.alice_magicfolder) - d.addCallback(_wait_for, Alice_to_write_a_file) - - @log_call_deferred(action_type=u"check_state") - @inline_callbacks - def check_state(ignored): - yield self._check_version_in_dmd(self.alice_magicfolder, u"file1", 0) - self._check_version_in_local_db(self.alice_magicfolder, u"file1", 0) - self._check_uploader_count('objects_failed', 0) - self._check_uploader_count('objects_succeeded', 1) - self._check_uploader_count('files_uploaded', 1) - self._check_uploader_count('objects_queued', 0) - self._check_uploader_count('directories_created', 0) - self._check_uploader_count('objects_conflicted', 0) - self._check_uploader_count('objects_conflicted', 0, magic=self.bob_magicfolder) - - self._check_version_in_local_db(self.bob_magicfolder, u"file1", 0) - self._check_downloader_count('objects_failed', 0) - self._check_downloader_count('objects_downloaded', 1) - self._check_uploader_count('objects_succeeded', 0, magic=self.bob_magicfolder) - self._check_downloader_count('objects_downloaded', 1, magic=self.bob_magicfolder) - d.addCallback(check_state) - - @inline_callbacks - def Alice_to_delete_file(): - if _debug: print("Alice deletes the file!\n\n\n\n") - yield self.alice_fileops.delete(self.file_path) - yield iterate(self.alice_magicfolder) - yield iterate(self.bob_magicfolder) - d.addCallback(_wait_for, Alice_to_delete_file) - - @inline_callbacks - def notify_bob_moved(ign): - # WARNING: this is just directly notifying for the mock - # tests, because in the Real* tests the .backup file will - # me moved into place (from the original) - p = abspath_expanduser_unicode(u"file1", base=self.bob_magicfolder.uploader._local_path_u) - if self.bob_fileops._fake_inotify: - self.bob_magicfolder.uploader._notifier.event(to_filepath(p + u'.backup'), fake_inotify.IN_MOVED_TO) - yield iterate(self.bob_magicfolder) - d.addCallback(notify_bob_moved) - - @log_call_deferred(action_type=u"check_state") - @inline_callbacks - def check_state(ignored): - yield self._check_version_in_dmd(self.alice_magicfolder, u"file1", 1) - self._check_version_in_local_db(self.alice_magicfolder, u"file1", 1) - self._check_uploader_count('objects_failed', 0) - self._check_uploader_count('objects_succeeded', 2) - self._check_uploader_count('objects_succeeded', 0, magic=self.bob_magicfolder) - - self._check_version_in_local_db(self.bob_magicfolder, u"file1", 1) - self._check_version_in_dmd(self.bob_magicfolder, u"file1", 1) - self._check_file_gone(self.bob_magicfolder, u"file1") - self._check_downloader_count('objects_failed', 0) - self._check_downloader_count('objects_downloaded', 2) - self._check_downloader_count('objects_downloaded', 2, magic=self.bob_magicfolder) - d.addCallback(check_state) - - @inline_callbacks - def Alice_to_rewrite_file(): - if _debug: print("Alice rewrites file\n") - self.file_path = abspath_expanduser_unicode(u"file1", base=self.alice_magicfolder.uploader._local_path_u) - yield self.alice_fileops.write( - self.file_path, - "Alice suddenly sees the white rabbit running into the forest.", - ) - yield iterate(self.alice_magicfolder) - d.addCallback(_wait_for, Alice_to_rewrite_file) - d.addCallback(lambda ign: iterate(self.bob_magicfolder)) - - @log_call_deferred(action_type=u"check_state") - @inline_callbacks - def check_state(ignored): - yield self._check_version_in_dmd(self.alice_magicfolder, u"file1", 2) - self._check_version_in_local_db(self.alice_magicfolder, u"file1", 2) - self._check_uploader_count('objects_failed', 0) - self._check_uploader_count('objects_succeeded', 3) - self._check_uploader_count('files_uploaded', 3) - self._check_uploader_count('objects_queued', 0) - self._check_uploader_count('directories_created', 0) - self._check_downloader_count('objects_conflicted', 0, magic=self.alice_magicfolder) - self._check_downloader_count('objects_conflicted', 0) - - self._check_version_in_dmd(self.bob_magicfolder, u"file1", 2) - self._check_version_in_local_db(self.bob_magicfolder, u"file1", 2) - self._check_downloader_count('objects_failed', 0) - self._check_downloader_count('objects_downloaded', 3) - self._check_uploader_count('objects_succeeded', 0, magic=self.bob_magicfolder) - d.addCallback(check_state) - - path_u = u"/tmp/magic_folder_test" - encoded_path_u = magicpath.path2magic(u"/tmp/magic_folder_test") - - def Alice_tries_to_p0wn_Bob(ign): - if _debug: print("Alice tries to p0wn Bob\n") - iter_d = iterate(self.bob_magicfolder) - processed_d = self.bob_magicfolder.downloader.set_hook('processed') - - # upload a file that would provoke the security bug from #2506 - uploadable = Data("", self.alice_magicfolder._client.convergence) - alice_dmd = self.alice_magicfolder.uploader._upload_dirnode - - d2 = alice_dmd.add_file(encoded_path_u, uploadable, metadata={"version": 0}, overwrite=True) - d2.addCallback(lambda ign: self.failUnless(alice_dmd.has_child(encoded_path_u))) - d2.addCallback(lambda ign: iter_d) - d2.addCallback(_wait_for_Bob, processed_d) - return d2 - d.addCallback(Alice_tries_to_p0wn_Bob) - - @log_call(action_type=u"check_state", include_args=[], include_result=False) - def check_state(ignored): - self.failIf(os.path.exists(path_u)) - self._check_version_in_local_db(self.bob_magicfolder, encoded_path_u, None) - self._check_downloader_count('objects_downloaded', 3) - self._check_downloader_count('objects_conflicted', 0, magic=self.alice_magicfolder) - self._check_downloader_count('objects_conflicted', 0) - d.addCallback(check_state) - - @inline_callbacks - def Bob_to_rewrite_file(): - if _debug: print("Bob rewrites file\n") - self.file_path = abspath_expanduser_unicode(u"file1", base=self.bob_magicfolder.uploader._local_path_u) - if _debug: print("---- bob's file is %r" % (self.file_path,)) - yield self.bob_fileops.write(self.file_path, "No white rabbit to be found.") - yield iterate(self.bob_magicfolder) - d.addCallback(lambda ign: _wait_for(None, Bob_to_rewrite_file, alice=False)) - - @log_call_deferred(action_type=u"check_state") - @inline_callbacks - def check_state(ignored): - yield self._check_version_in_dmd(self.bob_magicfolder, u"file1", 3) - self._check_version_in_local_db(self.bob_magicfolder, u"file1", 3) - self._check_uploader_count('objects_failed', 0, magic=self.bob_magicfolder) - self._check_uploader_count('objects_succeeded', 1, magic=self.bob_magicfolder) - self._check_uploader_count('files_uploaded', 1, magic=self.bob_magicfolder) - self._check_uploader_count('objects_queued', 0, magic=self.bob_magicfolder) - self._check_uploader_count('directories_created', 0, magic=self.bob_magicfolder) - self._check_downloader_count('objects_conflicted', 0, magic=self.bob_magicfolder) - - self._check_version_in_dmd(self.alice_magicfolder, u"file1", 3) - self._check_version_in_local_db(self.alice_magicfolder, u"file1", 3) - self._check_downloader_count('objects_failed', 0, magic=self.alice_magicfolder) - self._check_downloader_count('objects_downloaded', 1, magic=self.alice_magicfolder) - self._check_downloader_count('objects_conflicted', 0, magic=self.alice_magicfolder) - d.addCallback(check_state) - - def Alice_conflicts_with_Bobs_last_downloaded_uri(): - if _debug: print("Alice conflicts with Bob\n") - downloaded_d = self.bob_magicfolder.downloader.set_hook('processed') - uploadable = Data("do not follow the white rabbit", self.alice_magicfolder._client.convergence) - alice_dmd = self.alice_magicfolder.uploader._upload_dirnode - d2 = alice_dmd.add_file(u"file1", uploadable, - metadata={"version": 5, - "last_downloaded_uri" : "URI:LIT:" }, - overwrite=True) - if _debug: print("Waiting for Alice to upload\n") - d2.addCallback(lambda ign: bob_clock.advance(6)) - d2.addCallback(lambda ign: downloaded_d) - d2.addCallback(lambda ign: self.failUnless(alice_dmd.has_child(encoded_path_u))) - return d2 - d.addCallback(lambda ign: Alice_conflicts_with_Bobs_last_downloaded_uri()) - - @log_call(action_type=u"check_state", include_args=[], include_result=False) - def check_state(ignored): - self._check_downloader_count('objects_downloaded', 4, magic=self.bob_magicfolder) - self._check_downloader_count('objects_conflicted', 1, magic=self.bob_magicfolder) - self._check_downloader_count('objects_downloaded', 1, magic=self.alice_magicfolder) - self._check_downloader_count('objects_failed', 0, magic=self.alice_magicfolder) - self._check_downloader_count('objects_conflicted', 0, magic=self.alice_magicfolder) - self._check_uploader_count('files_uploaded', 1, magic=self.bob_magicfolder) - self._check_uploader_count('objects_succeeded', 1, magic=self.bob_magicfolder) - d.addCallback(check_state) - - # prepare to perform another conflict test - @log_call_deferred(action_type=u"alice:to-write:file2") - @inline_callbacks - def Alice_to_write_file2(): - if _debug: print("Alice writes a file2\n") - self.file_path = abspath_expanduser_unicode(u"file2", base=self.alice_magicfolder.uploader._local_path_u) - d = self.alice_fileops.write(self.file_path, "something") - self.bob_clock.advance(4) - yield d - d.addCallback(_wait_for, Alice_to_write_file2) - - @log_call_deferred(action_type=u"check_state") - @inline_callbacks - def check_state(ignored): - yield self._check_version_in_dmd(self.alice_magicfolder, u"file2", 0) - self._check_version_in_local_db(self.alice_magicfolder, u"file2", 0) - self._check_downloader_count('objects_failed', 0, magic=self.alice_magicfolder) - self._check_downloader_count('objects_conflicted', 0, magic=self.alice_magicfolder) - self._check_uploader_count('files_uploaded', 1, magic=self.bob_magicfolder) - d.addCallback(check_state) - - def advance(ign): - alice_clock.advance(4) - bob_clock.advance(4) - # we need to pause here, or make "is_new_file()" more - # robust, because this is now fast enough that the mtime - # of the allegedly-new file matches, so Bob decides not to - # upload (and the test hangs). Not sure why it worked - # before; must have been *just* slow enough? - # XXX FIXME for the new real-test had to jack this to 0.5; - # related to the 0.1 notify pause?? - return task.deferLater(reactor, 0.5, lambda: None) - d.addCallback(advance) - d.addCallback(lambda ign: self._check_version_in_local_db(self.bob_magicfolder, u"file2", 0)) - - @inline_callbacks - def Bob_to_rewrite_file2(): - if _debug: print("Bob rewrites file2\n") - self.file_path = abspath_expanduser_unicode(u"file2", base=self.bob_magicfolder.uploader._local_path_u) - if _debug: print("---- bob's file is %r" % (self.file_path,)) - yield iterate(self.bob_magicfolder) - yield self.bob_fileops.write(self.file_path, "roger roger. what vector?") - if _debug: print("---- bob rewrote file2") - yield iterate(self.bob_magicfolder) - if _debug: print("---- iterated bob's magicfolder") - d.addCallback(lambda ign: _wait_for(None, Bob_to_rewrite_file2, alice=False)) - - @log_call_deferred(action_type=u"check_state") - @inline_callbacks - def check_state(ignored): - yield self._check_version_in_dmd(self.bob_magicfolder, u"file2", 1) - self._check_downloader_count('objects_downloaded', 5, magic=self.bob_magicfolder) - self._check_downloader_count('objects_conflicted', 1, magic=self.bob_magicfolder) - self._check_uploader_count('objects_failed', 0, magic=self.bob_magicfolder) - self._check_uploader_count('objects_succeeded', 2, magic=self.bob_magicfolder) - self._check_uploader_count('files_uploaded', 2, magic=self.bob_magicfolder) - self._check_uploader_count('objects_queued', 0, magic=self.bob_magicfolder) - self._check_uploader_count('directories_created', 0, magic=self.bob_magicfolder) - self._check_downloader_count('objects_conflicted', 0, magic=self.alice_magicfolder) - self._check_downloader_count('objects_failed', 0, magic=self.alice_magicfolder) - self._check_downloader_count('objects_downloaded', 2, magic=self.alice_magicfolder) - d.addCallback(check_state) - - # XXX here we advance the clock and then test again to make sure no values are monotonically increasing - # with each queue turn ;-p - alice_clock.advance(6) - bob_clock.advance(6) - - @log_call_deferred(action_type=u"check_state") - @inline_callbacks - def check_state(ignored): - yield self._check_version_in_dmd(self.bob_magicfolder, u"file2", 1) - self._check_downloader_count('objects_downloaded', 5) - self._check_downloader_count('objects_conflicted', 1) - self._check_uploader_count('objects_failed', 0, magic=self.bob_magicfolder) - self._check_uploader_count('objects_succeeded', 2, magic=self.bob_magicfolder) - self._check_uploader_count('files_uploaded', 2, magic=self.bob_magicfolder) - self._check_uploader_count('directories_created', 0, magic=self.bob_magicfolder) - self._check_downloader_count('objects_conflicted', 0, magic=self.alice_magicfolder) - self._check_downloader_count('objects_failed', 0, magic=self.alice_magicfolder) - self._check_downloader_count('objects_downloaded', 2, magic=self.alice_magicfolder) - self._check_uploader_count('files_uploaded', 2, magic=self.bob_magicfolder) - d.addCallback(check_state) - - def Alice_conflicts_with_Bobs_last_uploaded_uri(): - if _debug: print("Alice conflicts with Bob\n") - encoded_path_u = magicpath.path2magic(u"file2") - downloaded_d = self.bob_magicfolder.downloader.set_hook('processed') - uploadable = Data("rabbits with sharp fangs", self.alice_magicfolder._client.convergence) - alice_dmd = self.alice_magicfolder.uploader._upload_dirnode - d2 = alice_dmd.add_file(u"file2", uploadable, - metadata={"version": 5, - "last_uploaded_uri" : "URI:LIT:" }, - overwrite=True) - if _debug: print("Waiting for Alice to upload\n") - d2.addCallback(lambda ign: bob_clock.advance(6)) - d2.addCallback(lambda ign: downloaded_d) - d2.addCallback(lambda ign: self.failUnless(alice_dmd.has_child(encoded_path_u))) - return d2 - d.addCallback(lambda ign: Alice_conflicts_with_Bobs_last_uploaded_uri()) - - @log_call_deferred(action_type=u"check_state") - @inline_callbacks - def check_state(ignored): - yield self._check_version_in_dmd(self.bob_magicfolder, u"file2", 5) - self._check_downloader_count('objects_downloaded', 6) - self._check_downloader_count('objects_conflicted', 1) - self._check_uploader_count('objects_failed', 0, magic=self.bob_magicfolder) - self._check_uploader_count('objects_succeeded', 2, magic=self.bob_magicfolder) - self._check_uploader_count('files_uploaded', 2, magic=self.bob_magicfolder) - self._check_uploader_count('directories_created', 0, magic=self.bob_magicfolder) - self._check_downloader_count('objects_conflicted', 0, magic=self.alice_magicfolder) - self._check_downloader_count('objects_failed', 0, magic=self.alice_magicfolder) - self._check_downloader_count('objects_downloaded', 2, magic=self.alice_magicfolder) - d.addCallback(check_state) - - def foo(ign): - alice_clock.advance(6) - bob_clock.advance(6) - alice_clock.advance(6) - bob_clock.advance(6) - d.addCallback(foo) - - @log_call(action_type=u"check_state", include_args=[], include_result=False) - def check_state(ignored): - self._check_downloader_count('objects_downloaded', 2, magic=self.alice_magicfolder) - self._check_downloader_count('objects_conflicted', 0, magic=self.alice_magicfolder) - self._check_downloader_count('objects_conflicted', 1) - self._check_downloader_count('objects_downloaded', 6) - d.addCallback(check_state) - - # prepare to perform another conflict test - @inline_callbacks - def Alice_to_write_file3(): - if _debug: print("Alice writes a file\n") - self.file_path = abspath_expanduser_unicode(u"file3", base=self.alice_magicfolder.uploader._local_path_u) - yield self.alice_fileops.write(self.file_path, "something") - yield iterate(self.alice_magicfolder) - # Make sure Bob gets the file before we do anything else. - yield iterate(self.bob_magicfolder) - d.addCallback(_wait_for, Alice_to_write_file3) - - @log_call_deferred(action_type=u"check_state") - @inline_callbacks - def check_state(ignored): - yield self._check_version_in_dmd(self.alice_magicfolder, u"file3", 0) - self._check_downloader_count('objects_failed', 0, magic=self.alice_magicfolder) - self._check_downloader_count('objects_downloaded', 7) - self._check_downloader_count('objects_downloaded', 2, magic=self.alice_magicfolder) - self._check_downloader_count('objects_conflicted', 1) - self._check_downloader_count('objects_conflicted', 0, magic=self.alice_magicfolder) - d.addCallback(check_state) - - @inline_callbacks - def Bob_to_rewrite_file3(): - if _debug: print("Bob rewrites file3\n") - self.file_path = abspath_expanduser_unicode(u"file3", base=self.bob_magicfolder.uploader._local_path_u) - if _debug: print("---- bob's file is %r" % (self.file_path,)) - yield iterate(self.bob_magicfolder) - yield self.bob_fileops.write(self.file_path, "roger roger") - yield iterate(self.bob_magicfolder) - d.addCallback(lambda ign: _wait_for(None, Bob_to_rewrite_file3, alice=False)) - - @log_call_deferred(action_type=u"check_state") - @inline_callbacks - def check_state(ignored): - yield self._check_version_in_dmd(self.bob_magicfolder, u"file3", 1) - self._check_downloader_count('objects_downloaded', 7) - self._check_downloader_count('objects_conflicted', 1) - self._check_uploader_count('objects_failed', 0, magic=self.bob_magicfolder) - self._check_uploader_count('objects_succeeded', 3, magic=self.bob_magicfolder) - self._check_uploader_count('files_uploaded', 3, magic=self.bob_magicfolder) - self._check_uploader_count('objects_queued', 0, magic=self.bob_magicfolder) - self._check_uploader_count('directories_created', 0, magic=self.bob_magicfolder) - self._check_downloader_count('objects_conflicted', 0, magic=self.alice_magicfolder) - self._check_downloader_count('objects_failed', 0, magic=self.alice_magicfolder) - self._check_downloader_count('objects_downloaded', 3, magic=self.alice_magicfolder) - d.addCallback(check_state) - - return d.addActionFinish() - - -class SingleMagicFolderTestMixin(MagicFolderCLITestMixin, ShouldFailMixin, ReallyEqualMixin, CheckerMixin): - """ - These tests will be run both with a mock notifier, and (on platforms that support it) - with the real INotify. - """ - - def setUp(self): - self.assertIs(None, super(SingleMagicFolderTestMixin, self).setUp()) - temp = self.mktemp() - self.basedir = abspath_expanduser_unicode(temp.decode(get_filesystem_encoding())) - self.magicfolder = None - self.set_up_grid(oneshare=True) - self.local_dir = os.path.join(self.basedir, u"local_dir") - self.mkdir_nonascii(self.local_dir) - - # Magic-folder implementation somehow manages to leave a DelayedCall - # in the reactor from the eventual queue by the end of the test. It - # may have something to do with the upload process but it's not - # entirely clear. It's difficult to track things through the eventual - # queue. It is almost certainly the case that some other Deferred - # involved in magic-folder that is already being waited on elsewhere - # *should* encompass this DelayedCall but I wasn't able to figure out - # where that association needs to be made. So, as a work-around, - # explicitly flush the eventual queue at the end of the test, too. - from foolscap.eventual import flushEventualQueue - self.addCleanup(flushEventualQueue) - - # Sometimes a collective scan fails with UnrecoverableFileError. It's - # not clear to me why. :/ This fixes the issue, though, and all other - # asserted-about behavior is provided whether this case is hit or not. - self.addCleanup( - lambda: self.eliot_logger.flushTracebacks(UnrecoverableFileError) - ) - - d = DeferredContext(self.create_invite_join_magic_folder(self.alice_nickname, self.local_dir)) - d.addCallback(self._restart_client) - # note: _restart_client ultimately sets self.magicfolder to not-None - return d.result - - def tearDown(self): - d = DeferredContext(super(SingleMagicFolderTestMixin, self).tearDown()) - d.addCallback(self.cleanup) - return d.result - - def _createdb(self): - dbfile = abspath_expanduser_unicode(u"magicfolder_default.sqlite", base=self.basedir) - mdb = magicfolderdb.get_magicfolderdb(dbfile, create_version=(magicfolderdb.SCHEMA_v1, 1)) - self.failUnless(mdb, "unable to create magicfolderdb from %r" % (dbfile,)) - self.failUnlessEqual(mdb.VERSION, 1) - return mdb - - @log_call_deferred(action_type=u"restart-client") - def _restart_client(self, ign): - #print "_restart_client" - d = DeferredContext(self.restart_client()) - d.addCallback(self._wait_until_started) - return d.result - - @log_call_deferred(action_type=u"wait-until-started") - def _wait_until_started(self, ign): - #print "_wait_until_started" - self.magicfolder = self.get_client().getServiceNamed('magic-folder-default') - self.fileops = FileOperationsHelper(self.magicfolder.uploader, self.inject_inotify) - self.up_clock = task.Clock() - self.down_clock = task.Clock() - self.magicfolder.uploader._clock = self.up_clock - self.magicfolder.downloader._clock = self.down_clock - - # XXX should probably be passing the reactor to instances when - # they're created, but that's a ton of re-factoring, so we - # side-step that issue by hacking it in here. However, we - # *have* to "hack it in" before we call ready() so that the - # first iteration of the loop doesn't call the "real" - # reactor's callLater. :( - return self.magicfolder.ready() - - def test_db_basic(self): - fileutil.make_dirs(self.basedir) - self._createdb() - - @inline_callbacks - def test_scan_once_on_startup(self): - # What is this test? Maybe it is just a stub and needs finishing. - self.magicfolder.uploader._clock.advance(99) - - yield self._check_uploader_count('files_uploaded', 0, magic=self.magicfolder) - yield self._check_uploader_count('objects_queued', 0, magic=self.magicfolder) - yield self._check_downloader_count('objects_conflicted', 0, magic=self.magicfolder) - yield self._check_uploader_count('objects_succeeded', 0, magic=self.magicfolder) - yield self._check_downloader_count('objects_failed', 0, magic=self.magicfolder) - yield self._check_downloader_count('objects_downloaded', 0, magic=self.magicfolder) - - def test_db_persistence(self): - """Test that a file upload creates an entry in the database.""" - - fileutil.make_dirs(self.basedir) - db = self._createdb() - - relpath1 = u"myFile1" - pathinfo = fileutil.PathInfo(isdir=False, isfile=True, islink=False, - exists=True, size=1, mtime_ns=123, ctime_ns=456) - db.did_upload_version(relpath1, 0, 'URI:LIT:1', 'URI:LIT:0', 0, pathinfo) - - c = db.cursor - c.execute("SELECT size, mtime_ns, ctime_ns" - " FROM local_files" - " WHERE path=?", - (relpath1,)) - row = c.fetchone() - self.failUnlessEqual(row, (pathinfo.size, pathinfo.mtime_ns, pathinfo.ctime_ns)) - - # Second test uses magic_folder.is_new_file instead of SQL query directly - # to confirm the previous upload entry in the db. - relpath2 = u"myFile2" - path2 = os.path.join(self.basedir, relpath2) - fileutil.write(path2, "meow\n") - pathinfo = fileutil.get_pathinfo(path2) - db.did_upload_version(relpath2, 0, 'URI:LIT:2', 'URI:LIT:1', 0, pathinfo) - db_entry = db.get_db_entry(relpath2) - self.assertFalse(magic_folder.is_new_file(pathinfo, db_entry)) - - different_pathinfo = fileutil.PathInfo(isdir=False, isfile=True, islink=False, - exists=True, size=0, mtime_ns=pathinfo.mtime_ns, - ctime_ns=pathinfo.ctime_ns) - self.assertTrue(magic_folder.is_new_file(different_pathinfo, db_entry)) - - def _test_magicfolder_start_service(self): - # what is this even testing? - d = defer.succeed(None) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.dirs_monitored'), 0)) - - d.addCallback(lambda ign: self.create_invite_join_magic_folder(self.alice_nickname, self.local_dir)) - d.addCallback(self._restart_client) - - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.dirs_monitored'), 1)) - d.addBoth(self.cleanup) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.dirs_monitored'), 0)) - return d - - @skipIf(sys.platform == "linux2", "fails on certain linux flavors: see ticket #2834") - def test_move_tree(self): - """ - create an empty directory tree and 'mv' it into the magic folder, - noting the new directory and uploading it. - - also creates a directory tree with one file in it and 'mv's it - into the magic folder, so we upload the file and record the - directory. (XXX split to separate test) - """ - empty_tree_name = self.unicode_or_fallback(u"empty_tr\u00EAe", u"empty_tree") - empty_tree_dir = abspath_expanduser_unicode(empty_tree_name, base=self.basedir) - new_empty_tree_dir = abspath_expanduser_unicode(empty_tree_name, base=self.local_dir) - - small_tree_name = self.unicode_or_fallback(u"small_tr\u00EAe", u"empty_tree") - small_tree_dir = abspath_expanduser_unicode(small_tree_name, base=self.basedir) - new_small_tree_dir = abspath_expanduser_unicode(small_tree_name, base=self.local_dir) - - d = DeferredContext(defer.succeed(None)) - - @inline_callbacks - def _check_move_empty_tree(res): - self.mkdir_nonascii(empty_tree_dir) - yield self.fileops.move(empty_tree_dir, new_empty_tree_dir) - yield iterate(self.magicfolder) - - d.addCallback(_check_move_empty_tree) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_failed'), 0)) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_succeeded'), 1)) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.files_uploaded'), 0)) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_queued'), 0)) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.directories_created'), 1)) - - @inline_callbacks - def _check_move_small_tree(res): - self.mkdir_nonascii(small_tree_dir) - what_path = abspath_expanduser_unicode(u"what", base=small_tree_dir) - fileutil.write(what_path, "say when") - yield self.fileops.move(small_tree_dir, new_small_tree_dir) - upstatus = list(self.magicfolder.uploader.get_status()) - downstatus = list(self.magicfolder.downloader.get_status()) - - self.assertEqual(2, len(upstatus)) - self.assertEqual(0, len(downstatus)) - yield iterate(self.magicfolder) - - # when we add the dir, we queue a scan of it; so we want - # the upload to "go" as well requiring 1 more iteration - yield iterate(self.magicfolder) - - d.addCallback(_check_move_small_tree) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_failed'), 0)) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.files_uploaded'), 1)) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_succeeded'), 3)) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_queued'), 0)) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.directories_created'), 2)) - - @inline_callbacks - def _check_moved_tree_is_watched(res): - another_path = abspath_expanduser_unicode(u"another", base=new_small_tree_dir) - yield self.fileops.write(another_path, "file") - yield iterate(self.magicfolder) - yield iterate(self.magicfolder) # windows; why? - - d.addCallback(_check_moved_tree_is_watched) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_failed'), 0)) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_succeeded'), 4)) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.files_uploaded'), 2)) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_queued'), 0)) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.directories_created'), 2)) - - return d.result - - def test_persistence(self): - """ - Perform an upload of a given file and then stop the client. - Start a new client and magic-folder service... and verify that the file is NOT uploaded - a second time. This test is meant to test the database persistence along with - the startup and shutdown code paths of the magic-folder service. - """ - self.collective_dircap = "" # XXX hmmm? - - d = DeferredContext(defer.succeed(None)) - - @inline_callbacks - def create_test_file(filename): - test_file = abspath_expanduser_unicode(filename, base=self.local_dir) - yield self.fileops.write(test_file, "meow %s" % filename) - yield iterate(self.magicfolder) - yield iterate(self.magicfolder) # windows; why? - - d.addCallback(lambda ign: create_test_file(u"what1")) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_failed'), 0)) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_succeeded'), 1)) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_queued'), 0)) - d.addCallback(self.cleanup) - - d.addCallback(self._restart_client) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_failed'), 0)) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_succeeded'), 0)) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_queued'), 0)) - d.addCallback(lambda ign: create_test_file(u"what2")) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_failed'), 0)) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_succeeded'), 1)) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_queued'), 0)) - return d.result - - # all this "self.*" state via 9000 mix-ins is really really - # hard to read, keep track of, etc. Very hard to understand - # what each test uses for setup, etc. :( - - @inline_callbacks - def test_delete(self): - # setup: create a file 'foo' - path = os.path.join(self.local_dir, u'foo') - yield self.fileops.write(path, 'foo\n') - yield iterate_uploader(self.magicfolder) - yield iterate_uploader(self.magicfolder) # req'd for windows; not sure why? - self.assertTrue(os.path.exists(path)) - node, metadata = yield self.magicfolder.downloader._get_collective_latest_file(u'foo') - self.assertTrue(node is not None, "Failed to find %r in DMD" % (path,)) - - # the test: delete the file (and do fake notifies) - yield self.fileops.delete(path) - - yield iterate_uploader(self.magicfolder) - self.assertFalse(os.path.exists(path)) - - yield iterate_downloader(self.magicfolder) - # ensure we still have a DB entry, and that the version is 1 - node, metadata = yield self.magicfolder.downloader._get_collective_latest_file(u'foo') - self.assertTrue(node is not None, "Failed to find %r in DMD" % (path,)) - self.failUnlessEqual(metadata['version'], 1) - - @inline_callbacks - def test_batched_process(self): - """ - status APIs correctly function when there are 2 items queued at - once for processing - """ - # setup: get at least two items into the deque - path0 = os.path.join(self.local_dir, u'foo') - yield self.fileops.write(path0, 'foo\n') - path1 = os.path.join(self.local_dir, u'bar') - yield self.fileops.write(path1, 'bar\n') - - # get the status before we've processed anything - upstatus0 = list(self.magicfolder.uploader.get_status()) - upstatus1 = [] - - def one_item(item): - # grab status after we've processed a single item - us = list(self.magicfolder.uploader.get_status()) - upstatus1.extend(us) - one_d = self.magicfolder.uploader.set_hook('item_processed') - # can't 'yield' here because the hook isn't called until - # inside iterate() - one_d.addCallbacks(one_item, self.fail) - - yield iterate_uploader(self.magicfolder) - yield iterate_uploader(self.magicfolder) # req'd for windows; not sure why? - - # no matter which part of the queue the items are in, we - # should see the same status from the outside - self.assertEqual(upstatus0, upstatus1) - - @inline_callbacks - def test_real_notify_failure(self): - """ - Simulate an exception from the _real_notify helper in - magic-folder's uploader, confirming error-handling works. - """ - - orig_notify = self.magicfolder.uploader._real_notify - - class BadStuff(Exception): - pass - - def bad_stuff(*args, **kw): - # call original method .. - orig_notify(*args, **kw) - # ..but then cause a special problem - raise BadStuff("the bad stuff") - - patch_notify = mock.patch.object( - self.magicfolder.uploader, - '_real_notify', - mock.Mock(side_effect=bad_stuff), - ) - with patch_notify: - path0 = os.path.join(self.local_dir, u'foo') - yield self.fileops.write(path0, 'foo\n') - # this actually triggers two notifies - - # do a reactor turn; this is necessary because our "bad_stuff" - # method calls the hook (so the above 'yield' resumes) right - # *before* it raises the exception; thus, we ensure all the - # pending callbacks including the exception are processed - # before we flush the errors. - yield task.deferLater(reactor, 0, lambda: None) - - errors = self.eliot_logger.flushTracebacks(BadStuff) - # it seems on Windows the "RealTest" variant only produces 1 - # notification for some reason.. - self.assertTrue(len(errors) >= 1) - - @inline_callbacks - def test_delete_and_restore(self): - # setup: create a file - path = os.path.join(self.local_dir, u'foo') - yield self.fileops.write(path, 'foo\n') - yield iterate_uploader(self.magicfolder) - yield iterate_uploader(self.magicfolder) # req'd for windows; why? - self.assertTrue(os.path.exists(path)) - - # ...and delete the file - yield self.fileops.delete(path) - yield iterate_uploader(self.magicfolder) - self.assertFalse(os.path.exists(path)) - - # ensure we still have a DB entry, and that the version is 1 - node, metadata = yield self.magicfolder.downloader._get_collective_latest_file(u'foo') - self.assertTrue(node is not None, "Failed to find %r in DMD" % (path,)) - self.failUnlessEqual(metadata['version'], 1) - - # restore the file, with different contents - path = os.path.join(self.local_dir, u'foo') - yield self.fileops.write(path, 'bar\n') - yield iterate_uploader(self.magicfolder) - - # ensure we still have a DB entry, and that the version is 2 - node, metadata = yield self.magicfolder.downloader._get_collective_latest_file(u'foo') - self.assertTrue(node is not None, "Failed to find %r in DMD" % (path,)) - self.failUnlessEqual(metadata['version'], 2) - - def test_write_short_file(self): - # Write something short enough for a LIT file. - return self._check_file(u"short", "test") - - def test_magic_folder(self): - d = DeferredContext(defer.succeed(None)) - # Write something short enough for a LIT file. - d.addCallback(lambda ign: self._check_file(u"short", "test")) - - # Write to the same file again with different data. - d.addCallback(lambda ign: self._check_file(u"short", "different")) - - # Test that temporary files are not uploaded. - d.addCallback(lambda ign: self._check_file(u"tempfile", "test", temporary=True)) - - # Test creation of a subdirectory. - d.addCallback(lambda ign: self._check_mkdir(u"directory")) - - # Write something longer, and also try to test a Unicode name if the fs can represent it. - name_u = self.unicode_or_fallback(u"l\u00F8ng", u"long") - d.addCallback(lambda ign: self._check_file(name_u, "test"*100)) - - # TODO: test that causes an upload failure. - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.objects_failed'), 0)) - - return d.result - - @inline_callbacks - def _create_directory_with_file(self, relpath_u, content): - path_f = os.path.join(self.local_dir, relpath_u) - path_d = os.path.dirname(path_f) - # Create a new directory in the monitored directory. - yield self.fileops.mkdir(path_d) - # Give the system a chance to notice and process it. - yield iterate(self.magicfolder) - # Create a new file in that new directory. - yield self.fileops.write(path_f, content) - # Another opportunity to process. - yield iterate(self.magicfolder) - - @inline_callbacks - def test_create_file_in_sub_directory(self): - reldir_u = u'subdir' - # The OS and the DMD may have conflicting conventions for directory - # the separator. Construct a value for each. - dmd_relpath_u = u'/'.join((reldir_u, u'some-file')) - platform_relpath_u = join(reldir_u, u'some-file') - content = u'some great content' - yield self._create_directory_with_file( - platform_relpath_u, - content, - ) - # The new directory and file should have been noticed and uploaded. - downloader = self.magicfolder.downloader - encoded_dir_u = magicpath.path2magic(reldir_u + u"/") - encoded_path_u = magicpath.path2magic(dmd_relpath_u) - - with start_action(action_type=u"retrieve-metadata"): - dir_node, dir_meta = yield downloader._get_collective_latest_file( - encoded_dir_u, - ) - path_node, path_meta = yield downloader._get_collective_latest_file( - encoded_path_u, - ) - - self.expectThat(dir_node, Not(Is(None)), "dir node") - self.expectThat(dir_meta, ContainsDict({'version': Equals(0)}), "dir meta") - self.expectThat(path_node, Not(Is(None)), "path node") - self.expectThat(path_meta, ContainsDict({'version': Equals(0)}), "path meta") - - @inline_callbacks - def test_delete_file_in_sub_directory(self): - dmd_relpath_u = u'/'.join((u'subdir', u'some-file')) - platform_relpath_u = join(u'subdir', u'some-file') - content = u'some great content' - yield self._create_directory_with_file( - platform_relpath_u, - content, - ) - # Delete the file in the sub-directory. - yield self.fileops.delete(os.path.join(self.local_dir, platform_relpath_u)) - # Let the deletion be processed. - yield iterate(self.magicfolder) - # Verify the deletion was uploaded. - encoded_path_u = magicpath.path2magic(dmd_relpath_u) - downloader = self.magicfolder.downloader - node, metadata = yield downloader._get_collective_latest_file(encoded_path_u) - self.assertThat(node, Not(Is(None))) - self.assertThat(metadata['version'], Equals(1)) - self.assertThat(metadata['deleted'], Equals(True)) - - def test_delete_sub_directory_containing_file(self): - reldir_u = u'subdir' - relpath_u = os.path.join(reldir_u, u'some-file') - content = u'some great content' - yield self._create_directory_with_file( - relpath_u, - content, - ) - # Delete the sub-directory and the file in it. Don't wait in between - # because the case where all events are delivered before any - # processing happens is interesting. And don't use the fileops API to - # delete the contained file so that we don't necessarily generate a - # notification for that path at all. We require that the - # implementation behave correctly when receiving only the notification - # for the containing directory. - os.unlink(os.path.join(self.local_dir, relpath_u)) - yield self.fileops.delete(os.path.join(self.local_dir, reldir_u)) - - # Now allow processing. - yield iterate(self.magicfolder) - # Give it some extra time because of recursive directory processing. - yield iterate(self.magicfolder) - - # Deletion of both entities should have been uploaded. - downloader = self.magicfolder.downloader - encoded_dir_u = magicpath.path2magic(reldir_u + u"/") - encoded_path_u = magicpath.path2magic(relpath_u) - - dir_node, dir_meta = yield downloader._get_collective_latest_file(encoded_dir_u) - path_node, path_meta = yield downloader._get_collective_latest_file(encoded_path_u) - - self.expectThat(dir_node, Not(Is(None)), "dir node") - self.expectThat(dir_meta, ContainsDict({ - "version": Equals(1), - "deleted": Equals(True), - }), "dir meta") - - self.expectThat(path_node, Not(Is(None)), "path node") - self.expectThat(path_meta, ContainsDict({ - "version": Equals(1), - "deleted": Equals(True), - }), "path meta") - - -@skipIf(support_missing, support_message) -class MockTestAliceBob(MagicFolderAliceBobTestMixin, AsyncTestCase): - inject_inotify = True - - def setUp(self): - self.inotify = fake_inotify - self.patch(magic_folder, 'get_inotify_module', lambda: self.inotify) - return super(MockTestAliceBob, self).setUp() - - -@skipIf(support_missing, support_message) -class MockTest(SingleMagicFolderTestMixin, AsyncTestCase): - """This can run on any platform, and even if twisted.internet.inotify can't be imported.""" - inject_inotify = True - - def setUp(self): - self.inotify = fake_inotify - self.patch(magic_folder, 'get_inotify_module', lambda: self.inotify) - return super(MockTest, self).setUp() - - def test_errors(self): - self.set_up_grid(oneshare=True) - - errors_dir = abspath_expanduser_unicode(u"errors_dir", base=self.basedir) - os.mkdir(errors_dir) - not_a_dir = abspath_expanduser_unicode(u"NOT_A_DIR", base=self.basedir) - fileutil.write(not_a_dir, "") - magicfolderdb = abspath_expanduser_unicode(u"magicfolderdb", base=self.basedir) - doesnotexist = abspath_expanduser_unicode(u"doesnotexist", base=self.basedir) - - client = self.g.clients[0] - d = DeferredContext(client.create_dirnode()) - def _check_errors(n): - self.failUnless(IDirectoryNode.providedBy(n)) - upload_dircap = n.get_uri() - readonly_dircap = n.get_readonly_uri() - - self.shouldFail(ValueError, 'does not exist', 'does not exist', - MagicFolder, client, upload_dircap, '', doesnotexist, magicfolderdb, 0o077, 'default') - self.shouldFail(ValueError, 'is not a directory', 'is not a directory', - MagicFolder, client, upload_dircap, '', not_a_dir, magicfolderdb, 0o077, 'default') - self.shouldFail(AssertionError, 'bad upload.dircap', 'does not refer to a directory', - MagicFolder, client, 'bad', '', errors_dir, magicfolderdb, 0o077, 'default') - self.shouldFail(AssertionError, 'non-directory upload.dircap', 'does not refer to a directory', - MagicFolder, client, 'URI:LIT:foo', '', errors_dir, magicfolderdb, 0o077, 'default') - self.shouldFail(AssertionError, 'readonly upload.dircap', 'is not a writecap to a directory', - MagicFolder, client, readonly_dircap, '', errors_dir, magicfolderdb, 0o077, 'default') - self.shouldFail(AssertionError, 'collective dircap', 'is not a readonly cap to a directory', - MagicFolder, client, upload_dircap, upload_dircap, errors_dir, magicfolderdb, 0o077, 'default') - - def _not_implemented(): - raise NotImplementedError("blah") - self.patch(magic_folder, 'get_inotify_module', _not_implemented) - self.shouldFail(NotImplementedError, 'unsupported', 'blah', - MagicFolder, client, upload_dircap, '', errors_dir, magicfolderdb, 0o077, 'default') - d.addCallback(_check_errors) - return d.result - - def test_write_downloaded_file(self): - workdir = fileutil.abspath_expanduser_unicode(u"cli/MagicFolder/write-downloaded-file") - local_file = fileutil.abspath_expanduser_unicode(u"foobar", base=workdir) - - class TestWriteFileMixin(WriteFileMixin): - def _log(self, msg): - pass - - writefile = TestWriteFileMixin() - writefile._umask = 0o077 - - # create a file with name "foobar" with content "foo" - # write downloaded file content "bar" into "foobar" with is_conflict = False - fileutil.make_dirs(workdir) - fileutil.write(local_file, "foo") - - # if is_conflict is False, then the .conflict file shouldn't exist. - now = time.time() - writefile._write_downloaded_file(workdir, local_file, "bar", False, now=now) - conflicted_path = local_file + u".conflict" - self.failIf(os.path.exists(conflicted_path)) - - # no backup - backup_path = local_file + u".backup" - self.failIf(os.path.exists(backup_path)) - - # .tmp file shouldn't exist - self.failIf(os.path.exists(local_file + u".tmp")) - - # The original file should have the new content - self.failUnlessEqual(fileutil.read(local_file), "bar") - - # .. and approximately the correct timestamp. - pathinfo = fileutil.get_pathinfo(local_file) - error_ns = pathinfo.mtime_ns - fileutil.seconds_to_ns(now - WriteFileMixin.FUDGE_SECONDS) - permitted_error_ns = fileutil.seconds_to_ns(WriteFileMixin.FUDGE_SECONDS)/4 - self.failUnless(abs(error_ns) < permitted_error_ns, (error_ns, permitted_error_ns)) - - # now a test for conflicted case - writefile._write_downloaded_file(workdir, local_file, "bar", True, None) - self.failUnless(os.path.exists(conflicted_path)) - - # .tmp file shouldn't exist - self.failIf(os.path.exists(local_file + u".tmp")) - - def test_periodic_full_scan(self): - """ - Create a file in a subdir without doing a notify on it and - fast-forward time to prove we do a full scan periodically. - """ - sub_dir = abspath_expanduser_unicode(u"subdir", base=self.local_dir) - self.mkdir_nonascii(sub_dir) - - d = DeferredContext(defer.succeed(None)) - - def _create_file_without_event(res): - processed_d = self.magicfolder.uploader.set_hook('processed') - what_path = abspath_expanduser_unicode(u"what", base=sub_dir) - fileutil.write(what_path, "say when") - self.magicfolder.uploader._clock.advance(self.magicfolder.uploader._periodic_full_scan_duration + 1) - # this will have now done the full scan, so we have to do - # an iteration to process anything from it - iterate_d = iterate_uploader(self.magicfolder) - return processed_d.addCallback(lambda ignored: iterate_d) - d.addCallback(_create_file_without_event) - def _advance_clock(res): - processed_d = self.magicfolder.uploader.set_hook('processed') - self.magicfolder.uploader._clock.advance(4) - return processed_d - d.addCallback(_advance_clock) - d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('uploader.files_uploaded'), 1)) - return d.result - - def test_statistics(self): - d = DeferredContext(defer.succeed(None)) - # Write something short enough for a LIT file. - d.addCallback(lambda ign: self._check_file(u"short", "test")) - - # test magic-folder statistics - d.addCallback(lambda res: self.GET("statistics")) - def _got_stats(res): - self.assertIn("Operational Statistics", res) - self.assertIn("Magic Folder", res) - self.assertIn("
  • Local Directories Monitored: 1 directories
    • ", res) - self.assertIn("
  • Files Uploaded: 1 files
    • ", res) - self.assertIn("
  • Files Queued for Upload: 0 files
    • ", res) - self.assertIn("
  • Failed Uploads: 0 files
    • ", res) - self.assertIn("
  • Files Downloaded: 0 files
    • ", res) - self.assertIn("
  • Files Queued for Download: 0 files
    • ", res) - self.assertIn("
  • Failed Downloads: 0 files
    • ", res) - d.addCallback(_got_stats) - d.addCallback(lambda res: self.GET("statistics?t=json")) - def _got_stats_json(res): - data = json.loads(res) - self.assertEqual(data["counters"]["magic_folder.uploader.dirs_monitored"], 1) - self.assertEqual(data["counters"]["magic_folder.uploader.objects_succeeded"], 1) - self.assertEqual(data["counters"]["magic_folder.uploader.files_uploaded"], 1) - self.assertEqual(data["counters"]["magic_folder.uploader.objects_queued"], 0) - d.addCallback(_got_stats_json) - return d.result - - -@skipIf(support_missing, support_message) -class RealTest(SingleMagicFolderTestMixin, AsyncTestCase): - """This is skipped unless both Twisted and the platform support inotify.""" - inject_inotify = False - - def setUp(self): - d = super(RealTest, self).setUp() - self.inotify = magic_folder.get_inotify_module() - return d - - -@skipIf(support_missing, support_message) -class RealTestAliceBob(MagicFolderAliceBobTestMixin, AsyncTestCase): - """This is skipped unless both Twisted and the platform support inotify.""" - inject_inotify = False - - def setUp(self): - d = super(RealTestAliceBob, self).setUp() - self.inotify = magic_folder.get_inotify_module() - return d diff --git a/src/allmydata/test/test_magicpath.py b/src/allmydata/test/test_magicpath.py deleted file mode 100644 index 1227a2c4d..000000000 --- a/src/allmydata/test/test_magicpath.py +++ /dev/null @@ -1,28 +0,0 @@ - -from twisted.trial import unittest - -from allmydata import magicpath - - -class MagicPath(unittest.TestCase): - tests = { - u"Documents/work/critical-project/qed.txt": u"Documents@_work@_critical-project@_qed.txt", - u"Documents/emails/bunnyfufu@hoppingforest.net": u"Documents@_emails@_bunnyfufu@@hoppingforest.net", - u"foo/@/bar": u"foo@_@@@_bar", - } - - def test_path2magic(self): - for test, expected in self.tests.items(): - self.failUnlessEqual(magicpath.path2magic(test), expected) - - def test_magic2path(self): - for expected, test in self.tests.items(): - self.failUnlessEqual(magicpath.magic2path(test), expected) - - def test_should_ignore(self): - self.failUnlessEqual(magicpath.should_ignore_file(u".bashrc"), True) - self.failUnlessEqual(magicpath.should_ignore_file(u"bashrc."), False) - self.failUnlessEqual(magicpath.should_ignore_file(u"forest/tree/branch/.bashrc"), True) - self.failUnlessEqual(magicpath.should_ignore_file(u"forest/tree/.branch/bashrc"), True) - self.failUnlessEqual(magicpath.should_ignore_file(u"forest/.tree/branch/bashrc"), True) - self.failUnlessEqual(magicpath.should_ignore_file(u"forest/tree/branch/bashrc"), False) diff --git a/src/allmydata/test/test_python2_regressions.py b/src/allmydata/test/test_python2_regressions.py index 565e3ad80..84484f1cf 100644 --- a/src/allmydata/test/test_python2_regressions.py +++ b/src/allmydata/test/test_python2_regressions.py @@ -15,9 +15,6 @@ from testtools.matchers import ( BLACKLIST = { "allmydata.test.check_load", - "allmydata.watchdog._watchdog_541", - "allmydata.watchdog.inotify", - "allmydata.windows.inotify", "allmydata.windows.registry", } diff --git a/src/allmydata/test/test_websocket_logs.py b/src/allmydata/test/test_websocket_logs.py index 983613a28..e666a4902 100644 --- a/src/allmydata/test/test_websocket_logs.py +++ b/src/allmydata/test/test_websocket_logs.py @@ -40,7 +40,7 @@ class TestStreamingLogs(unittest.TestCase): messages.append(json.loads(msg)) proto.on("message", got_message) - @log_call(action_type=u"test:cli:magic-folder:cleanup") + @log_call(action_type=u"test:cli:some-exciting-action") def do_a_thing(): pass diff --git a/src/allmydata/test/web/test_web.py b/src/allmydata/test/web/test_web.py index 346e0e2c3..3e0967811 100644 --- a/src/allmydata/test/web/test_web.py +++ b/src/allmydata/test/web/test_web.py @@ -35,7 +35,6 @@ from allmydata.immutable import upload from allmydata.immutable.downloader.status import DownloadStatus from allmydata.dirnode import DirectoryNode from allmydata.nodemaker import NodeMaker -from allmydata.frontends.magic_folder import QueuedItem from allmydata.web import status from allmydata.web.common import WebError, MultiFormatPage from allmydata.util import fileutil, base32, hashutil @@ -125,29 +124,6 @@ class FakeUploader(service.Service): return (self.helper_furl, self.helper_connected) -def create_test_queued_item(relpath_u, history=[]): - progress = mock.Mock() - progress.progress = 100.0 - item = QueuedItem(relpath_u, progress, 1234) - for the_status, timestamp in history: - item.set_status(the_status, current_time=timestamp) - return item - - -class FakeMagicFolder(object): - def __init__(self): - self.uploader = FakeStatus() - self.downloader = FakeStatus() - - def get_public_status(self): - return ( - True, - [ - 'a magic-folder status message' - ], - ) - - def build_one_ds(): ds = DownloadStatus("storage_index", 1234) now = time.time() @@ -282,7 +258,6 @@ class FakeClient(_Client): # don't upcall to Client.__init__, since we only want to initialize a # minimal subset service.MultiService.__init__(self) - self._magic_folders = dict() self.all_contents = {} self.nodeid = "fake_nodeid" self.nickname = u"fake_nickname \u263A" @@ -997,79 +972,6 @@ class Web(WebMixin, WebErrorMixin, testutil.StallMixin, testutil.ReallyEqualMixi d.addCallback(_check) return d - @defer.inlineCallbacks - def test_magicfolder_status_bad_token(self): - with self.assertRaises(Error): - yield self.POST( - '/magic_folder?t=json', - t='json', - name='default', - token='not the token you are looking for', - ) - - @defer.inlineCallbacks - def test_magicfolder_status_wrong_folder(self): - with self.assertRaises(Exception) as ctx: - yield self.POST( - '/magic_folder?t=json', - t='json', - name='a non-existent magic-folder', - token=self.s.get_auth_token(), - ) - self.assertIn( - "Not Found", - str(ctx.exception) - ) - - @defer.inlineCallbacks - def test_magicfolder_status_success(self): - self.s._magic_folders['default'] = mf = FakeMagicFolder() - mf.uploader.status = [ - create_test_queued_item(u"rel/uppath", [('done', 12345)]) - ] - mf.downloader.status = [ - create_test_queued_item(u"rel/downpath", [('done', 23456)]) - ] - data = yield self.POST( - '/magic_folder?t=json', - t='json', - name='default', - token=self.s.get_auth_token(), - ) - data = json.loads(data) - self.assertEqual( - data, - [ - { - "status": "done", - "path": "rel/uppath", - "kind": "upload", - "percent_done": 100.0, - "done_at": 12345, - "size": 1234, - }, - { - "status": "done", - "path": "rel/downpath", - "kind": "download", - "percent_done": 100.0, - "done_at": 23456, - "size": 1234, - }, - ] - ) - - @defer.inlineCallbacks - def test_magicfolder_root_success(self): - self.s._magic_folders['default'] = mf = FakeMagicFolder() - mf.uploader.status = [ - create_test_queued_item(u"rel/path", [('done', 12345)]) - ] - data = yield self.GET( - '/', - ) - del data - def test_status(self): h = self.s.get_history() dl_num = h.list_all_download_statuses()[0].get_counter() diff --git a/src/allmydata/util/eliotutil.py b/src/allmydata/util/eliotutil.py index 7b36a27d1..16ab373e0 100644 --- a/src/allmydata/util/eliotutil.py +++ b/src/allmydata/util/eliotutil.py @@ -16,15 +16,6 @@ __all__ = [ "opt_help_eliot_destinations", "validateInstanceOf", "validateSetMembership", - "MAYBE_NOTIFY", - "CALLBACK", - "INOTIFY_EVENTS", - "RELPATH", - "VERSION", - "LAST_UPLOADED_URI", - "LAST_DOWNLOADED_URI", - "LAST_DOWNLOADED_TIMESTAMP", - "PATHINFO", ] from sys import ( @@ -86,14 +77,6 @@ from twisted.internet.defer import ( ) from twisted.application.service import Service - -from .fileutil import ( - PathInfo, -) -from .fake_inotify import ( - humanReadableMask, -) - def validateInstanceOf(t): """ Return an Eliot validator that requires values to be instances of ``t``. @@ -112,72 +95,6 @@ def validateSetMembership(s): raise ValidationError("{} not in {}".format(v, s)) return validator -RELPATH = Field.for_types( - u"relpath", - [unicode], - u"The relative path of a file in a magic-folder.", -) - -VERSION = Field.for_types( - u"version", - [int, long], - u"The version of the file.", -) - -LAST_UPLOADED_URI = Field.for_types( - u"last_uploaded_uri", - [unicode, bytes, None], - u"The filecap to which this version of this file was uploaded.", -) - -LAST_DOWNLOADED_URI = Field.for_types( - u"last_downloaded_uri", - [unicode, bytes, None], - u"The filecap from which the previous version of this file was downloaded.", -) - -LAST_DOWNLOADED_TIMESTAMP = Field.for_types( - u"last_downloaded_timestamp", - [float, int, long], - u"(XXX probably not really, don't trust this) The timestamp of the last download of this file.", -) - -PATHINFO = Field( - u"pathinfo", - lambda v: None if v is None else { - "isdir": v.isdir, - "isfile": v.isfile, - "islink": v.islink, - "exists": v.exists, - "size": v.size, - "mtime_ns": v.mtime_ns, - "ctime_ns": v.ctime_ns, - }, - u"The metadata for this version of this file.", - validateInstanceOf((type(None), PathInfo)), -) - -INOTIFY_EVENTS = Field( - u"inotify_events", - humanReadableMask, - u"Details about a filesystem event generating a notification event.", - validateInstanceOf((int, long)), -) - -MAYBE_NOTIFY = ActionType( - u"filesystem:notification:maybe-notify", - [], - [], - u"A filesystem event is being considered for dispatch to an application handler.", -) - -CALLBACK = ActionType( - u"filesystem:notification:callback", - [INOTIFY_EVENTS], - [], - u"A filesystem event is being dispatched to an application callback." -) - def eliot_logging_service(reactor, destinations): """ Parse the given Eliot destination descriptions and return an ``IService`` diff --git a/src/allmydata/util/fake_inotify.py b/src/allmydata/util/fake_inotify.py deleted file mode 100644 index c6d0b16e2..000000000 --- a/src/allmydata/util/fake_inotify.py +++ /dev/null @@ -1,109 +0,0 @@ - -# Most of this is copied from Twisted 11.0. The reason for this hack is that -# twisted.internet.inotify can't be imported when the platform does not support inotify. - -import six - -if six.PY3: - long = int - -# from /usr/src/linux/include/linux/inotify.h - -IN_ACCESS = long(0x00000001) # File was accessed -IN_MODIFY = long(0x00000002) # File was modified -IN_ATTRIB = long(0x00000004) # Metadata changed -IN_CLOSE_WRITE = long(0x00000008) # Writeable file was closed -IN_CLOSE_NOWRITE = long(0x00000010) # Unwriteable file closed -IN_OPEN = long(0x00000020) # File was opened -IN_MOVED_FROM = long(0x00000040) # File was moved from X -IN_MOVED_TO = long(0x00000080) # File was moved to Y -IN_CREATE = long(0x00000100) # Subfile was created -IN_DELETE = long(0x00000200) # Subfile was delete -IN_DELETE_SELF = long(0x00000400) # Self was deleted -IN_MOVE_SELF = long(0x00000800) # Self was moved -IN_UNMOUNT = long(0x00002000) # Backing fs was unmounted -IN_Q_OVERFLOW = long(0x00004000) # Event queued overflowed -IN_IGNORED = long(0x00008000) # File was ignored - -IN_ONLYDIR = 0x01000000 # only watch the path if it is a directory -IN_DONT_FOLLOW = 0x02000000 # don't follow a sym link -IN_MASK_ADD = 0x20000000 # add to the mask of an already existing watch -IN_ISDIR = 0x40000000 # event occurred against dir -IN_ONESHOT = 0x80000000 # only send event once - -IN_CLOSE = IN_CLOSE_WRITE | IN_CLOSE_NOWRITE # closes -IN_MOVED = IN_MOVED_FROM | IN_MOVED_TO # moves -IN_CHANGED = IN_MODIFY | IN_ATTRIB # changes - -IN_WATCH_MASK = (IN_MODIFY | IN_ATTRIB | - IN_CREATE | IN_DELETE | - IN_DELETE_SELF | IN_MOVE_SELF | - IN_UNMOUNT | IN_MOVED_FROM | IN_MOVED_TO) - - -_FLAG_TO_HUMAN = [ - (IN_ACCESS, 'access'), - (IN_MODIFY, 'modify'), - (IN_ATTRIB, 'attrib'), - (IN_CLOSE_WRITE, 'close_write'), - (IN_CLOSE_NOWRITE, 'close_nowrite'), - (IN_OPEN, 'open'), - (IN_MOVED_FROM, 'moved_from'), - (IN_MOVED_TO, 'moved_to'), - (IN_CREATE, 'create'), - (IN_DELETE, 'delete'), - (IN_DELETE_SELF, 'delete_self'), - (IN_MOVE_SELF, 'move_self'), - (IN_UNMOUNT, 'unmount'), - (IN_Q_OVERFLOW, 'queue_overflow'), - (IN_IGNORED, 'ignored'), - (IN_ONLYDIR, 'only_dir'), - (IN_DONT_FOLLOW, 'dont_follow'), - (IN_MASK_ADD, 'mask_add'), - (IN_ISDIR, 'is_dir'), - (IN_ONESHOT, 'one_shot') -] - - - -def humanReadableMask(mask): - """ - Auxiliary function that converts an hexadecimal mask into a series - of human readable flags. - """ - s = [] - for k, v in _FLAG_TO_HUMAN: - if k & mask: - s.append(v) - return s - - -from eliot import start_action - -# This class is not copied from Twisted; it acts as a mock. -class INotify(object): - def startReading(self): - pass - - def stopReading(self): - pass - - def loseConnection(self): - pass - - def watch(self, filepath, mask=IN_WATCH_MASK, autoAdd=False, callbacks=None, recursive=False): - self.callbacks = callbacks - - def event(self, filepath, mask): - with start_action(action_type=u"fake-inotify:event", path=filepath.asTextMode().path, mask=mask): - for cb in self.callbacks: - cb(None, filepath, mask) - - -__all__ = ["INotify", "humanReadableMask", "IN_WATCH_MASK", "IN_ACCESS", - "IN_MODIFY", "IN_ATTRIB", "IN_CLOSE_NOWRITE", "IN_CLOSE_WRITE", - "IN_OPEN", "IN_MOVED_FROM", "IN_MOVED_TO", "IN_CREATE", - "IN_DELETE", "IN_DELETE_SELF", "IN_MOVE_SELF", "IN_UNMOUNT", - "IN_Q_OVERFLOW", "IN_IGNORED", "IN_ONLYDIR", "IN_DONT_FOLLOW", - "IN_MASK_ADD", "IN_ISDIR", "IN_ONESHOT", "IN_CLOSE", - "IN_MOVED", "IN_CHANGED"] diff --git a/src/allmydata/watchdog/__init__.py b/src/allmydata/watchdog/__init__.py deleted file mode 100644 index e69de29bb..000000000 diff --git a/src/allmydata/watchdog/_watchdog_541.py b/src/allmydata/watchdog/_watchdog_541.py deleted file mode 100644 index 87686ce6d..000000000 --- a/src/allmydata/watchdog/_watchdog_541.py +++ /dev/null @@ -1,16 +0,0 @@ -""" -Hotfix for https://github.com/gorakhargosh/watchdog/issues/541 -""" - -from watchdog.observers.fsevents import FSEventsEmitter - -# The class object has already been bundled up in the default arguments to -# FSEventsObserver.__init__. So mutate the class object (instead of replacing -# it with a safer version). -original_on_thread_stop = FSEventsEmitter.on_thread_stop -def safe_on_thread_stop(self): - if self.is_alive(): - return original_on_thread_stop(self) - -def patch(): - FSEventsEmitter.on_thread_stop = safe_on_thread_stop diff --git a/src/allmydata/watchdog/inotify.py b/src/allmydata/watchdog/inotify.py deleted file mode 100644 index 68a121a60..000000000 --- a/src/allmydata/watchdog/inotify.py +++ /dev/null @@ -1,212 +0,0 @@ - -""" -An implementation of an inotify-like interface on top of the ``watchdog`` library. -""" - -from __future__ import ( - unicode_literals, - print_function, - absolute_import, - division, -) - -__all__ = [ - "humanReadableMask", "INotify", - "IN_WATCH_MASK", "IN_ACCESS", "IN_MODIFY", "IN_ATTRIB", "IN_CLOSE_NOWRITE", - "IN_CLOSE_WRITE", "IN_OPEN", "IN_MOVED_FROM", "IN_MOVED_TO", "IN_CREATE", - "IN_DELETE", "IN_DELETE_SELF", "IN_MOVE_SELF", "IN_UNMOUNT", "IN_ONESHOT", - "IN_Q_OVERFLOW", "IN_IGNORED", "IN_ONLYDIR", "IN_DONT_FOLLOW", "IN_MOVED", - "IN_MASK_ADD", "IN_ISDIR", "IN_CLOSE", "IN_CHANGED", "_FLAG_TO_HUMAN", -] - -from watchdog.observers import Observer -from watchdog.events import ( - FileSystemEvent, - FileSystemEventHandler, DirCreatedEvent, FileCreatedEvent, - DirDeletedEvent, FileDeletedEvent, FileModifiedEvent -) - -from twisted.internet import reactor -from twisted.python.filepath import FilePath -from allmydata.util.fileutil import abspath_expanduser_unicode - -from eliot import ( - ActionType, - Message, - Field, - preserve_context, - start_action, -) - -from allmydata.util.pollmixin import PollMixin -from allmydata.util.assertutil import _assert, precondition -from allmydata.util import encodingutil -from allmydata.util.fake_inotify import humanReadableMask, \ - IN_WATCH_MASK, IN_ACCESS, IN_MODIFY, IN_ATTRIB, IN_CLOSE_NOWRITE, IN_CLOSE_WRITE, \ - IN_OPEN, IN_MOVED_FROM, IN_MOVED_TO, IN_CREATE, IN_DELETE, IN_DELETE_SELF, \ - IN_MOVE_SELF, IN_UNMOUNT, IN_Q_OVERFLOW, IN_IGNORED, IN_ONLYDIR, IN_DONT_FOLLOW, \ - IN_MASK_ADD, IN_ISDIR, IN_ONESHOT, IN_CLOSE, IN_MOVED, IN_CHANGED, \ - _FLAG_TO_HUMAN - -from ..util.eliotutil import ( - MAYBE_NOTIFY, - CALLBACK, - validateInstanceOf, -) - -from . import _watchdog_541 - -_watchdog_541.patch() - -NOT_STARTED = "NOT_STARTED" -STARTED = "STARTED" -STOPPING = "STOPPING" -STOPPED = "STOPPED" - -_PATH = Field.for_types( - u"path", - [bytes, unicode], - u"The path an inotify event concerns.", -) - -_EVENT = Field( - u"event", - lambda e: e.__class__.__name__, - u"The watchdog event that has taken place.", - validateInstanceOf(FileSystemEvent), -) - -ANY_INOTIFY_EVENT = ActionType( - u"watchdog:inotify:any-event", - [_PATH, _EVENT], - [], - u"An inotify event is being dispatched.", -) - -class INotifyEventHandler(FileSystemEventHandler): - def __init__(self, path, mask, callbacks, pending_delay): - FileSystemEventHandler.__init__(self) - self._path = path - self._mask = mask - self._callbacks = callbacks - self._pending_delay = pending_delay - self._pending = set() - - def _maybe_notify(self, path, event): - with MAYBE_NOTIFY(): - event_mask = IN_CHANGED - if isinstance(event, FileModifiedEvent): - event_mask = event_mask | IN_CLOSE_WRITE - event_mask = event_mask | IN_MODIFY - if isinstance(event, (DirCreatedEvent, FileCreatedEvent)): - # For our purposes, IN_CREATE is irrelevant. - event_mask = event_mask | IN_CLOSE_WRITE - if isinstance(event, (DirDeletedEvent, FileDeletedEvent)): - event_mask = event_mask | IN_DELETE - if event.is_directory: - event_mask = event_mask | IN_ISDIR - if not (self._mask & event_mask): - return - for cb in self._callbacks: - try: - with CALLBACK(inotify_events=event_mask): - cb(None, FilePath(path), event_mask) - except: - # Eliot already logged the exception for us. - # There's nothing else we can do about it here. - pass - - def process(self, event): - event_filepath_u = event.src_path.decode(encodingutil.get_filesystem_encoding()) - event_filepath_u = abspath_expanduser_unicode(event_filepath_u, base=self._path) - - if event_filepath_u == self._path: - # ignore events for parent directory - return - - self._maybe_notify(event_filepath_u, event) - - def on_any_event(self, event): - with ANY_INOTIFY_EVENT(path=event.src_path, event=event): - reactor.callFromThread( - preserve_context(self.process), - event, - ) - - -class INotify(PollMixin): - """ - I am a prototype INotify, made to work on Mac OS X (Darwin) - using the Watchdog python library. This is actually a simplified subset - of the twisted Linux INotify class because we do not utilize the watch mask - and only implement the following methods: - - watch - - startReading - - stopReading - - wait_until_stopped - - set_pending_delay - """ - def __init__(self): - self._pending_delay = 1.0 - self.recursive_includes_new_subdirectories = False - self._callbacks = {} - self._watches = {} - self._state = NOT_STARTED - self._observer = Observer(timeout=self._pending_delay) - - def set_pending_delay(self, delay): - Message.log(message_type=u"watchdog:inotify:set-pending-delay", delay=delay) - assert self._state != STARTED - self._pending_delay = delay - - def startReading(self): - with start_action(action_type=u"watchdog:inotify:start-reading"): - assert self._state != STARTED - try: - # XXX twisted.internet.inotify doesn't require watches to - # be set before startReading is called. - # _assert(len(self._callbacks) != 0, "no watch set") - self._observer.start() - self._state = STARTED - except: - self._state = STOPPED - raise - - def stopReading(self): - with start_action(action_type=u"watchdog:inotify:stop-reading"): - if self._state != STOPPED: - self._state = STOPPING - self._observer.unschedule_all() - self._observer.stop() - self._observer.join() - self._state = STOPPED - - def wait_until_stopped(self): - return self.poll(lambda: self._state == STOPPED) - - def _isWatched(self, path_u): - return path_u in self._callbacks.keys() - - def ignore(self, path): - path_u = path.path - self._observer.unschedule(self._watches[path_u]) - del self._callbacks[path_u] - del self._watches[path_u] - - def watch(self, path, mask=IN_WATCH_MASK, autoAdd=False, callbacks=None, recursive=False): - precondition(isinstance(autoAdd, bool), autoAdd=autoAdd) - precondition(isinstance(recursive, bool), recursive=recursive) - assert autoAdd == False - - path_u = path.path - if not isinstance(path_u, unicode): - path_u = path_u.decode('utf-8') - _assert(isinstance(path_u, unicode), path_u=path_u) - - if path_u not in self._callbacks.keys(): - self._callbacks[path_u] = callbacks or [] - self._watches[path_u] = self._observer.schedule( - INotifyEventHandler(path_u, mask, self._callbacks[path_u], self._pending_delay), - path=path_u, - recursive=False, - ) diff --git a/src/allmydata/web/magic_folder.py b/src/allmydata/web/magic_folder.py deleted file mode 100644 index 27d715d3a..000000000 --- a/src/allmydata/web/magic_folder.py +++ /dev/null @@ -1,52 +0,0 @@ -import json - -from allmydata.web.common import TokenOnlyWebApi, get_arg, WebError - - -class MagicFolderWebApi(TokenOnlyWebApi): - """ - I provide the web-based API for Magic Folder status etc. - """ - - def __init__(self, client): - TokenOnlyWebApi.__init__(self, client) - self.client = client - - def post_json(self, req): - req.setHeader("content-type", "application/json") - nick = get_arg(req, 'name', 'default') - - try: - magic_folder = self.client._magic_folders[nick] - except KeyError: - raise WebError( - "No such magic-folder '{}'".format(nick), - 404, - ) - - data = [] - for item in magic_folder.uploader.get_status(): - d = dict( - path=item.relpath_u, - status=item.status_history()[-1][0], - kind='upload', - ) - for (status, ts) in item.status_history(): - d[status + '_at'] = ts - d['percent_done'] = item.progress.progress - d['size'] = item.size - data.append(d) - - for item in magic_folder.downloader.get_status(): - d = dict( - path=item.relpath_u, - status=item.status_history()[-1][0], - kind='download', - ) - for (status, ts) in item.status_history(): - d[status + '_at'] = ts - d['percent_done'] = item.progress.progress - d['size'] = item.size - data.append(d) - - return json.dumps(data) diff --git a/src/allmydata/web/root.py b/src/allmydata/web/root.py index 4ededb8c9..8726cb00f 100644 --- a/src/allmydata/web/root.py +++ b/src/allmydata/web/root.py @@ -21,7 +21,7 @@ from allmydata.version_checks import get_package_versions_string from allmydata.util import log from allmydata.interfaces import IFileNode from allmydata.web import filenode, directory, unlinked, status -from allmydata.web import storage, magic_folder +from allmydata.web import storage from allmydata.web.common import ( abbreviate_size, getxmlfile, @@ -200,9 +200,6 @@ class Root(MultiFormatPage): self.putChild("uri", URIHandler(client)) self.putChild("cap", URIHandler(client)) - # handler for "/magic_folder" URIs - self.putChild("magic_folder", magic_folder.MagicFolderWebApi(client)) - # Handler for everything beneath "/private", an area of the resource # hierarchy which is only accessible with the private per-node API # auth token. @@ -299,30 +296,6 @@ class Root(MultiFormatPage): return description - def data_magic_folders(self, ctx, data): - return self.client._magic_folders.keys() - - def render_magic_folder_row(self, ctx, data): - magic_folder = self.client._magic_folders[data] - (ok, messages) = magic_folder.get_public_status() - ctx.fillSlots("magic_folder_name", data) - if ok: - ctx.fillSlots("magic_folder_status", "yes") - ctx.fillSlots("magic_folder_status_alt", "working") - else: - ctx.fillSlots("magic_folder_status", "no") - ctx.fillSlots("magic_folder_status_alt", "not working") - - status = T.ul(class_="magic-folder-status") - for msg in messages: - status[T.li[str(msg)]] - return ctx.tag[status] - - def render_magic_folder(self, ctx, data): - if not self.client._magic_folders: - return T.p() - return ctx.tag - def render_services(self, ctx, data): ul = T.ul() try: diff --git a/src/allmydata/web/static/css/new-tahoe.css b/src/allmydata/web/static/css/new-tahoe.css index caf540c07..38bd662e1 100644 --- a/src/allmydata/web/static/css/new-tahoe.css +++ b/src/allmydata/web/static/css/new-tahoe.css @@ -53,11 +53,6 @@ body { .connection-status { } -.magic-folder-status { - clear: left; - margin-left: 40px; /* width of status-indicator + margins */ -} - .furl { font-size: 0.8em; word-wrap: break-word; diff --git a/src/allmydata/web/statistics.xhtml b/src/allmydata/web/statistics.xhtml index fac0607e7..11f5c9643 100644 --- a/src/allmydata/web/statistics.xhtml +++ b/src/allmydata/web/statistics.xhtml @@ -20,18 +20,6 @@
  • Files Retrieved (mutable):
    • -

    Magic Folder

    - -
      -
    • Local Directories Monitored:
      • -
    • Files Uploaded:
      • -
    • Files Queued for Upload:
      • -
    • Failed Uploads:
      • -
    • Files Downloaded:
      • -
    • Files Queued for Download:
      • -
    • Failed Downloads:
      • -
    -

    Raw Stats:

     
diff --git a/src/allmydata/web/status.py b/src/allmydata/web/status.py
index 50e5ec2f8..67c2fffdb 100644
--- a/src/allmydata/web/status.py
+++ b/src/allmydata/web/status.py
@@ -1210,36 +1210,6 @@ class Statistics(MultiFormatPage):
         return "%s files / %s bytes (%s)" % (files, bytes,
                                              abbreviate_size(bytes))
 
-    def render_magic_uploader_monitored(self, ctx, data):
-        dirs = data["counters"].get("magic_folder.uploader.dirs_monitored", 0)
-        return "%s directories" % (dirs,)
-
-    def render_magic_uploader_succeeded(self, ctx, data):
-        # TODO: bytes uploaded
-        files = data["counters"].get("magic_folder.uploader.objects_succeeded", 0)
-        return "%s files" % (files,)
-
-    def render_magic_uploader_queued(self, ctx, data):
-        files = data["counters"].get("magic_folder.uploader.objects_queued", 0)
-        return "%s files" % (files,)
-
-    def render_magic_uploader_failed(self, ctx, data):
-        files = data["counters"].get("magic_folder.uploader.objects_failed", 0)
-        return "%s files" % (files,)
-
-    def render_magic_downloader_succeeded(self, ctx, data):
-        # TODO: bytes uploaded
-        files = data["counters"].get("magic_folder.downloader.objects_succeeded", 0)
-        return "%s files" % (files,)
-
-    def render_magic_downloader_queued(self, ctx, data):
-        files = data["counters"].get("magic_folder.downloader.objects_queued", 0)
-        return "%s files" % (files,)
-
-    def render_magic_downloader_failed(self, ctx, data):
-        files = data["counters"].get("magic_folder.downloader.objects_failed", 0)
-        return "%s files" % (files,)
-
     def render_raw(self, ctx, data):
         raw = pprint.pformat(data)
         return ctx.tag[raw]
diff --git a/src/allmydata/web/welcome.xhtml b/src/allmydata/web/welcome.xhtml
index 079a7cb21..5022bce45 100644
--- a/src/allmydata/web/welcome.xhtml
+++ b/src/allmydata/web/welcome.xhtml
@@ -159,13 +159,6 @@
             
           
 
-          
    
-            
    Magic Folders
    
-            
    
-	      
    img/connected-.png
    
-            
    
-          
    
-
           
    
             
    
               Connected to 
diff --git a/src/allmydata/windows/inotify.py b/src/allmydata/windows/inotify.py
deleted file mode 100644
index 17bea3597..000000000
--- a/src/allmydata/windows/inotify.py
+++ /dev/null
@@ -1,379 +0,0 @@
-
-# Windows near-equivalent to twisted.internet.inotify
-# This should only be imported on Windows.
-
-from __future__ import print_function
-
-import six
-import os, sys
-
-from eliot import (
-    start_action,
-    Message,
-    log_call,
-)
-
-from twisted.internet import reactor
-from twisted.internet.threads import deferToThread
-
-from allmydata.util.fake_inotify import humanReadableMask, \
-    IN_WATCH_MASK, IN_ACCESS, IN_MODIFY, IN_ATTRIB, IN_CLOSE_NOWRITE, IN_CLOSE_WRITE, \
-    IN_OPEN, IN_MOVED_FROM, IN_MOVED_TO, IN_CREATE, IN_DELETE, IN_DELETE_SELF, \
-    IN_MOVE_SELF, IN_UNMOUNT, IN_Q_OVERFLOW, IN_IGNORED, IN_ONLYDIR, IN_DONT_FOLLOW, \
-    IN_MASK_ADD, IN_ISDIR, IN_ONESHOT, IN_CLOSE, IN_MOVED, IN_CHANGED
-[humanReadableMask, \
-    IN_WATCH_MASK, IN_ACCESS, IN_MODIFY, IN_ATTRIB, IN_CLOSE_NOWRITE, IN_CLOSE_WRITE, \
-    IN_OPEN, IN_MOVED_FROM, IN_MOVED_TO, IN_CREATE, IN_DELETE, IN_DELETE_SELF, \
-    IN_MOVE_SELF, IN_UNMOUNT, IN_Q_OVERFLOW, IN_IGNORED, IN_ONLYDIR, IN_DONT_FOLLOW, \
-    IN_MASK_ADD, IN_ISDIR, IN_ONESHOT, IN_CLOSE, IN_MOVED, IN_CHANGED]
-
-from allmydata.util.assertutil import _assert, precondition
-from allmydata.util.encodingutil import quote_output
-from allmydata.util import log, fileutil
-from allmydata.util.pollmixin import PollMixin
-from ..util.eliotutil import (
-    MAYBE_NOTIFY,
-    CALLBACK,
-)
-
-from ctypes import WINFUNCTYPE, WinError, windll, POINTER, byref, create_string_buffer, \
-    addressof, get_last_error
-from ctypes.wintypes import BOOL, HANDLE, DWORD, LPCWSTR, LPVOID
-
-if six.PY3:
-    long = int
-
-# 
-FILE_LIST_DIRECTORY              = 1
-
-# 
-CreateFileW = WINFUNCTYPE(
-    HANDLE,  LPCWSTR, DWORD, DWORD, LPVOID, DWORD, DWORD, HANDLE,
-    use_last_error=True
-)(("CreateFileW", windll.kernel32))
-
-FILE_SHARE_READ                  = 0x00000001
-FILE_SHARE_WRITE                 = 0x00000002
-FILE_SHARE_DELETE                = 0x00000004
-
-OPEN_EXISTING                    = 3
-
-FILE_FLAG_BACKUP_SEMANTICS       = 0x02000000
-
-# 
-CloseHandle = WINFUNCTYPE(
-    BOOL,  HANDLE,
-    use_last_error=True
-)(("CloseHandle", windll.kernel32))
-
-# 
-ReadDirectoryChangesW = WINFUNCTYPE(
-    BOOL,  HANDLE, LPVOID, DWORD, BOOL, DWORD, POINTER(DWORD), LPVOID, LPVOID,
-    use_last_error=True
-)(("ReadDirectoryChangesW", windll.kernel32))
-
-FILE_NOTIFY_CHANGE_FILE_NAME     = 0x00000001
-FILE_NOTIFY_CHANGE_DIR_NAME      = 0x00000002
-FILE_NOTIFY_CHANGE_ATTRIBUTES    = 0x00000004
-#FILE_NOTIFY_CHANGE_SIZE         = 0x00000008
-FILE_NOTIFY_CHANGE_LAST_WRITE    = 0x00000010
-FILE_NOTIFY_CHANGE_LAST_ACCESS   = 0x00000020
-#FILE_NOTIFY_CHANGE_CREATION     = 0x00000040
-FILE_NOTIFY_CHANGE_SECURITY      = 0x00000100
-
-# 
-FILE_ACTION_ADDED                = 0x00000001
-FILE_ACTION_REMOVED              = 0x00000002
-FILE_ACTION_MODIFIED             = 0x00000003
-FILE_ACTION_RENAMED_OLD_NAME     = 0x00000004
-FILE_ACTION_RENAMED_NEW_NAME     = 0x00000005
-
-_action_to_string = {
-    FILE_ACTION_ADDED            : "FILE_ACTION_ADDED",
-    FILE_ACTION_REMOVED          : "FILE_ACTION_REMOVED",
-    FILE_ACTION_MODIFIED         : "FILE_ACTION_MODIFIED",
-    FILE_ACTION_RENAMED_OLD_NAME : "FILE_ACTION_RENAMED_OLD_NAME",
-    FILE_ACTION_RENAMED_NEW_NAME : "FILE_ACTION_RENAMED_NEW_NAME",
-}
-
-_action_to_inotify_mask = {
-    FILE_ACTION_ADDED            : IN_CREATE,
-    FILE_ACTION_REMOVED          : IN_DELETE,
-    FILE_ACTION_MODIFIED         : IN_CHANGED,
-    FILE_ACTION_RENAMED_OLD_NAME : IN_MOVED_FROM,
-    FILE_ACTION_RENAMED_NEW_NAME : IN_MOVED_TO,
-}
-
-INVALID_HANDLE_VALUE             = 0xFFFFFFFF
-
-FALSE = 0
-TRUE = 1
-
-class Event(object):
-    """
-    * action:   a FILE_ACTION_* constant (not a bit mask)
-    * filename: a Unicode string, giving the name relative to the watched directory
-    """
-    def __init__(self, action, filename):
-        self.action = action
-        self.filename = filename
-
-    def __repr__(self):
-        return "Event(%r, %r)" % (_action_to_string.get(self.action, self.action), self.filename)
-
-
-class FileNotifyInformation(object):
-    """
-    I represent a buffer containing FILE_NOTIFY_INFORMATION structures, and can
-    iterate over those structures, decoding them into Event objects.
-    """
-
-    def __init__(self, size=1024):
-        self.size = size
-        self.buffer = create_string_buffer(size)
-        address = addressof(self.buffer)
-        _assert(address & 3 == 0, "address 0x%X returned by create_string_buffer is not DWORD-aligned" % (address,))
-        self.data = None
-
-    def read_changes(self, hDirectory, recursive, filter):
-        bytes_returned = DWORD(0)
-        r = ReadDirectoryChangesW(hDirectory,
-                                  self.buffer,
-                                  self.size,
-                                  recursive,
-                                  filter,
-                                  byref(bytes_returned),
-                                  None,  # NULL -> no overlapped I/O
-                                  None   # NULL -> no completion routine
-                                 )
-        if r == 0:
-            self.data = None
-            raise WinError(get_last_error())
-        self.data = self.buffer.raw[:bytes_returned.value]
-
-    def __iter__(self):
-        # Iterator implemented as generator: 
-        if self.data is None:
-            return
-        pos = 0
-        while True:
-            bytes = self._read_dword(pos+8)
-            s = Event(self._read_dword(pos+4),
-                      self.data[pos+12 : pos+12+bytes].decode('utf-16-le'))
-            Message.log(message_type="fni", info=repr(s))
-
-            next_entry_offset = self._read_dword(pos)
-            yield s
-            if next_entry_offset == 0:
-                break
-            pos = pos + next_entry_offset
-
-    def _read_dword(self, i):
-        # little-endian
-        return ( ord(self.data[i])          |
-                (ord(self.data[i+1]) <<  8) |
-                (ord(self.data[i+2]) << 16) |
-                (ord(self.data[i+3]) << 24))
-
-
-def _open_directory(path_u):
-    hDirectory = CreateFileW(path_u,
-                             FILE_LIST_DIRECTORY,         # access rights
-                             FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,
-                                                          # don't prevent other processes from accessing
-                             None,                        # no security descriptor
-                             OPEN_EXISTING,               # directory must already exist
-                             FILE_FLAG_BACKUP_SEMANTICS,  # necessary to open a directory
-                             None                         # no template file
-                            )
-    if hDirectory == INVALID_HANDLE_VALUE:
-        e = WinError(get_last_error())
-        raise OSError("Opening directory %s gave WinError: %s" % (quote_output(path_u), e))
-    return hDirectory
-
-
-def simple_test():
-    path_u = u"test"
-    filter = FILE_NOTIFY_CHANGE_FILE_NAME | FILE_NOTIFY_CHANGE_DIR_NAME | FILE_NOTIFY_CHANGE_LAST_WRITE
-    recursive = TRUE
-
-    hDirectory = _open_directory(path_u)
-    fni = FileNotifyInformation()
-    print("Waiting...")
-    while True:
-        fni.read_changes(hDirectory, recursive, filter)
-        print(repr(fni.data))
-        for info in fni:
-            print(info)
-
-def medium_test():
-    from twisted.python.filepath import FilePath
-
-    def print_(*event):
-        print(event)
-
-    notifier = INotify()
-    notifier.set_pending_delay(1.0)
-    IN_EXCL_UNLINK = long(0x04000000)
-    mask = (  IN_CREATE
-            | IN_CLOSE_WRITE
-            | IN_MOVED_TO
-            | IN_MOVED_FROM
-            | IN_DELETE
-            | IN_ONLYDIR
-            | IN_EXCL_UNLINK
-    )
-    notifier.watch(FilePath(u"test"), mask, callbacks=[print_], recursive=True)
-    notifier.startReading()
-    reactor.run()
-
-
-NOT_STARTED = "NOT_STARTED"
-STARTED     = "STARTED"
-STOPPING    = "STOPPING"
-STOPPED     = "STOPPED"
-
-class INotify(PollMixin):
-    def __init__(self):
-        self._state = NOT_STARTED
-        self._filter = None
-        self._callbacks = None
-        self._hDirectory = None
-        self._path = None
-        self._pending = set()
-        self._pending_delay = 1.0
-        self._pending_call = None
-        self.recursive_includes_new_subdirectories = True
-
-    def set_pending_delay(self, delay):
-        self._pending_delay = delay
-
-    def startReading(self):
-        deferToThread(self._thread)
-        return self.poll(lambda: self._state != NOT_STARTED)
-
-    def stopReading(self):
-        # FIXME race conditions
-        if self._state != STOPPED:
-            self._state = STOPPING
-        if self._pending_call:
-            self._pending_call.cancel()
-            self._pending_call = None
-
-    def wait_until_stopped(self):
-        try:
-            fileutil.write(os.path.join(self._path.path, u".ignore-me"), "")
-        except IOError:
-            pass
-        return self.poll(lambda: self._state == STOPPED)
-
-    def watch(self, path, mask=IN_WATCH_MASK, autoAdd=False, callbacks=None, recursive=False):
-        precondition(self._state == NOT_STARTED, "watch() can only be called before startReading()", state=self._state)
-        precondition(self._filter is None, "only one watch is supported")
-        precondition(isinstance(autoAdd, bool), autoAdd=autoAdd)
-        precondition(isinstance(recursive, bool), recursive=recursive)
-        #precondition(autoAdd == recursive, "need autoAdd and recursive to be the same", autoAdd=autoAdd, recursive=recursive)
-
-        self._path = path
-        path_u = path.path
-        if not isinstance(path_u, unicode):
-            path_u = path_u.decode(sys.getfilesystemencoding())
-            _assert(isinstance(path_u, unicode), path_u=path_u)
-
-        self._filter = FILE_NOTIFY_CHANGE_FILE_NAME | FILE_NOTIFY_CHANGE_DIR_NAME | FILE_NOTIFY_CHANGE_LAST_WRITE
-
-        if mask & (IN_ACCESS | IN_CLOSE_NOWRITE | IN_OPEN):
-            self._filter = self._filter | FILE_NOTIFY_CHANGE_LAST_ACCESS
-        if mask & IN_ATTRIB:
-            self._filter = self._filter | FILE_NOTIFY_CHANGE_ATTRIBUTES | FILE_NOTIFY_CHANGE_SECURITY
-
-        self._recursive = TRUE if recursive else FALSE
-        self._callbacks = callbacks or []
-        self._hDirectory = _open_directory(path_u)
-
-    def _thread(self):
-        try:
-            _assert(self._filter is not None, "no watch set")
-
-            # To call Twisted or Tahoe APIs, use reactor.callFromThread as described in
-            # .
-
-            fni = FileNotifyInformation()
-
-            while True:
-                self._state = STARTED
-                action = start_action(
-                    action_type=u"read-changes",
-                    directory=self._path.path,
-                    recursive=self._recursive,
-                    filter=self._filter,
-                )
-                try:
-                    with action:
-                        fni.read_changes(self._hDirectory, self._recursive, self._filter)
-                except WindowsError as e:
-                    self._state = STOPPING
-
-                if self._check_stop():
-                    return
-                for info in fni:
-                    path = self._path.preauthChild(info.filename)  # FilePath with Unicode path
-                    if info.action == FILE_ACTION_MODIFIED and path.isdir():
-                        Message.log(
-                            message_type=u"filtering-out",
-                            info=repr(info),
-                        )
-                        continue
-                    else:
-                        Message.log(
-                            message_type=u"processing",
-                            info=repr(info),
-                        )
-                    #mask = _action_to_inotify_mask.get(info.action, IN_CHANGED)
-
-                    @log_call(
-                        action_type=MAYBE_NOTIFY.action_type,
-                        include_args=[],
-                        include_result=False,
-                    )
-                    def _do_pending_calls():
-                        event_mask = IN_CHANGED
-                        self._pending_call = None
-                        for path1 in self._pending:
-                            if self._callbacks:
-                                for cb in self._callbacks:
-                                    try:
-                                        with CALLBACK(inotify_events=event_mask):
-                                            cb(None, path1, event_mask)
-                                    except Exception as e2:
-                                        log.err(e2)
-                        self._pending = set()
-
-                    def _maybe_notify(path2):
-                        if path2 not in self._pending:
-                            self._pending.add(path2)
-                        if self._state not in [STOPPING, STOPPED]:
-                            _do_pending_calls()
-#                        if self._pending_call is None and self._state not in [STOPPING, STOPPED]:
-#                            self._pending_call = reactor.callLater(self._pending_delay, _do_pending_calls)
-
-                    reactor.callFromThread(_maybe_notify, path)
-                    if self._check_stop():
-                        return
-        except Exception as e:
-            log.err(e)
-            self._state = STOPPED
-            raise
-
-    def _check_stop(self):
-        if self._state == STOPPING:
-            hDirectory = self._hDirectory
-            self._callbacks = None
-            self._hDirectory = None
-            CloseHandle(hDirectory)
-            self._state = STOPPED
-            if self._pending_call:
-                self._pending_call.cancel()
-                self._pending_call = None
-
-        return self._state == STOPPED

From 6f1f30e4bf5bf1e5c825e92786541213360dabdc Mon Sep 17 00:00:00 2001
From: Jean-Paul Calderone 
Date: Thu, 13 Feb 2020 12:15:09 -0500
Subject: [PATCH 07/25] pyflakes cleanups

---
 src/allmydata/test/test_client.py  | 2 +-
 src/allmydata/test/web/test_web.py | 2 --
 src/allmydata/util/eliotutil.py    | 2 --
 3 files changed, 1 insertion(+), 5 deletions(-)

diff --git a/src/allmydata/test/test_client.py b/src/allmydata/test/test_client.py
index 85d508093..41a44d5a6 100644
--- a/src/allmydata/test/test_client.py
+++ b/src/allmydata/test/test_client.py
@@ -39,7 +39,7 @@ from testtools.twistedsupport import (
 import allmydata
 import allmydata.util.log
 
-from allmydata.node import OldConfigError, OldConfigOptionError, UnescapedHashError, _Config, create_node_dir
+from allmydata.node import OldConfigError, UnescapedHashError, _Config, create_node_dir
 from allmydata.frontends.auth import NeedRootcapLookupScheme
 from allmydata.version_checks import (
     get_package_versions_string,
diff --git a/src/allmydata/test/web/test_web.py b/src/allmydata/test/web/test_web.py
index 3e0967811..f84923521 100644
--- a/src/allmydata/test/web/test_web.py
+++ b/src/allmydata/test/web/test_web.py
@@ -3,7 +3,6 @@ from __future__ import print_function
 import os.path, re, urllib, time, cgi
 import json
 import treq
-import mock
 
 from bs4 import BeautifulSoup
 
@@ -63,7 +62,6 @@ from ..common_web import (
 )
 from allmydata.client import _Client, SecretHolder
 from .common import unknown_rwcap, unknown_rocap, unknown_immcap, FAVICON_MARKUP
-from ..status import FakeStatus
 
 # create a fake uploader/downloader, and a couple of fake dirnodes, then
 # create a webserver that works against them
diff --git a/src/allmydata/util/eliotutil.py b/src/allmydata/util/eliotutil.py
index 16ab373e0..f6f40945d 100644
--- a/src/allmydata/util/eliotutil.py
+++ b/src/allmydata/util/eliotutil.py
@@ -42,8 +42,6 @@ from attr.validators import (
 from eliot import (
     ILogger,
     Message,
-    Field,
-    ActionType,
     FileDestination,
     add_destinations,
     remove_destination,

From 25edce62576b14bcc14e2d5f66aca23462c55a7c Mon Sep 17 00:00:00 2001
From: Jean-Paul Calderone 
Date: Mon, 13 Apr 2020 12:13:45 -0400
Subject: [PATCH 08/25] Don't create the magic-folder directories for each node

---
 integration/conftest.py | 10 ----------
 1 file changed, 10 deletions(-)

diff --git a/integration/conftest.py b/integration/conftest.py
index a85bc15f6..5395d7c5f 100644
--- a/integration/conftest.py
+++ b/integration/conftest.py
@@ -332,11 +332,6 @@ def storage_nodes(reactor, temp_dir, introducer, introducer_furl, flog_gatherer,
 @pytest.fixture(scope='session')
 @log_call(action_type=u"integration:alice", include_args=[], include_result=False)
 def alice(reactor, temp_dir, introducer_furl, flog_gatherer, storage_nodes, request):
-    try:
-        mkdir(join(temp_dir, 'magic-alice'))
-    except OSError:
-        pass
-
     process = pytest_twisted.blockon(
         _create_node(
             reactor, request, temp_dir, introducer_furl, flog_gatherer, "alice",
@@ -351,11 +346,6 @@ def alice(reactor, temp_dir, introducer_furl, flog_gatherer, storage_nodes, requ
 @pytest.fixture(scope='session')
 @log_call(action_type=u"integration:bob", include_args=[], include_result=False)
 def bob(reactor, temp_dir, introducer_furl, flog_gatherer, storage_nodes, request):
-    try:
-        mkdir(join(temp_dir, 'magic-bob'))
-    except OSError:
-        pass
-
     process = pytest_twisted.blockon(
         _create_node(
             reactor, request, temp_dir, introducer_furl, flog_gatherer, "bob",

From e336d977dd4c0cd71ecf817c66b4609aeead908e Mon Sep 17 00:00:00 2001
From: Jean-Paul Calderone 
Date: Mon, 13 Apr 2020 12:19:41 -0400
Subject: [PATCH 09/25] Filter push trigger down to master branch

---
 .github/workflows/ci.yml | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index a59581180..5d41a78b4 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -2,6 +2,8 @@ name: CI
 
 on:
   push:
+    branches:
+      - "master"
   pull_request:
 
 jobs:

From 2c6b402e0adfe042a1195d07c1ea77e209a926ca Mon Sep 17 00:00:00 2001
From: Jean-Paul Calderone 
Date: Mon, 13 Apr 2020 12:20:03 -0400
Subject: [PATCH 10/25] news fragment

---
 newsfragments/3300.minor | 0
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 newsfragments/3300.minor

diff --git a/newsfragments/3300.minor b/newsfragments/3300.minor
new file mode 100644
index 000000000..e69de29bb

From c73953d94aa391a4170691c8e85fa8a123b24221 Mon Sep 17 00:00:00 2001
From: Jean-Paul Calderone 
Date: Tue, 14 Apr 2020 08:40:48 -0400
Subject: [PATCH 11/25] Capture and report Foolscap Tub Listener creation stack
 on error

---
 src/allmydata/test/__init__.py | 38 ++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/src/allmydata/test/__init__.py b/src/allmydata/test/__init__.py
index 804d37935..af55be353 100644
--- a/src/allmydata/test/__init__.py
+++ b/src/allmydata/test/__init__.py
@@ -52,6 +52,44 @@ def _configure_hypothesis():
     settings.load_profile(profile_name)
 _configure_hypothesis()
 
+def logging_for_pb_listener():
+    """
+    Make Foolscap listen error reports include Listener creation stack
+    information.
+    """
+    from traceback import extract_stack, format_list
+    from foolscap.pb import Listener
+    from twisted.python.log import err
+    from twisted.application import service
+
+    original__init__ = Listener.__init__
+    def _listener__init__(self, *a, **kw):
+        original__init__(self, *a, **kw)
+        # Capture the stack here, where Listener is instantiated.  This is
+        # likely to explain what code is responsible for this Listener, useful
+        # information to have when the Listener eventually fails to listen.
+        self._creation_stack = extract_stack()
+
+    # Override the Foolscap implementation with one that has an errback
+    def _listener_startService(self):
+        service.Service.startService(self)
+        d = self._ep.listen(self)
+        def _listening(lp):
+            self._lp = lp
+        d.addCallbacks(
+            _listening,
+            # Make sure that this listen failure is reported promptly and with
+            # the creation stack.
+            err,
+            errbackArgs=(
+                "Listener created at {}".format(
+                    "".join(format_list(self._creation_stack)),
+                ),
+            ),
+        )
+    Listener.__init__ = _listener__init__
+    Listener.startService = _listener_startService
+logging_for_pb_listener()
 
 import sys
 if sys.platform == "win32":

From 47bef6038f8213fde691a4b9e8ee7861d9747fd5 Mon Sep 17 00:00:00 2001
From: Jean-Paul Calderone 
Date: Tue, 14 Apr 2020 08:41:10 -0400
Subject: [PATCH 12/25] news fragment

---
 newsfragments/3302.minor | 0
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 newsfragments/3302.minor

diff --git a/newsfragments/3302.minor b/newsfragments/3302.minor
new file mode 100644
index 000000000..e69de29bb

From f89ad002f87666b148e31d2b0654160bacfddcf9 Mon Sep 17 00:00:00 2001
From: Sajith Sasidharan 
Date: Tue, 14 Apr 2020 16:14:07 -0400
Subject: [PATCH 13/25] Use 2.7-buster Docker image for pypy

Fix failing PyPy CI job.

Fixes: ticket:3299
---
 .circleci/Dockerfile.pypy | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/.circleci/Dockerfile.pypy b/.circleci/Dockerfile.pypy
index fd32c1f0d..471611ecc 100644
--- a/.circleci/Dockerfile.pypy
+++ b/.circleci/Dockerfile.pypy
@@ -1,4 +1,4 @@
-FROM pypy:2.7-7.1.1-jessie
+FROM pypy:2.7-buster
 
 ENV WHEELHOUSE_PATH /tmp/wheelhouse
 ENV VIRTUALENV_PATH /tmp/venv

From 6e6cdb353e902db6ed5e6c491a31b42a6061ae06 Mon Sep 17 00:00:00 2001
From: Sajith Sasidharan 
Date: Tue, 14 Apr 2020 16:29:17 -0400
Subject: [PATCH 14/25] Add newsfragment

---
 newsfragments/3299.minor | 0
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 newsfragments/3299.minor

diff --git a/newsfragments/3299.minor b/newsfragments/3299.minor
new file mode 100644
index 000000000..e69de29bb

From 4fc6840c109216c35b56dbbe8f0d07ea6a9fa026 Mon Sep 17 00:00:00 2001
From: Sajith Sasidharan 
Date: Tue, 14 Apr 2020 17:35:44 -0400
Subject: [PATCH 15/25] Build Docker images for pypy 2.7-buster on CircleCI

---
 .circleci/config.yml | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/.circleci/config.yml b/.circleci/config.yml
index 41e4ea7fa..61ed12a5d 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -27,8 +27,8 @@ workflows:
 
       - "nixos-19.09"
 
-      # Test against PyPy 2.7/7.1.1
-      - "pypy2.7-7.1"
+      # Test against PyPy 2.7
+      - "pypy2.7-buster"
 
       # Other assorted tasks and configurations
       - "lint"
@@ -69,7 +69,7 @@ workflows:
       - "build-image-fedora-29"
       - "build-image-centos-8"
       - "build-image-slackware-14.2"
-      - "build-image-pypy-2.7-7.1.1-jessie"
+      - "build-image-pypy-2.7-buster"
 
 
 jobs:
@@ -198,10 +198,10 @@ jobs:
         user: "nobody"
 
 
-  pypy2.7-7.1:
+  pypy2.7-buster:
     <<: *DEBIAN
     docker:
-      - image: "tahoelafsci/pypy:2.7-7.1.1-jessie"
+      - image: "tahoelafsci/pypy:2.7-buster"
         user: "nobody"
 
     environment:
@@ -513,9 +513,9 @@ jobs:
       TAG: "14.2"
 
 
-  build-image-pypy-2.7-7.1.1-jessie:
+  build-image-pypy-2.7-buster:
     <<: *BUILD_IMAGE
 
     environment:
       DISTRO: "pypy"
-      TAG: "2.7-7.1.1-jessie"
+      TAG: "2.7-buster"

From 72ca4e8c53191aabe6599ac72c113106552f1b8f Mon Sep 17 00:00:00 2001
From: Jean-Paul Calderone 
Date: Wed, 15 Apr 2020 08:54:50 -0400
Subject: [PATCH 16/25] move the imports

---
 src/allmydata/test/__init__.py | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/src/allmydata/test/__init__.py b/src/allmydata/test/__init__.py
index af55be353..fb62e438f 100644
--- a/src/allmydata/test/__init__.py
+++ b/src/allmydata/test/__init__.py
@@ -1,3 +1,8 @@
+from traceback import extract_stack, format_list
+from foolscap.pb import Listener
+from twisted.python.log import err
+from twisted.application import service
+
 
 from foolscap.logging.incident import IncidentQualifier
 class NonQualifier(IncidentQualifier, object):
@@ -57,11 +62,6 @@ def logging_for_pb_listener():
     Make Foolscap listen error reports include Listener creation stack
     information.
     """
-    from traceback import extract_stack, format_list
-    from foolscap.pb import Listener
-    from twisted.python.log import err
-    from twisted.application import service
-
     original__init__ = Listener.__init__
     def _listener__init__(self, *a, **kw):
         original__init__(self, *a, **kw)

From d4d8ce4e13d6f5b9103c18c1da722fe26f0f0bde Mon Sep 17 00:00:00 2001
From: Jean-Paul Calderone 
Date: Wed, 15 Apr 2020 08:55:00 -0400
Subject: [PATCH 17/25] missing front matter

---
 src/allmydata/test/__init__.py | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/src/allmydata/test/__init__.py b/src/allmydata/test/__init__.py
index fb62e438f..60d9b87d6 100644
--- a/src/allmydata/test/__init__.py
+++ b/src/allmydata/test/__init__.py
@@ -1,3 +1,18 @@
+# Tahoe-LAFS -- secure, distributed storage grid
+#
+# Copyright © 2020 The Tahoe-LAFS Software Foundation
+#
+# This file is part of Tahoe-LAFS.
+#
+# See the docs/about.rst file for licensing information.
+
+"""
+Some setup that should apply across the entire test suite.
+
+Rather than defining interesting APIs for other code to use, this just causes
+some side-effects which make things better when the test suite runs.
+"""
+
 from traceback import extract_stack, format_list
 from foolscap.pb import Listener
 from twisted.python.log import err

From 71aa17ab0bbb3371f43c205aeda8092867c300b2 Mon Sep 17 00:00:00 2001
From: Jean-Paul Calderone 
Date: Wed, 15 Apr 2020 09:07:23 -0400
Subject: [PATCH 18/25] I assume it's utf-8

---
 src/allmydata/test/__init__.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/allmydata/test/__init__.py b/src/allmydata/test/__init__.py
index 60d9b87d6..abbde919f 100644
--- a/src/allmydata/test/__init__.py
+++ b/src/allmydata/test/__init__.py
@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 # Tahoe-LAFS -- secure, distributed storage grid
 #
 # Copyright © 2020 The Tahoe-LAFS Software Foundation

From 30d0953bcf1585fb91f7df1b9742d4f8f35eaa18 Mon Sep 17 00:00:00 2001
From: Jean-Paul Calderone 
Date: Thu, 16 Apr 2020 08:24:15 -0400
Subject: [PATCH 19/25] Disable the Eliot test suite

---
 nix/eliot.nix | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/nix/eliot.nix b/nix/eliot.nix
index df41b3a61..c5975e990 100644
--- a/nix/eliot.nix
+++ b/nix/eliot.nix
@@ -12,15 +12,14 @@ buildPythonPackage rec {
   postPatch = ''
     substituteInPlace setup.py \
       --replace "boltons >= 19.0.1" boltons
-    # depends on eliot.prettyprint._main which we don't have here.
-    rm eliot/tests/test_prettyprint.py
-
-    # Fails intermittently.
-    substituteInPlace eliot/tests/test_validation.py \
-      --replace "def test_omitLoggerFromActionType" "def xtest_omitLoggerFromActionType" \
-      --replace "def test_logCallsDefaultLoggerWrite" "def xtest_logCallsDefaultLoggerWrite"
   '';
 
+  # A seemingly random subset of the test suite fails intermittently.  After
+  # Tahoe-LAFS is ported to Python 3 we can update to a newer Eliot and, if
+  # the test suite continues to fail, maybe it will be more likely that we can
+  # have upstream fix it for us.
+  doCheck = false;
+
   checkInputs = [ testtools pytest hypothesis ];
   propagatedBuildInputs = [ zope_interface pyrsistent boltons ];
 

From d308577397a1635ad49e22f16f7d0a760ae9453b Mon Sep 17 00:00:00 2001
From: Jean-Paul Calderone 
Date: Thu, 16 Apr 2020 08:24:31 -0400
Subject: [PATCH 20/25] Fixes: ticket:3303

---
 newsfragments/3303.minor | 0
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 newsfragments/3303.minor

diff --git a/newsfragments/3303.minor b/newsfragments/3303.minor
new file mode 100644
index 000000000..e69de29bb

From 652b3f5bc016b765b865fae7428220e8e125bdd5 Mon Sep 17 00:00:00 2001
From: Sajith Sasidharan 
Date: Thu, 16 Apr 2020 17:23:46 -0400
Subject: [PATCH 21/25] Use twisted.web.template to render Statistics page

---
 src/allmydata/web/statistics.xhtml | 18 +++----
 src/allmydata/web/status.py        | 77 +++++++++++++++++++-----------
 2 files changed, 59 insertions(+), 36 deletions(-)

diff --git a/src/allmydata/web/statistics.xhtml b/src/allmydata/web/statistics.xhtml
index 11f5c9643..c35aafce7 100644
--- a/src/allmydata/web/statistics.xhtml
+++ b/src/allmydata/web/statistics.xhtml
@@ -1,27 +1,27 @@
-
+
   
     Tahoe-LAFS - Operational Statistics
     
     
     
   
-  
+  
 
 
    Operational Statistics
    
 
 
    General
    
 
 
      
-  
    • Load Average: 
      • 
-  
    • Peak Load: 
      • 
-  
    • Files Uploaded (immutable): 
      • 
-  
    • Files Downloaded (immutable): 
      • 
-  
    • Files Published (mutable): 
      • 
-  
    • Files Retrieved (mutable): 
      • 
+  
    • Load Average: 
      • 
+  
    • Peak Load: 
      • 
+  
    • Files Uploaded (immutable): 
      • 
+  
    • Files Downloaded (immutable): 
      • 
+  
    • Files Published (mutable): 
      • 
+  
    • Files Retrieved (mutable): 
      • 
 
    
 
 
    Raw Stats:
    
-
    +
     
 
    Return to the Welcome Page
    
 
diff --git a/src/allmydata/web/status.py b/src/allmydata/web/status.py
index 67c2fffdb..7983ca438 100644
--- a/src/allmydata/web/status.py
+++ b/src/allmydata/web/status.py
@@ -2,7 +2,15 @@
 import pprint, itertools, hashlib
 import json
 from twisted.internet import defer
+from twisted.python.filepath import FilePath
 from twisted.web.resource import Resource
+from twisted.web.template import (
+    Element,
+    XMLFile,
+    renderer,
+    renderElement,
+    tags
+)
 from nevow import rend, tags as T
 from allmydata.util import base32, idlib
 from allmydata.web.common import (
@@ -14,6 +22,7 @@ from allmydata.web.common import (
     compute_rate,
     render_time,
     MultiFormatPage,
+    MultiFormatResource,
 )
 from allmydata.interfaces import IUploadStatus, IDownloadStatus, \
      IPublishStatus, IRetrieveStatus, IServermapUpdaterStatus
@@ -1164,52 +1173,66 @@ class HelperStatus(MultiFormatPage):
     def render_upload_bytes_encoded(self, ctx, data):
         return str(data["chk_upload_helper.encoded_bytes"])
 
-
-class Statistics(MultiFormatPage):
-    docFactory = getxmlfile("statistics.xhtml")
+# Render "/statistics" page.
+class Statistics(MultiFormatResource):
 
     def __init__(self, provider):
-        rend.Page.__init__(self, provider)
-        self.provider = provider
+        super(Statistics, self).__init__()
+        self._provider = provider
+
+    def render_HTML(self, req):
+        return renderElement(req, StatisticsElement(self._provider))
 
     def render_JSON(self, req):
-        stats = self.provider.get_stats()
+        stats = self._provider.get_stats()
         req.setHeader("content-type", "text/plain")
         return json.dumps(stats, indent=1) + "\n"
 
-    def data_get_stats(self, ctx, data):
-        return self.provider.get_stats()
+class StatisticsElement(Element):
 
-    def render_load_average(self, ctx, data):
-        return str(data["stats"].get("load_monitor.avg_load"))
+    loader = XMLFile(FilePath(__file__).sibling("statistics.xhtml"))
 
-    def render_peak_load(self, ctx, data):
-        return str(data["stats"].get("load_monitor.max_load"))
+    def __init__(self, provider):
+        super(StatisticsElement, self).__init__()
+        self._stats = provider.get_stats()
 
-    def render_uploads(self, ctx, data):
-        files = data["counters"].get("uploader.files_uploaded", 0)
-        bytes = data["counters"].get("uploader.bytes_uploaded", 0)
+    @renderer
+    def load_average(self, req, tag):
+        return str(self._stats["stats"].get("load_monitor.avg_load"))
+
+    @renderer
+    def peak_load(self, req, tag):
+        return str(self._stats["stats"].get("load_monitor.max_load"))
+
+    @renderer
+    def uploads(self, req, tag):
+        files = self._stats["counters"].get("uploader.files_uploaded", 0)
+        bytes = self._stats["counters"].get("uploader.bytes_uploaded", 0)
         return ("%s files / %s bytes (%s)" %
                 (files, bytes, abbreviate_size(bytes)))
 
-    def render_downloads(self, ctx, data):
-        files = data["counters"].get("downloader.files_downloaded", 0)
-        bytes = data["counters"].get("downloader.bytes_downloaded", 0)
+    @renderer
+    def downloads(self, req, tag):
+        files = self._stats["counters"].get("downloader.files_downloaded", 0)
+        bytes = self._stats["counters"].get("downloader.bytes_downloaded", 0)
         return ("%s files / %s bytes (%s)" %
                 (files, bytes, abbreviate_size(bytes)))
 
-    def render_publishes(self, ctx, data):
-        files = data["counters"].get("mutable.files_published", 0)
-        bytes = data["counters"].get("mutable.bytes_published", 0)
+    @renderer
+    def publishes(self, req, tag):
+        files = self._stats["counters"].get("mutable.files_published", 0)
+        bytes = self._stats["counters"].get("mutable.bytes_published", 0)
         return "%s files / %s bytes (%s)" % (files, bytes,
                                              abbreviate_size(bytes))
 
-    def render_retrieves(self, ctx, data):
-        files = data["counters"].get("mutable.files_retrieved", 0)
-        bytes = data["counters"].get("mutable.bytes_retrieved", 0)
+    @renderer
+    def retrieves(self, req, tag):
+        files = self._stats["counters"].get("mutable.files_retrieved", 0)
+        bytes = self._stats["counters"].get("mutable.bytes_retrieved", 0)
         return "%s files / %s bytes (%s)" % (files, bytes,
                                              abbreviate_size(bytes))
 
-    def render_raw(self, ctx, data):
-        raw = pprint.pformat(data)
-        return ctx.tag[raw]
+    @renderer
+    def raw(self, req, tag):
+        raw = pprint.pformat(self._stats)
+        return tag(raw)

From 1e3ba25cc3a1f431bb041af5b433d61f02425ab0 Mon Sep 17 00:00:00 2001
From: Sajith Sasidharan 
Date: Thu, 16 Apr 2020 17:25:21 -0400
Subject: [PATCH 22/25] Add some documentation to Statistics page

---
 src/allmydata/web/status.py | 28 +++++++++++++++++++++++++++-
 1 file changed, 27 insertions(+), 1 deletion(-)

diff --git a/src/allmydata/web/status.py b/src/allmydata/web/status.py
index 7983ca438..e826173f2 100644
--- a/src/allmydata/web/status.py
+++ b/src/allmydata/web/status.py
@@ -9,7 +9,6 @@ from twisted.web.template import (
     XMLFile,
     renderer,
     renderElement,
-    tags
 )
 from nevow import rend, tags as T
 from allmydata.util import base32, idlib
@@ -1175,6 +1174,11 @@ class HelperStatus(MultiFormatPage):
 
 # Render "/statistics" page.
 class Statistics(MultiFormatResource):
+    """Class that renders "/statistics" page.
+
+    :param _allmydata.stats.StatsProvider provider: node statistics
+           provider.
+    """
 
     def __init__(self, provider):
         super(Statistics, self).__init__()
@@ -1194,6 +1198,28 @@ class StatisticsElement(Element):
 
     def __init__(self, provider):
         super(StatisticsElement, self).__init__()
+        # provider.get_stats() returns a dict of the below form, for
+        # example (there's often more data than this):
+        #
+        #  {
+        #    'stats': {
+        #      'storage_server.disk_used': 809601609728,
+        #      'storage_server.accepting_immutable_shares': 1,
+        #      'storage_server.disk_free_for_root': 131486851072,
+        #      'storage_server.reserved_space': 1000000000,
+        #      'node.uptime': 0.16520118713378906,
+        #      'storage_server.disk_total': 941088460800,
+        #      'cpu_monitor.total': 0.004513999999999907,
+        #      'storage_server.disk_avail': 82610759168,
+        #      'storage_server.allocated': 0,
+        #      'storage_server.disk_free_for_nonroot': 83610759168 },
+        #    'counters': {
+        #      'uploader.files_uploaded': 0,
+        #      'uploader.bytes_uploaded': 0,
+        #       ... }
+        #  }
+        #
+        # Note that `counters` can be empty.
         self._stats = provider.get_stats()
 
     @renderer

From a0eb00b2cc186be6c8c462e13348013ef109c350 Mon Sep 17 00:00:00 2001
From: Sajith Sasidharan 
Date: Thu, 16 Apr 2020 17:26:21 -0400
Subject: [PATCH 23/25] Wrap renderer return values in tags

---
 src/allmydata/web/status.py | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/src/allmydata/web/status.py b/src/allmydata/web/status.py
index e826173f2..e1ae15d9d 100644
--- a/src/allmydata/web/status.py
+++ b/src/allmydata/web/status.py
@@ -1224,39 +1224,39 @@ class StatisticsElement(Element):
 
     @renderer
     def load_average(self, req, tag):
-        return str(self._stats["stats"].get("load_monitor.avg_load"))
+        return tag(str(self._stats["stats"].get("load_monitor.avg_load")))
 
     @renderer
     def peak_load(self, req, tag):
-        return str(self._stats["stats"].get("load_monitor.max_load"))
+        return tag(str(self._stats["stats"].get("load_monitor.max_load")))
 
     @renderer
     def uploads(self, req, tag):
         files = self._stats["counters"].get("uploader.files_uploaded", 0)
         bytes = self._stats["counters"].get("uploader.bytes_uploaded", 0)
-        return ("%s files / %s bytes (%s)" %
-                (files, bytes, abbreviate_size(bytes)))
+        return tag(("%s files / %s bytes (%s)" %
+                    (files, bytes, abbreviate_size(bytes))))
 
     @renderer
     def downloads(self, req, tag):
         files = self._stats["counters"].get("downloader.files_downloaded", 0)
         bytes = self._stats["counters"].get("downloader.bytes_downloaded", 0)
-        return ("%s files / %s bytes (%s)" %
-                (files, bytes, abbreviate_size(bytes)))
+        return tag("%s files / %s bytes (%s)" %
+                   (files, bytes, abbreviate_size(bytes)))
 
     @renderer
     def publishes(self, req, tag):
         files = self._stats["counters"].get("mutable.files_published", 0)
         bytes = self._stats["counters"].get("mutable.bytes_published", 0)
-        return "%s files / %s bytes (%s)" % (files, bytes,
-                                             abbreviate_size(bytes))
+        return tag("%s files / %s bytes (%s)" % (files, bytes,
+                                                 abbreviate_size(bytes)))
 
     @renderer
     def retrieves(self, req, tag):
         files = self._stats["counters"].get("mutable.files_retrieved", 0)
         bytes = self._stats["counters"].get("mutable.bytes_retrieved", 0)
-        return "%s files / %s bytes (%s)" % (files, bytes,
-                                             abbreviate_size(bytes))
+        return tag("%s files / %s bytes (%s)" % (files, bytes,
+                                                 abbreviate_size(bytes)))
 
     @renderer
     def raw(self, req, tag):

From d844a8f89f50df391a8a6e217c0bc76f8e149cd5 Mon Sep 17 00:00:00 2001
From: Sajith Sasidharan 
Date: Thu, 16 Apr 2020 17:29:54 -0400
Subject: [PATCH 24/25] Use twisted transparent tag to render data

This way the test suite will be happy, and we won't have extraneous
`
    ` tags in the rendered page.
---
 src/allmydata/web/statistics.xhtml | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/src/allmydata/web/statistics.xhtml b/src/allmydata/web/statistics.xhtml
index c35aafce7..42376079d 100644
--- a/src/allmydata/web/statistics.xhtml
+++ b/src/allmydata/web/statistics.xhtml
@@ -11,14 +11,14 @@
 
 
    General
    
 
-
      
-  
    • Load Average: 
      • 
-  
    • Peak Load: 
      • 
-  
    • Files Uploaded (immutable): 
      • 
-  
    • Files Downloaded (immutable): 
      • 
-  
    • Files Published (mutable): 
      • 
-  
    • Files Retrieved (mutable): 
      • 
-
    
+    
      
+      
    • Load Average: 
      • 
+      
    • Peak Load: 
      • 
+      
    • Files Uploaded (immutable): 
      • 
+      
    • Files Downloaded (immutable): 
      • 
+      
    • Files Published (mutable): 
      • 
+      
    • Files Retrieved (mutable): 
      • 
+    
    
 
 
    Raw Stats:
    
 
    
From 225bb5854a93e0ba8244346cda79b289c2f7d937 Mon Sep 17 00:00:00 2001
From: Sajith Sasidharan 
Date: Tue, 24 Mar 2020 17:36:15 -0400
Subject: [PATCH 25/25] Add empty news fragment re. Statistics page rendering

---
 newsfragments/3292.minor | 0
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 newsfragments/3292.minor

diff --git a/newsfragments/3292.minor b/newsfragments/3292.minor
new file mode 100644
index 000000000..e69de29bb