Rename drop-upload to Magic Folder. fixes ticket:2405

Signed-off-by: Daira Hopwood <daira@jacaranda.org>
This commit is contained in:
Daira Hopwood 2015-07-21 00:42:15 +01:00
parent 3120499069
commit e68b09b081
5 changed files with 150 additions and 147 deletions

View File

@ -1,8 +1,8 @@
.. -*- coding: utf-8-with-signature -*- .. -*- coding: utf-8-with-signature -*-
=============================== ================================
Tahoe-LAFS Drop-Upload Frontend Tahoe-LAFS Magic Folder Frontend
=============================== ================================
1. `Introduction`_ 1. `Introduction`_
2. `Configuration`_ 2. `Configuration`_
@ -12,7 +12,7 @@ Tahoe-LAFS Drop-Upload Frontend
Introduction Introduction
============ ============
The drop-upload frontend allows an upload to a Tahoe-LAFS grid to be triggered The Magic Folder frontend allows an upload to a Tahoe-LAFS grid to be triggered
automatically whenever a file is created or changed in a specific local automatically whenever a file is created or changed in a specific local
directory. It currently works on Linux and Windows. directory. It currently works on Linux and Windows.
@ -30,18 +30,18 @@ suggestions to improve its usability, functionality, and reliability.
Configuration Configuration
============= =============
The drop-upload frontend runs as part of a gateway node. To set it up, you The Magic Folder frontend runs as part of a gateway node. To set it up, you
need to choose the local directory to monitor for file changes, and a mutable need to choose the local directory to monitor for file changes, and a mutable
directory on the grid to which files will be uploaded. directory on the grid to which files will be uploaded.
These settings are configured in the ``[drop_upload]`` section of the These settings are configured in the ``[magic_folder]`` section of the
gateway's ``tahoe.cfg`` file. gateway's ``tahoe.cfg`` file.
``[drop_upload]`` ``[magic_folder]``
``enabled = (boolean, optional)`` ``enabled = (boolean, optional)``
If this is ``True``, drop-upload will be enabled. The default value is If this is ``True``, Magic Folder will be enabled. The default value is
``False``. ``False``.
``local.directory = (UTF-8 path)`` ``local.directory = (UTF-8 path)``
@ -51,10 +51,11 @@ gateway's ``tahoe.cfg`` file.
in UTF-8 regardless of the system's filesystem encoding. Relative paths in UTF-8 regardless of the system's filesystem encoding. Relative paths
will be interpreted starting from the node's base directory. will be interpreted starting from the node's base directory.
In addition, the file ``private/drop_upload_dircap`` must contain a In addition:
writecap pointing to an existing mutable directory to be used as the target * the file ``private/magic_folder_dircap`` must contain a writecap pointing
of uploads. It will start with ``URI:DIR2:``, and cannot include an alias to an existing mutable directory to be used as the target of uploads.
or path. It will start with ``URI:DIR2:``, and cannot include an alias or path.
* the file ``private/collective_dircap`` must contain a readcap
After setting the above fields and starting or restarting the gateway, After setting the above fields and starting or restarting the gateway,
you can confirm that the feature is working by copying a file into the you can confirm that the feature is working by copying a file into the
@ -91,11 +92,11 @@ The only way to determine whether uploads have failed is to look at the
'Operational Statistics' page linked from the Welcome page. This only shows 'Operational Statistics' page linked from the Welcome page. This only shows
a count of failures, not the names of files. Uploads are never retried. a count of failures, not the names of files. Uploads are never retried.
The drop-upload frontend performs its uploads sequentially (i.e. it waits The Magic Folder frontend performs its uploads sequentially (i.e. it waits
until each upload is finished before starting the next), even when there until each upload is finished before starting the next), even when there
would be enough memory and bandwidth to efficiently perform them in parallel. would be enough memory and bandwidth to efficiently perform them in parallel.
A drop-upload can occur in parallel with an upload by a different frontend, A Magic Folder upload can occur in parallel with an upload by a different
though. (`#1459`_) frontend, though. (`#1459`_)
On Linux, if there are a large number of near-simultaneous file creation or On Linux, if there are a large number of near-simultaneous file creation or
change events (greater than the number specified in the file change events (greater than the number specified in the file
@ -126,8 +127,8 @@ up-to-date. (`#1440`_)
Files deleted from the local directory will not be unlinked from the upload Files deleted from the local directory will not be unlinked from the upload
directory. (`#1710`_) directory. (`#1710`_)
The ``private/drop_upload_dircap`` file cannot use an alias or path to The ``private/magic_folder_dircap`` and ``private/collective_dircap`` files
specify the upload directory. (`#1711`_) cannot use an alias or path to specify the upload directory. (`#1711`_)
Files are always uploaded as immutable. If there is an existing mutable file Files are always uploaded as immutable. If there is an existing mutable file
of the same name in the upload directory, it will be unlinked and replaced of the same name in the upload directory, it will be unlinked and replaced
@ -146,7 +147,7 @@ The expected encoding is that printed by
On Windows, local directories with non-ASCII names are not currently working. On Windows, local directories with non-ASCII names are not currently working.
(`#2219`_) (`#2219`_)
On Windows, when a node has drop-upload enabled, it is unresponsive to Ctrl-C On Windows, when a node has Magic Folder enabled, it is unresponsive to Ctrl-C
(it can only be killed using Task Manager or similar). (`#2218`_) (it can only be killed using Task Manager or similar). (`#2218`_)
.. _`#1105`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1105 .. _`#1105`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1105

View File

@ -151,7 +151,7 @@ class Client(node.Node, pollmixin.PollMixin):
# ControlServer and Helper are attached after Tub startup # ControlServer and Helper are attached after Tub startup
self.init_ftp_server() self.init_ftp_server()
self.init_sftp_server() self.init_sftp_server()
self.init_drop_uploader() self.init_magic_folder()
# If the node sees an exit_trigger file, it will poll every second to see # If the node sees an exit_trigger file, it will poll every second to see
# whether the file still exists, and what its mtime is. If the file does not # whether the file still exists, and what its mtime is. If the file does not
@ -492,33 +492,33 @@ class Client(node.Node, pollmixin.PollMixin):
sftp_portstr, pubkey_file, privkey_file) sftp_portstr, pubkey_file, privkey_file)
s.setServiceParent(self) s.setServiceParent(self)
def init_drop_uploader(self): def init_magic_folder(self):
if self.get_config("drop_upload", "enabled", False, boolean=True): if self.get_config("drop_upload", "enabled", False, boolean=True):
if self.get_config("drop_upload", "upload.dircap", None): raise OldConfigOptionError("The [drop_upload] section must be renamed to [magic_folder].\n"
raise OldConfigOptionError("The [drop_upload]upload.dircap option is no longer supported; please " "See docs/frontends/magic-folder.rst for more information.")
"put the cap in a 'private/drop_upload_dircap' file, and delete this option.")
upload_dircap = self.get_or_create_private_config("drop_upload_dircap") if self.get_config("magic_folder", "enabled", False, boolean=True):
local_dir_config = self.get_config("drop_upload", "local.directory").decode("utf-8") upload_dircap = self.get_or_create_private_config("magic_folder_dircap")
local_dir_config = self.get_config("magic_folder", "local.directory").decode("utf-8")
local_dir = abspath_expanduser_unicode(local_dir_config, base=self.basedir) local_dir = abspath_expanduser_unicode(local_dir_config, base=self.basedir)
try: try:
from allmydata.frontends import drop_upload from allmydata.frontends import magic_folder
dbfile = os.path.join(self.basedir, "private", "magicfolderdb.sqlite") dbfile = os.path.join(self.basedir, "private", "magicfolderdb.sqlite")
dbfile = abspath_expanduser_unicode(dbfile) dbfile = abspath_expanduser_unicode(dbfile)
parent_dircap_path = os.path.join(self.basedir, "private", "magic_folder_parent_dircap") collective_dircap_path = os.path.join(self.basedir, "private", "collective_dircap")
parent_dircap_path = abspath_expanduser_unicode(parent_dircap_path) collective_dircap_path = abspath_expanduser_unicode(collective_dircap_path)
parent_dircap = fileutil.read(parent_dircap_path).strip() collective_dircap = fileutil.read(collective_dircap_path).strip()
s = drop_upload.DropUploader(self, upload_dircap, parent_dircap, local_dir, dbfile) s = magic_folder.MagicFolder(self, upload_dircap, collective_dircap, local_dir, dbfile)
s.setServiceParent(self) s.setServiceParent(self)
s.startService() s.startService()
# start processing the upload queue when we've connected to enough servers # start processing the upload queue when we've connected to enough servers
self.upload_ready_d.addCallback(s.upload_ready) self.upload_ready_d.addCallback(s.upload_ready)
except Exception, e: except Exception, e:
self.log("couldn't start drop-uploader: %r", args=(e,)) self.log("couldn't start Magic Folder: %r", args=(e,))
def _check_exit_trigger(self, exit_trigger_file): def _check_exit_trigger(self, exit_trigger_file):
if os.path.exists(exit_trigger_file): if os.path.exists(exit_trigger_file):

View File

@ -38,10 +38,10 @@ def get_inotify_module():
raise raise
class DropUploader(service.MultiService): class MagicFolder(service.MultiService):
name = 'drop-upload' name = 'magic-folder'
def __init__(self, client, upload_dircap, parent_dircap, local_dir, dbfile, inotify=None, def __init__(self, client, upload_dircap, collective_dircap, local_dir, dbfile, inotify=None,
pending_delay=1.0): pending_delay=1.0):
precondition_abspath(local_dir) precondition_abspath(local_dir)
@ -61,20 +61,20 @@ class DropUploader(service.MultiService):
self._inotify = inotify or get_inotify_module() self._inotify = inotify or get_inotify_module()
if not self._local_path.exists(): if not self._local_path.exists():
raise AssertionError("The '[drop_upload] local.directory' parameter was %s " raise AssertionError("The '[magic_folder] local.directory' parameter was %s "
"but there is no directory at that location." "but there is no directory at that location."
% quote_local_unicode_path(local_dir)) % quote_local_unicode_path(local_dir))
if not self._local_path.isdir(): if not self._local_path.isdir():
raise AssertionError("The '[drop_upload] local.directory' parameter was %s " raise AssertionError("The '[magic_folder] local.directory' parameter was %s "
"but the thing at that location is not a directory." "but the thing at that location is not a directory."
% quote_local_unicode_path(local_dir)) % quote_local_unicode_path(local_dir))
# TODO: allow a path rather than a cap URI. # TODO: allow a path rather than a cap URI.
self._parent = self._client.create_node_from_uri(upload_dircap) self._upload_dirnode = self._client.create_node_from_uri(upload_dircap)
if not IDirectoryNode.providedBy(self._parent): if not IDirectoryNode.providedBy(self._upload_dirnode):
raise AssertionError("The URI in 'private/drop_upload_dircap' does not refer to a directory.") raise AssertionError("The URI in 'private/magic_folder_dircap' does not refer to a directory.")
if self._parent.is_unknown() or self._parent.is_readonly(): if self._upload_dirnode.is_unknown() or self._upload_dirnode.is_readonly():
raise AssertionError("The URI in 'private/drop_upload_dircap' is not a writecap to a directory.") raise AssertionError("The URI in 'private/magic_folder_dircap' is not a writecap to a directory.")
self._processed_callback = lambda ign: None self._processed_callback = lambda ign: None
self._ignore_count = 0 self._ignore_count = 0
@ -150,7 +150,7 @@ class DropUploader(service.MultiService):
self._scan(self._local_dir) self._scan(self._local_dir)
self._stats_provider.count('drop_upload.dirs_monitored', 1) self._stats_provider.count('magic_folder.dirs_monitored', 1)
return d return d
def upload_ready(self): def upload_ready(self):
@ -163,7 +163,7 @@ class DropUploader(service.MultiService):
def _append_to_deque(self, path): def _append_to_deque(self, path):
self._upload_deque.append(path) self._upload_deque.append(path)
self._pending.add(path) self._pending.add(path)
self._stats_provider.count('drop_upload.objects_queued', 1) self._stats_provider.count('magic_folder.objects_queued', 1)
if self.is_upload_ready: if self.is_upload_ready:
reactor.callLater(0, self._turn_deque) reactor.callLater(0, self._turn_deque)
@ -188,16 +188,16 @@ class DropUploader(service.MultiService):
def _add_file(name): def _add_file(name):
u = FileName(path, self._convergence) u = FileName(path, self._convergence)
return self._parent.add_file(name, u, overwrite=True) return self._upload_dirnode.add_file(name, u, overwrite=True)
def _add_dir(name): def _add_dir(name):
self._notifier.watch(to_filepath(path), mask=self.mask, callbacks=[self._notify], recursive=True) self._notifier.watch(to_filepath(path), mask=self.mask, callbacks=[self._notify], recursive=True)
u = Data("", self._convergence) u = Data("", self._convergence)
name += "@_" name += "@_"
d2 = self._parent.add_file(name, u, overwrite=True) d2 = self._upload_dirnode.add_file(name, u, overwrite=True)
def _succeeded(ign): def _succeeded(ign):
self._log("created subdirectory %r" % (path,)) self._log("created subdirectory %r" % (path,))
self._stats_provider.count('drop_upload.directories_created', 1) self._stats_provider.count('magic_folder.directories_created', 1)
def _failed(f): def _failed(f):
self._log("failed to create subdirectory %r" % (path,)) self._log("failed to create subdirectory %r" % (path,))
return f return f
@ -213,7 +213,7 @@ class DropUploader(service.MultiService):
if not os.path.exists(path): if not os.path.exists(path):
self._log("drop-upload: notified object %r disappeared " self._log("drop-upload: notified object %r disappeared "
"(this is normal for temporary objects)" % (path,)) "(this is normal for temporary objects)" % (path,))
self._stats_provider.count('drop_upload.objects_disappeared', 1) self._stats_provider.count('magic_folder.objects_disappeared', 1)
return None return None
elif os.path.islink(path): elif os.path.islink(path):
raise Exception("symlink not being processed") raise Exception("symlink not being processed")
@ -229,7 +229,7 @@ class DropUploader(service.MultiService):
ctime = s[stat.ST_CTIME] ctime = s[stat.ST_CTIME]
mtime = s[stat.ST_MTIME] mtime = s[stat.ST_MTIME]
self._db.did_upload_file(filecap, path, mtime, ctime, size) self._db.did_upload_file(filecap, path, mtime, ctime, size)
self._stats_provider.count('drop_upload.files_uploaded', 1) self._stats_provider.count('magic_folder.files_uploaded', 1)
d2.addCallback(add_db_entry) d2.addCallback(add_db_entry)
return d2 return d2
else: else:
@ -238,12 +238,12 @@ class DropUploader(service.MultiService):
d.addCallback(_maybe_upload) d.addCallback(_maybe_upload)
def _succeeded(res): def _succeeded(res):
self._stats_provider.count('drop_upload.objects_queued', -1) self._stats_provider.count('magic_folder.objects_queued', -1)
self._stats_provider.count('drop_upload.objects_succeeded', 1) self._stats_provider.count('magic_folder.objects_succeeded', 1)
return res return res
def _failed(f): def _failed(f):
self._stats_provider.count('drop_upload.objects_queued', -1) self._stats_provider.count('magic_folder.objects_queued', -1)
self._stats_provider.count('drop_upload.objects_failed', 1) self._stats_provider.count('magic_folder.objects_failed', 1)
self._log("%r while processing %r" % (f, path)) self._log("%r while processing %r" % (f, path))
return f return f
d.addCallbacks(_succeeded, _failed) d.addCallbacks(_succeeded, _failed)
@ -267,7 +267,7 @@ class DropUploader(service.MultiService):
def finish(self, for_tests=False): def finish(self, for_tests=False):
self._notifier.stopReading() self._notifier.stopReading()
self._stats_provider.count('drop_upload.dirs_monitored', -1) self._stats_provider.count('magic_folder.dirs_monitored', -1)
if for_tests and hasattr(self._notifier, 'wait_until_stopped'): if for_tests and hasattr(self._notifier, 'wait_until_stopped'):
return self._notifier.wait_until_stopped() return self._notifier.wait_until_stopped()
else: else:

View File

@ -4,7 +4,7 @@ from twisted.trial import unittest
from twisted.application import service from twisted.application import service
import allmydata import allmydata
import allmydata.frontends.drop_upload import allmydata.frontends.magic_folder
import allmydata.util.log import allmydata.util.log
from allmydata.node import Node, OldConfigError, OldConfigOptionError, MissingConfigEntry, UnescapedHashError from allmydata.node import Node, OldConfigError, OldConfigOptionError, MissingConfigEntry, UnescapedHashError
@ -302,20 +302,21 @@ class Basic(testutil.ReallyEqualMixin, testutil.NonASCIIPathMixin, unittest.Test
_check("helper.furl = None", None) _check("helper.furl = None", None)
_check("helper.furl = pb://blah\n", "pb://blah") _check("helper.furl = pb://blah\n", "pb://blah")
def test_create_drop_uploader(self): def test_create_magic_folder(self):
class MockDropUploader(service.MultiService): class MockMagicFolder(service.MultiService):
name = 'drop-upload' name = 'magic-folder'
def __init__(self, client, upload_dircap, parent_dircap, local_dir, dbfile, inotify=None, def __init__(self, client, upload_dircap, collective_dircap, local_dir, dbfile, inotify=None,
pending_delay=1.0): pending_delay=1.0):
service.MultiService.__init__(self) service.MultiService.__init__(self)
self.client = client self.client = client
self.upload_dircap = upload_dircap self.upload_dircap = upload_dircap
self.collective_dircap = collective_dircap
self.local_dir = local_dir self.local_dir = local_dir
self.dbfile = dbfile self.dbfile = dbfile
self.inotify = inotify self.inotify = inotify
self.patch(allmydata.frontends.drop_upload, 'DropUploader', MockDropUploader) self.patch(allmydata.frontends.magic_folder, 'MagicFolder', MockMagicFolder)
upload_dircap = "URI:DIR2:blah" upload_dircap = "URI:DIR2:blah"
local_dir_u = self.unicode_or_fallback(u"loc\u0101l_dir", u"local_dir") local_dir_u = self.unicode_or_fallback(u"loc\u0101l_dir", u"local_dir")
@ -323,10 +324,10 @@ class Basic(testutil.ReallyEqualMixin, testutil.NonASCIIPathMixin, unittest.Test
config = (BASECONFIG + config = (BASECONFIG +
"[storage]\n" + "[storage]\n" +
"enabled = false\n" + "enabled = false\n" +
"[drop_upload]\n" + "[magic_folder]\n" +
"enabled = true\n") "enabled = true\n")
basedir1 = "test_client.Basic.test_create_drop_uploader1" basedir1 = "test_client.Basic.test_create_magic_folder1"
os.mkdir(basedir1) os.mkdir(basedir1)
fileutil.write(os.path.join(basedir1, "tahoe.cfg"), fileutil.write(os.path.join(basedir1, "tahoe.cfg"),
@ -334,48 +335,49 @@ class Basic(testutil.ReallyEqualMixin, testutil.NonASCIIPathMixin, unittest.Test
self.failUnlessRaises(MissingConfigEntry, client.Client, basedir1) self.failUnlessRaises(MissingConfigEntry, client.Client, basedir1)
fileutil.write(os.path.join(basedir1, "tahoe.cfg"), config) fileutil.write(os.path.join(basedir1, "tahoe.cfg"), config)
fileutil.write(os.path.join(basedir1, "private", "drop_upload_dircap"), "URI:DIR2:blah") fileutil.write(os.path.join(basedir1, "private", "magic_folder_dircap"), "URI:DIR2:blah")
fileutil.write(os.path.join(basedir1, "private", "magic_folder_parent_dircap"), "URI:DIR2:meow") fileutil.write(os.path.join(basedir1, "private", "collective_dircap"), "URI:DIR2:meow")
self.failUnlessRaises(MissingConfigEntry, client.Client, basedir1) self.failUnlessRaises(MissingConfigEntry, client.Client, basedir1)
fileutil.write(os.path.join(basedir1, "tahoe.cfg"), fileutil.write(os.path.join(basedir1, "tahoe.cfg"),
config + "upload.dircap = " + upload_dircap + "\n") config.replace("[magic_folder]\n", "[drop_upload]\n"))
self.failUnlessRaises(OldConfigOptionError, client.Client, basedir1) self.failUnlessRaises(OldConfigOptionError, client.Client, basedir1)
fileutil.write(os.path.join(basedir1, "tahoe.cfg"), fileutil.write(os.path.join(basedir1, "tahoe.cfg"),
config + "local.directory = " + local_dir_utf8 + "\n") config + "local.directory = " + local_dir_utf8 + "\n")
c1 = client.Client(basedir1) c1 = client.Client(basedir1)
uploader = c1.getServiceNamed('drop-upload') magicfolder = c1.getServiceNamed('magic-folder')
self.failUnless(isinstance(uploader, MockDropUploader), uploader) self.failUnless(isinstance(magicfolder, MockMagicFolder), magicfolder)
self.failUnlessReallyEqual(uploader.client, c1) self.failUnlessReallyEqual(magicfolder.client, c1)
self.failUnlessReallyEqual(uploader.upload_dircap, upload_dircap) self.failUnlessReallyEqual(magicfolder.upload_dircap, upload_dircap)
self.failUnlessReallyEqual(os.path.basename(uploader.local_dir), local_dir_u) self.failUnlessReallyEqual(os.path.basename(magicfolder.local_dir), local_dir_u)
self.failUnless(uploader.inotify is None, uploader.inotify) self.failUnless(magicfolder.inotify is None, magicfolder.inotify)
self.failUnless(uploader.running) self.failUnless(magicfolder.running)
class Boom(Exception): class Boom(Exception):
pass pass
def BoomDropUploader(client, upload_dircap, local_dir_utf8, inotify=None): def BoomMagicFolder(self, client, upload_dircap, collective_dircap, local_dir, dbfile,
inotify=None, pending_delay=1.0):
raise Boom() raise Boom()
logged_messages = [] logged_messages = []
def mock_log(*args, **kwargs): def mock_log(*args, **kwargs):
logged_messages.append("%r %r" % (args, kwargs)) logged_messages.append("%r %r" % (args, kwargs))
self.patch(allmydata.util.log, 'msg', mock_log) self.patch(allmydata.util.log, 'msg', mock_log)
self.patch(allmydata.frontends.drop_upload, 'DropUploader', BoomDropUploader) self.patch(allmydata.frontends.magic_folder, 'MagicFolder', BoomMagicFolder)
basedir2 = "test_client.Basic.test_create_drop_uploader2" basedir2 = "test_client.Basic.test_create_magic_folder2"
os.mkdir(basedir2) os.mkdir(basedir2)
os.mkdir(os.path.join(basedir2, "private")) os.mkdir(os.path.join(basedir2, "private"))
fileutil.write(os.path.join(basedir2, "tahoe.cfg"), fileutil.write(os.path.join(basedir2, "tahoe.cfg"),
BASECONFIG + BASECONFIG +
"[drop_upload]\n" + "[magic_folder]\n" +
"enabled = true\n" + "enabled = true\n" +
"local.directory = " + local_dir_utf8 + "\n") "local.directory = " + local_dir_utf8 + "\n")
fileutil.write(os.path.join(basedir2, "private", "drop_upload_dircap"), "URI:DIR2:blah") fileutil.write(os.path.join(basedir2, "private", "magic_folder_dircap"), "URI:DIR2:blah")
fileutil.write(os.path.join(basedir2, "private", "magic_folder_parent_dircap"), "URI:DIR2:meow") fileutil.write(os.path.join(basedir2, "private", "collective_dircap"), "URI:DIR2:meow")
c2 = client.Client(basedir2) c2 = client.Client(basedir2)
self.failUnlessRaises(KeyError, c2.getServiceNamed, 'drop-upload') self.failUnlessRaises(KeyError, c2.getServiceNamed, 'magic-folder')
self.failUnless([True for arg in logged_messages if "Boom" in arg], self.failUnless([True for arg in logged_messages if "Boom" in arg],
logged_messages) logged_messages)

View File

@ -13,13 +13,13 @@ from allmydata.test.no_network import GridTestMixin
from allmydata.test.common_util import ReallyEqualMixin, NonASCIIPathMixin from allmydata.test.common_util import ReallyEqualMixin, NonASCIIPathMixin
from allmydata.test.common import ShouldFailMixin from allmydata.test.common import ShouldFailMixin
from allmydata.frontends import drop_upload from allmydata.frontends import magic_folder
from allmydata.frontends.drop_upload import DropUploader from allmydata.frontends.magic_folder import MagicFolder
from allmydata import backupdb from allmydata import backupdb
from allmydata.util.fileutil import abspath_expanduser_unicode from allmydata.util.fileutil import abspath_expanduser_unicode
class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonASCIIPathMixin): class MagicFolderTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonASCIIPathMixin):
""" """
These tests will be run both with a mock notifier, and (on platforms that support it) These tests will be run both with a mock notifier, and (on platforms that support it)
with the real INotify. with the real INotify.
@ -29,7 +29,7 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
GridTestMixin.setUp(self) GridTestMixin.setUp(self)
temp = self.mktemp() temp = self.mktemp()
self.basedir = abspath_expanduser_unicode(temp.decode(get_filesystem_encoding())) self.basedir = abspath_expanduser_unicode(temp.decode(get_filesystem_encoding()))
self.uploader = None self.magicfolder = None
self.dir_node = None self.dir_node = None
def _get_count(self, name): def _get_count(self, name):
@ -50,20 +50,20 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
self.failUnless(IDirectoryNode.providedBy(n)) self.failUnless(IDirectoryNode.providedBy(n))
self.upload_dirnode = n self.upload_dirnode = n
self.upload_dircap = n.get_uri() self.upload_dircap = n.get_uri()
self.parent_dircap = "abc123" self.collective_dircap = "abc123"
def _create_uploader(self, ign): def _create_magicfolder(self, ign):
dbfile = abspath_expanduser_unicode(u"magicfolderdb.sqlite", base=self.basedir) dbfile = abspath_expanduser_unicode(u"magicfolderdb.sqlite", base=self.basedir)
self.uploader = DropUploader(self.client, self.upload_dircap, self.parent_dircap, self.local_dir, self.magicfolder = MagicFolder(self.client, self.upload_dircap, self.collective_dircap, self.local_dir,
dbfile, inotify=self.inotify, pending_delay=0.2) dbfile, inotify=self.inotify, pending_delay=0.2)
self.uploader.setServiceParent(self.client) self.magicfolder.setServiceParent(self.client)
self.uploader.upload_ready() self.magicfolder.upload_ready()
# Prevent unclean reactor errors. # Prevent unclean reactor errors.
def _cleanup(self, res): def _cleanup(self, res):
d = defer.succeed(None) d = defer.succeed(None)
if self.uploader is not None: if self.magicfolder is not None:
d.addCallback(lambda ign: self.uploader.finish(for_tests=True)) d.addCallback(lambda ign: self.magicfolder.finish(for_tests=True))
d.addCallback(lambda ign: res) d.addCallback(lambda ign: res)
return d return d
@ -100,7 +100,7 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
r = db.check_file(path) r = db.check_file(path)
self.failUnless(r.was_uploaded()) self.failUnless(r.was_uploaded())
def test_uploader_start_service(self): def test_magicfolder_start_service(self):
self.set_up_grid() self.set_up_grid()
self.local_dir = abspath_expanduser_unicode(self.unicode_or_fallback(u"l\u00F8cal_dir", u"local_dir"), self.local_dir = abspath_expanduser_unicode(self.unicode_or_fallback(u"l\u00F8cal_dir", u"local_dir"),
@ -112,10 +112,10 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
d = self.client.create_dirnode() d = self.client.create_dirnode()
d.addCallback(self._made_upload_dir) d.addCallback(self._made_upload_dir)
d.addCallback(self._create_uploader) d.addCallback(self._create_magicfolder)
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.dirs_monitored'), 1)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.dirs_monitored'), 1))
d.addBoth(self._cleanup) d.addBoth(self._cleanup)
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.dirs_monitored'), 0)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.dirs_monitored'), 0))
return d return d
def test_move_tree(self): def test_move_tree(self):
@ -139,46 +139,46 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
d = self.client.create_dirnode() d = self.client.create_dirnode()
d.addCallback(self._made_upload_dir) d.addCallback(self._made_upload_dir)
d.addCallback(self._create_uploader) d.addCallback(self._create_magicfolder)
def _check_move_empty_tree(res): def _check_move_empty_tree(res):
self.mkdir_nonascii(empty_tree_dir) self.mkdir_nonascii(empty_tree_dir)
d2 = defer.Deferred() d2 = defer.Deferred()
self.uploader.set_processed_callback(d2.callback, ignore_count=0) self.magicfolder.set_processed_callback(d2.callback, ignore_count=0)
os.rename(empty_tree_dir, new_empty_tree_dir) os.rename(empty_tree_dir, new_empty_tree_dir)
self.notify(to_filepath(new_empty_tree_dir), self.inotify.IN_MOVED_TO) self.notify(to_filepath(new_empty_tree_dir), self.inotify.IN_MOVED_TO)
return d2 return d2
d.addCallback(_check_move_empty_tree) d.addCallback(_check_move_empty_tree)
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_succeeded'), 1)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_succeeded'), 1))
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.files_uploaded'), 0)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.files_uploaded'), 0))
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_queued'), 0)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_queued'), 0))
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.directories_created'), 1)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.directories_created'), 1))
def _check_move_small_tree(res): def _check_move_small_tree(res):
self.mkdir_nonascii(small_tree_dir) self.mkdir_nonascii(small_tree_dir)
fileutil.write(abspath_expanduser_unicode(u"what", base=small_tree_dir), "say when") fileutil.write(abspath_expanduser_unicode(u"what", base=small_tree_dir), "say when")
d2 = defer.Deferred() d2 = defer.Deferred()
self.uploader.set_processed_callback(d2.callback, ignore_count=1) self.magicfolder.set_processed_callback(d2.callback, ignore_count=1)
os.rename(small_tree_dir, new_small_tree_dir) os.rename(small_tree_dir, new_small_tree_dir)
self.notify(to_filepath(new_small_tree_dir), self.inotify.IN_MOVED_TO) self.notify(to_filepath(new_small_tree_dir), self.inotify.IN_MOVED_TO)
return d2 return d2
d.addCallback(_check_move_small_tree) d.addCallback(_check_move_small_tree)
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_succeeded'), 3)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_succeeded'), 3))
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.files_uploaded'), 1)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.files_uploaded'), 1))
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_queued'), 0)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_queued'), 0))
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.directories_created'), 2)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.directories_created'), 2))
def _check_moved_tree_is_watched(res): def _check_moved_tree_is_watched(res):
d2 = defer.Deferred() d2 = defer.Deferred()
self.uploader.set_processed_callback(d2.callback, ignore_count=0) self.magicfolder.set_processed_callback(d2.callback, ignore_count=0)
fileutil.write(abspath_expanduser_unicode(u"another", base=new_small_tree_dir), "file") fileutil.write(abspath_expanduser_unicode(u"another", base=new_small_tree_dir), "file")
self.notify(to_filepath(abspath_expanduser_unicode(u"another", base=new_small_tree_dir)), self.inotify.IN_CLOSE_WRITE) self.notify(to_filepath(abspath_expanduser_unicode(u"another", base=new_small_tree_dir)), self.inotify.IN_CLOSE_WRITE)
return d2 return d2
d.addCallback(_check_moved_tree_is_watched) d.addCallback(_check_moved_tree_is_watched)
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_succeeded'), 4)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_succeeded'), 4))
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.files_uploaded'), 2)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.files_uploaded'), 2))
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_queued'), 0)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_queued'), 0))
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.directories_created'), 2)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.directories_created'), 2))
# Files that are moved out of the upload directory should no longer be watched. # Files that are moved out of the upload directory should no longer be watched.
def _move_dir_away(ign): def _move_dir_away(ign):
@ -192,10 +192,10 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
return return
d.addCallback(create_file) d.addCallback(create_file)
d.addCallback(lambda ign: time.sleep(1)) d.addCallback(lambda ign: time.sleep(1))
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_succeeded'), 4)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_succeeded'), 4))
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.files_uploaded'), 2)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.files_uploaded'), 2))
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_queued'), 0)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_queued'), 0))
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.directories_created'), 2)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.directories_created'), 2))
d.addBoth(self._cleanup) d.addBoth(self._cleanup)
return d return d
@ -203,9 +203,9 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
def test_persistence(self): def test_persistence(self):
""" """
Perform an upload of a given file and then stop the client. Perform an upload of a given file and then stop the client.
Start a new client and uploader... and verify that the file is NOT uploaded Start a new client and magic-folder service... and verify that the file is NOT uploaded
a second time. This test is meant to test the database persistence along with a second time. This test is meant to test the database persistence along with
the startup and shutdown code paths of the uploader. the startup and shutdown code paths of the magic-folder service.
""" """
self.set_up_grid() self.set_up_grid()
self.local_dir = abspath_expanduser_unicode(u"test_persistence", base=self.basedir) self.local_dir = abspath_expanduser_unicode(u"test_persistence", base=self.basedir)
@ -215,18 +215,18 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
self.stats_provider = self.client.stats_provider self.stats_provider = self.client.stats_provider
d = self.client.create_dirnode() d = self.client.create_dirnode()
d.addCallback(self._made_upload_dir) d.addCallback(self._made_upload_dir)
d.addCallback(self._create_uploader) d.addCallback(self._create_magicfolder)
def create_file(val): def create_file(val):
d2 = defer.Deferred() d2 = defer.Deferred()
self.uploader.set_processed_callback(d2.callback) self.magicfolder.set_processed_callback(d2.callback)
test_file = abspath_expanduser_unicode(u"what", base=self.local_dir) test_file = abspath_expanduser_unicode(u"what", base=self.local_dir)
fileutil.write(test_file, "meow") fileutil.write(test_file, "meow")
self.notify(to_filepath(test_file), self.inotify.IN_CLOSE_WRITE) self.notify(to_filepath(test_file), self.inotify.IN_CLOSE_WRITE)
return d2 return d2
d.addCallback(create_file) d.addCallback(create_file)
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_succeeded'), 1)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_succeeded'), 1))
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_queued'), 0)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_queued'), 0))
d.addCallback(self._cleanup) d.addCallback(self._cleanup)
def _restart(ign): def _restart(ign):
@ -234,14 +234,14 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
self.client = self.g.clients[0] self.client = self.g.clients[0]
self.stats_provider = self.client.stats_provider self.stats_provider = self.client.stats_provider
d.addCallback(_restart) d.addCallback(_restart)
d.addCallback(self._create_uploader) d.addCallback(self._create_magicfolder)
d.addCallback(lambda ign: time.sleep(3)) d.addCallback(lambda ign: time.sleep(3))
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_succeeded'), 0)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_succeeded'), 0))
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_queued'), 0)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_queued'), 0))
d.addBoth(self._cleanup) d.addBoth(self._cleanup)
return d return d
def test_drop_upload(self): def test_magic_folder(self):
self.set_up_grid() self.set_up_grid()
self.local_dir = os.path.join(self.basedir, self.unicode_or_fallback(u"loc\u0101l_dir", u"local_dir")) self.local_dir = os.path.join(self.basedir, self.unicode_or_fallback(u"loc\u0101l_dir", u"local_dir"))
self.mkdir_nonascii(self.local_dir) self.mkdir_nonascii(self.local_dir)
@ -252,7 +252,7 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
d = self.client.create_dirnode() d = self.client.create_dirnode()
d.addCallback(self._made_upload_dir) d.addCallback(self._made_upload_dir)
d.addCallback(self._create_uploader) d.addCallback(self._create_magicfolder)
# Write something short enough for a LIT file. # Write something short enough for a LIT file.
d.addCallback(lambda ign: self._check_file(u"short", "test")) d.addCallback(lambda ign: self._check_file(u"short", "test"))
@ -271,20 +271,20 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
d.addCallback(lambda ign: self._check_file(name_u, "test"*100)) d.addCallback(lambda ign: self._check_file(name_u, "test"*100))
# TODO: test that causes an upload failure. # TODO: test that causes an upload failure.
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.files_failed'), 0)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.files_failed'), 0))
d.addBoth(self._cleanup) d.addBoth(self._cleanup)
return d return d
def _check_file(self, name_u, data, temporary=False): def _check_file(self, name_u, data, temporary=False):
previously_uploaded = self._get_count('drop_upload.objects_succeeded') previously_uploaded = self._get_count('magic_folder.objects_succeeded')
previously_disappeared = self._get_count('drop_upload.objects_disappeared') previously_disappeared = self._get_count('magic_folder.objects_disappeared')
d = defer.Deferred() d = defer.Deferred()
# Note: this relies on the fact that we only get one IN_CLOSE_WRITE notification per file # Note: this relies on the fact that we only get one IN_CLOSE_WRITE notification per file
# (otherwise we would get a defer.AlreadyCalledError). Should we be relying on that? # (otherwise we would get a defer.AlreadyCalledError). Should we be relying on that?
self.uploader.set_processed_callback(d.callback) self.magicfolder.set_processed_callback(d.callback)
path_u = abspath_expanduser_unicode(name_u, base=self.local_dir) path_u = abspath_expanduser_unicode(name_u, base=self.local_dir)
path = to_filepath(path_u) path = to_filepath(path_u)
@ -307,28 +307,28 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
if temporary: if temporary:
d.addCallback(lambda ign: self.shouldFail(NoSuchChildError, 'temp file not uploaded', None, d.addCallback(lambda ign: self.shouldFail(NoSuchChildError, 'temp file not uploaded', None,
self.upload_dirnode.get, name_u)) self.upload_dirnode.get, name_u))
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_disappeared'), d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_disappeared'),
previously_disappeared + 1)) previously_disappeared + 1))
else: else:
d.addCallback(lambda ign: self.upload_dirnode.get(name_u)) d.addCallback(lambda ign: self.upload_dirnode.get(name_u))
d.addCallback(download_to_data) d.addCallback(download_to_data)
d.addCallback(lambda actual_data: self.failUnlessReallyEqual(actual_data, data)) d.addCallback(lambda actual_data: self.failUnlessReallyEqual(actual_data, data))
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_succeeded'), d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_succeeded'),
previously_uploaded + 1)) previously_uploaded + 1))
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_queued'), 0)) d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_queued'), 0))
return d return d
class MockTest(DropUploadTestMixin, unittest.TestCase): class MockTest(MagicFolderTestMixin, unittest.TestCase):
"""This can run on any platform, and even if twisted.internet.inotify can't be imported.""" """This can run on any platform, and even if twisted.internet.inotify can't be imported."""
def setUp(self): def setUp(self):
DropUploadTestMixin.setUp(self) MagicFolderTestMixin.setUp(self)
self.inotify = fake_inotify self.inotify = fake_inotify
def notify(self, path, mask): def notify(self, path, mask):
self.uploader._notifier.event(path, mask) self.magicfolder._notifier.event(path, mask)
def test_errors(self): def test_errors(self):
self.set_up_grid() self.set_up_grid()
@ -348,37 +348,37 @@ class MockTest(DropUploadTestMixin, unittest.TestCase):
readonly_dircap = n.get_readonly_uri() readonly_dircap = n.get_readonly_uri()
self.shouldFail(AssertionError, 'nonexistent local.directory', 'there is no directory', self.shouldFail(AssertionError, 'nonexistent local.directory', 'there is no directory',
DropUploader, client, upload_dircap, '', doesnotexist, magicfolderdb, inotify=fake_inotify) MagicFolder, client, upload_dircap, '', doesnotexist, magicfolderdb, inotify=fake_inotify)
self.shouldFail(AssertionError, 'non-directory local.directory', 'is not a directory', self.shouldFail(AssertionError, 'non-directory local.directory', 'is not a directory',
DropUploader, client, upload_dircap, '', not_a_dir, magicfolderdb, inotify=fake_inotify) MagicFolder, client, upload_dircap, '', not_a_dir, magicfolderdb, inotify=fake_inotify)
self.shouldFail(AssertionError, 'bad upload.dircap', 'does not refer to a directory', self.shouldFail(AssertionError, 'bad upload.dircap', 'does not refer to a directory',
DropUploader, client, 'bad', '', errors_dir, magicfolderdb, inotify=fake_inotify) MagicFolder, client, 'bad', '', errors_dir, magicfolderdb, inotify=fake_inotify)
self.shouldFail(AssertionError, 'non-directory upload.dircap', 'does not refer to a directory', self.shouldFail(AssertionError, 'non-directory upload.dircap', 'does not refer to a directory',
DropUploader, client, 'URI:LIT:foo', '', errors_dir, magicfolderdb, inotify=fake_inotify) MagicFolder, client, 'URI:LIT:foo', '', errors_dir, magicfolderdb, inotify=fake_inotify)
self.shouldFail(AssertionError, 'readonly upload.dircap', 'is not a writecap to a directory', self.shouldFail(AssertionError, 'readonly upload.dircap', 'is not a writecap to a directory',
DropUploader, client, readonly_dircap, '', errors_dir, magicfolderdb, inotify=fake_inotify) MagicFolder, client, readonly_dircap, '', errors_dir, magicfolderdb, inotify=fake_inotify)
def _not_implemented(): def _not_implemented():
raise NotImplementedError("blah") raise NotImplementedError("blah")
self.patch(drop_upload, 'get_inotify_module', _not_implemented) self.patch(magic_folder, 'get_inotify_module', _not_implemented)
self.shouldFail(NotImplementedError, 'unsupported', 'blah', self.shouldFail(NotImplementedError, 'unsupported', 'blah',
DropUploader, client, upload_dircap, '', errors_dir, magicfolderdb) MagicFolder, client, upload_dircap, '', errors_dir, magicfolderdb)
d.addCallback(_check_errors) d.addCallback(_check_errors)
return d return d
class RealTest(DropUploadTestMixin, unittest.TestCase): class RealTest(MagicFolderTestMixin, unittest.TestCase):
"""This is skipped unless both Twisted and the platform support inotify.""" """This is skipped unless both Twisted and the platform support inotify."""
def setUp(self): def setUp(self):
DropUploadTestMixin.setUp(self) MagicFolderTestMixin.setUp(self)
self.inotify = drop_upload.get_inotify_module() self.inotify = magic_folder.get_inotify_module()
def notify(self, path, mask): def notify(self, path, mask):
# Writing to the filesystem causes the notification. # Writing to the filesystem causes the notification.
pass pass
try: try:
drop_upload.get_inotify_module() magic_folder.get_inotify_module()
except NotImplementedError: except NotImplementedError:
RealTest.skip = "Drop-upload support can only be tested for-real on an OS that supports inotify or equivalent." RealTest.skip = "Magic Folder support can only be tested for-real on an OS that supports inotify or equivalent."