mirror of
https://github.com/tahoe-lafs/tahoe-lafs.git
synced 2025-01-20 11:38:52 +00:00
Rename drop-upload to Magic Folder. fixes ticket:2405
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
This commit is contained in:
parent
3120499069
commit
e68b09b081
@ -1,8 +1,8 @@
|
|||||||
.. -*- coding: utf-8-with-signature -*-
|
.. -*- coding: utf-8-with-signature -*-
|
||||||
|
|
||||||
===============================
|
================================
|
||||||
Tahoe-LAFS Drop-Upload Frontend
|
Tahoe-LAFS Magic Folder Frontend
|
||||||
===============================
|
================================
|
||||||
|
|
||||||
1. `Introduction`_
|
1. `Introduction`_
|
||||||
2. `Configuration`_
|
2. `Configuration`_
|
||||||
@ -12,7 +12,7 @@ Tahoe-LAFS Drop-Upload Frontend
|
|||||||
Introduction
|
Introduction
|
||||||
============
|
============
|
||||||
|
|
||||||
The drop-upload frontend allows an upload to a Tahoe-LAFS grid to be triggered
|
The Magic Folder frontend allows an upload to a Tahoe-LAFS grid to be triggered
|
||||||
automatically whenever a file is created or changed in a specific local
|
automatically whenever a file is created or changed in a specific local
|
||||||
directory. It currently works on Linux and Windows.
|
directory. It currently works on Linux and Windows.
|
||||||
|
|
||||||
@ -30,18 +30,18 @@ suggestions to improve its usability, functionality, and reliability.
|
|||||||
Configuration
|
Configuration
|
||||||
=============
|
=============
|
||||||
|
|
||||||
The drop-upload frontend runs as part of a gateway node. To set it up, you
|
The Magic Folder frontend runs as part of a gateway node. To set it up, you
|
||||||
need to choose the local directory to monitor for file changes, and a mutable
|
need to choose the local directory to monitor for file changes, and a mutable
|
||||||
directory on the grid to which files will be uploaded.
|
directory on the grid to which files will be uploaded.
|
||||||
|
|
||||||
These settings are configured in the ``[drop_upload]`` section of the
|
These settings are configured in the ``[magic_folder]`` section of the
|
||||||
gateway's ``tahoe.cfg`` file.
|
gateway's ``tahoe.cfg`` file.
|
||||||
|
|
||||||
``[drop_upload]``
|
``[magic_folder]``
|
||||||
|
|
||||||
``enabled = (boolean, optional)``
|
``enabled = (boolean, optional)``
|
||||||
|
|
||||||
If this is ``True``, drop-upload will be enabled. The default value is
|
If this is ``True``, Magic Folder will be enabled. The default value is
|
||||||
``False``.
|
``False``.
|
||||||
|
|
||||||
``local.directory = (UTF-8 path)``
|
``local.directory = (UTF-8 path)``
|
||||||
@ -51,10 +51,11 @@ gateway's ``tahoe.cfg`` file.
|
|||||||
in UTF-8 regardless of the system's filesystem encoding. Relative paths
|
in UTF-8 regardless of the system's filesystem encoding. Relative paths
|
||||||
will be interpreted starting from the node's base directory.
|
will be interpreted starting from the node's base directory.
|
||||||
|
|
||||||
In addition, the file ``private/drop_upload_dircap`` must contain a
|
In addition:
|
||||||
writecap pointing to an existing mutable directory to be used as the target
|
* the file ``private/magic_folder_dircap`` must contain a writecap pointing
|
||||||
of uploads. It will start with ``URI:DIR2:``, and cannot include an alias
|
to an existing mutable directory to be used as the target of uploads.
|
||||||
or path.
|
It will start with ``URI:DIR2:``, and cannot include an alias or path.
|
||||||
|
* the file ``private/collective_dircap`` must contain a readcap
|
||||||
|
|
||||||
After setting the above fields and starting or restarting the gateway,
|
After setting the above fields and starting or restarting the gateway,
|
||||||
you can confirm that the feature is working by copying a file into the
|
you can confirm that the feature is working by copying a file into the
|
||||||
@ -91,11 +92,11 @@ The only way to determine whether uploads have failed is to look at the
|
|||||||
'Operational Statistics' page linked from the Welcome page. This only shows
|
'Operational Statistics' page linked from the Welcome page. This only shows
|
||||||
a count of failures, not the names of files. Uploads are never retried.
|
a count of failures, not the names of files. Uploads are never retried.
|
||||||
|
|
||||||
The drop-upload frontend performs its uploads sequentially (i.e. it waits
|
The Magic Folder frontend performs its uploads sequentially (i.e. it waits
|
||||||
until each upload is finished before starting the next), even when there
|
until each upload is finished before starting the next), even when there
|
||||||
would be enough memory and bandwidth to efficiently perform them in parallel.
|
would be enough memory and bandwidth to efficiently perform them in parallel.
|
||||||
A drop-upload can occur in parallel with an upload by a different frontend,
|
A Magic Folder upload can occur in parallel with an upload by a different
|
||||||
though. (`#1459`_)
|
frontend, though. (`#1459`_)
|
||||||
|
|
||||||
On Linux, if there are a large number of near-simultaneous file creation or
|
On Linux, if there are a large number of near-simultaneous file creation or
|
||||||
change events (greater than the number specified in the file
|
change events (greater than the number specified in the file
|
||||||
@ -126,8 +127,8 @@ up-to-date. (`#1440`_)
|
|||||||
Files deleted from the local directory will not be unlinked from the upload
|
Files deleted from the local directory will not be unlinked from the upload
|
||||||
directory. (`#1710`_)
|
directory. (`#1710`_)
|
||||||
|
|
||||||
The ``private/drop_upload_dircap`` file cannot use an alias or path to
|
The ``private/magic_folder_dircap`` and ``private/collective_dircap`` files
|
||||||
specify the upload directory. (`#1711`_)
|
cannot use an alias or path to specify the upload directory. (`#1711`_)
|
||||||
|
|
||||||
Files are always uploaded as immutable. If there is an existing mutable file
|
Files are always uploaded as immutable. If there is an existing mutable file
|
||||||
of the same name in the upload directory, it will be unlinked and replaced
|
of the same name in the upload directory, it will be unlinked and replaced
|
||||||
@ -146,7 +147,7 @@ The expected encoding is that printed by
|
|||||||
On Windows, local directories with non-ASCII names are not currently working.
|
On Windows, local directories with non-ASCII names are not currently working.
|
||||||
(`#2219`_)
|
(`#2219`_)
|
||||||
|
|
||||||
On Windows, when a node has drop-upload enabled, it is unresponsive to Ctrl-C
|
On Windows, when a node has Magic Folder enabled, it is unresponsive to Ctrl-C
|
||||||
(it can only be killed using Task Manager or similar). (`#2218`_)
|
(it can only be killed using Task Manager or similar). (`#2218`_)
|
||||||
|
|
||||||
.. _`#1105`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1105
|
.. _`#1105`: https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1105
|
@ -151,7 +151,7 @@ class Client(node.Node, pollmixin.PollMixin):
|
|||||||
# ControlServer and Helper are attached after Tub startup
|
# ControlServer and Helper are attached after Tub startup
|
||||||
self.init_ftp_server()
|
self.init_ftp_server()
|
||||||
self.init_sftp_server()
|
self.init_sftp_server()
|
||||||
self.init_drop_uploader()
|
self.init_magic_folder()
|
||||||
|
|
||||||
# If the node sees an exit_trigger file, it will poll every second to see
|
# If the node sees an exit_trigger file, it will poll every second to see
|
||||||
# whether the file still exists, and what its mtime is. If the file does not
|
# whether the file still exists, and what its mtime is. If the file does not
|
||||||
@ -492,33 +492,33 @@ class Client(node.Node, pollmixin.PollMixin):
|
|||||||
sftp_portstr, pubkey_file, privkey_file)
|
sftp_portstr, pubkey_file, privkey_file)
|
||||||
s.setServiceParent(self)
|
s.setServiceParent(self)
|
||||||
|
|
||||||
def init_drop_uploader(self):
|
def init_magic_folder(self):
|
||||||
if self.get_config("drop_upload", "enabled", False, boolean=True):
|
if self.get_config("drop_upload", "enabled", False, boolean=True):
|
||||||
if self.get_config("drop_upload", "upload.dircap", None):
|
raise OldConfigOptionError("The [drop_upload] section must be renamed to [magic_folder].\n"
|
||||||
raise OldConfigOptionError("The [drop_upload]upload.dircap option is no longer supported; please "
|
"See docs/frontends/magic-folder.rst for more information.")
|
||||||
"put the cap in a 'private/drop_upload_dircap' file, and delete this option.")
|
|
||||||
|
|
||||||
upload_dircap = self.get_or_create_private_config("drop_upload_dircap")
|
if self.get_config("magic_folder", "enabled", False, boolean=True):
|
||||||
local_dir_config = self.get_config("drop_upload", "local.directory").decode("utf-8")
|
upload_dircap = self.get_or_create_private_config("magic_folder_dircap")
|
||||||
|
local_dir_config = self.get_config("magic_folder", "local.directory").decode("utf-8")
|
||||||
local_dir = abspath_expanduser_unicode(local_dir_config, base=self.basedir)
|
local_dir = abspath_expanduser_unicode(local_dir_config, base=self.basedir)
|
||||||
|
|
||||||
try:
|
try:
|
||||||
from allmydata.frontends import drop_upload
|
from allmydata.frontends import magic_folder
|
||||||
dbfile = os.path.join(self.basedir, "private", "magicfolderdb.sqlite")
|
dbfile = os.path.join(self.basedir, "private", "magicfolderdb.sqlite")
|
||||||
dbfile = abspath_expanduser_unicode(dbfile)
|
dbfile = abspath_expanduser_unicode(dbfile)
|
||||||
|
|
||||||
parent_dircap_path = os.path.join(self.basedir, "private", "magic_folder_parent_dircap")
|
collective_dircap_path = os.path.join(self.basedir, "private", "collective_dircap")
|
||||||
parent_dircap_path = abspath_expanduser_unicode(parent_dircap_path)
|
collective_dircap_path = abspath_expanduser_unicode(collective_dircap_path)
|
||||||
parent_dircap = fileutil.read(parent_dircap_path).strip()
|
collective_dircap = fileutil.read(collective_dircap_path).strip()
|
||||||
|
|
||||||
s = drop_upload.DropUploader(self, upload_dircap, parent_dircap, local_dir, dbfile)
|
s = magic_folder.MagicFolder(self, upload_dircap, collective_dircap, local_dir, dbfile)
|
||||||
s.setServiceParent(self)
|
s.setServiceParent(self)
|
||||||
s.startService()
|
s.startService()
|
||||||
|
|
||||||
# start processing the upload queue when we've connected to enough servers
|
# start processing the upload queue when we've connected to enough servers
|
||||||
self.upload_ready_d.addCallback(s.upload_ready)
|
self.upload_ready_d.addCallback(s.upload_ready)
|
||||||
except Exception, e:
|
except Exception, e:
|
||||||
self.log("couldn't start drop-uploader: %r", args=(e,))
|
self.log("couldn't start Magic Folder: %r", args=(e,))
|
||||||
|
|
||||||
def _check_exit_trigger(self, exit_trigger_file):
|
def _check_exit_trigger(self, exit_trigger_file):
|
||||||
if os.path.exists(exit_trigger_file):
|
if os.path.exists(exit_trigger_file):
|
||||||
|
@ -38,10 +38,10 @@ def get_inotify_module():
|
|||||||
raise
|
raise
|
||||||
|
|
||||||
|
|
||||||
class DropUploader(service.MultiService):
|
class MagicFolder(service.MultiService):
|
||||||
name = 'drop-upload'
|
name = 'magic-folder'
|
||||||
|
|
||||||
def __init__(self, client, upload_dircap, parent_dircap, local_dir, dbfile, inotify=None,
|
def __init__(self, client, upload_dircap, collective_dircap, local_dir, dbfile, inotify=None,
|
||||||
pending_delay=1.0):
|
pending_delay=1.0):
|
||||||
precondition_abspath(local_dir)
|
precondition_abspath(local_dir)
|
||||||
|
|
||||||
@ -61,20 +61,20 @@ class DropUploader(service.MultiService):
|
|||||||
self._inotify = inotify or get_inotify_module()
|
self._inotify = inotify or get_inotify_module()
|
||||||
|
|
||||||
if not self._local_path.exists():
|
if not self._local_path.exists():
|
||||||
raise AssertionError("The '[drop_upload] local.directory' parameter was %s "
|
raise AssertionError("The '[magic_folder] local.directory' parameter was %s "
|
||||||
"but there is no directory at that location."
|
"but there is no directory at that location."
|
||||||
% quote_local_unicode_path(local_dir))
|
% quote_local_unicode_path(local_dir))
|
||||||
if not self._local_path.isdir():
|
if not self._local_path.isdir():
|
||||||
raise AssertionError("The '[drop_upload] local.directory' parameter was %s "
|
raise AssertionError("The '[magic_folder] local.directory' parameter was %s "
|
||||||
"but the thing at that location is not a directory."
|
"but the thing at that location is not a directory."
|
||||||
% quote_local_unicode_path(local_dir))
|
% quote_local_unicode_path(local_dir))
|
||||||
|
|
||||||
# TODO: allow a path rather than a cap URI.
|
# TODO: allow a path rather than a cap URI.
|
||||||
self._parent = self._client.create_node_from_uri(upload_dircap)
|
self._upload_dirnode = self._client.create_node_from_uri(upload_dircap)
|
||||||
if not IDirectoryNode.providedBy(self._parent):
|
if not IDirectoryNode.providedBy(self._upload_dirnode):
|
||||||
raise AssertionError("The URI in 'private/drop_upload_dircap' does not refer to a directory.")
|
raise AssertionError("The URI in 'private/magic_folder_dircap' does not refer to a directory.")
|
||||||
if self._parent.is_unknown() or self._parent.is_readonly():
|
if self._upload_dirnode.is_unknown() or self._upload_dirnode.is_readonly():
|
||||||
raise AssertionError("The URI in 'private/drop_upload_dircap' is not a writecap to a directory.")
|
raise AssertionError("The URI in 'private/magic_folder_dircap' is not a writecap to a directory.")
|
||||||
|
|
||||||
self._processed_callback = lambda ign: None
|
self._processed_callback = lambda ign: None
|
||||||
self._ignore_count = 0
|
self._ignore_count = 0
|
||||||
@ -150,7 +150,7 @@ class DropUploader(service.MultiService):
|
|||||||
|
|
||||||
self._scan(self._local_dir)
|
self._scan(self._local_dir)
|
||||||
|
|
||||||
self._stats_provider.count('drop_upload.dirs_monitored', 1)
|
self._stats_provider.count('magic_folder.dirs_monitored', 1)
|
||||||
return d
|
return d
|
||||||
|
|
||||||
def upload_ready(self):
|
def upload_ready(self):
|
||||||
@ -163,7 +163,7 @@ class DropUploader(service.MultiService):
|
|||||||
def _append_to_deque(self, path):
|
def _append_to_deque(self, path):
|
||||||
self._upload_deque.append(path)
|
self._upload_deque.append(path)
|
||||||
self._pending.add(path)
|
self._pending.add(path)
|
||||||
self._stats_provider.count('drop_upload.objects_queued', 1)
|
self._stats_provider.count('magic_folder.objects_queued', 1)
|
||||||
if self.is_upload_ready:
|
if self.is_upload_ready:
|
||||||
reactor.callLater(0, self._turn_deque)
|
reactor.callLater(0, self._turn_deque)
|
||||||
|
|
||||||
@ -188,16 +188,16 @@ class DropUploader(service.MultiService):
|
|||||||
|
|
||||||
def _add_file(name):
|
def _add_file(name):
|
||||||
u = FileName(path, self._convergence)
|
u = FileName(path, self._convergence)
|
||||||
return self._parent.add_file(name, u, overwrite=True)
|
return self._upload_dirnode.add_file(name, u, overwrite=True)
|
||||||
|
|
||||||
def _add_dir(name):
|
def _add_dir(name):
|
||||||
self._notifier.watch(to_filepath(path), mask=self.mask, callbacks=[self._notify], recursive=True)
|
self._notifier.watch(to_filepath(path), mask=self.mask, callbacks=[self._notify], recursive=True)
|
||||||
u = Data("", self._convergence)
|
u = Data("", self._convergence)
|
||||||
name += "@_"
|
name += "@_"
|
||||||
d2 = self._parent.add_file(name, u, overwrite=True)
|
d2 = self._upload_dirnode.add_file(name, u, overwrite=True)
|
||||||
def _succeeded(ign):
|
def _succeeded(ign):
|
||||||
self._log("created subdirectory %r" % (path,))
|
self._log("created subdirectory %r" % (path,))
|
||||||
self._stats_provider.count('drop_upload.directories_created', 1)
|
self._stats_provider.count('magic_folder.directories_created', 1)
|
||||||
def _failed(f):
|
def _failed(f):
|
||||||
self._log("failed to create subdirectory %r" % (path,))
|
self._log("failed to create subdirectory %r" % (path,))
|
||||||
return f
|
return f
|
||||||
@ -213,7 +213,7 @@ class DropUploader(service.MultiService):
|
|||||||
if not os.path.exists(path):
|
if not os.path.exists(path):
|
||||||
self._log("drop-upload: notified object %r disappeared "
|
self._log("drop-upload: notified object %r disappeared "
|
||||||
"(this is normal for temporary objects)" % (path,))
|
"(this is normal for temporary objects)" % (path,))
|
||||||
self._stats_provider.count('drop_upload.objects_disappeared', 1)
|
self._stats_provider.count('magic_folder.objects_disappeared', 1)
|
||||||
return None
|
return None
|
||||||
elif os.path.islink(path):
|
elif os.path.islink(path):
|
||||||
raise Exception("symlink not being processed")
|
raise Exception("symlink not being processed")
|
||||||
@ -229,7 +229,7 @@ class DropUploader(service.MultiService):
|
|||||||
ctime = s[stat.ST_CTIME]
|
ctime = s[stat.ST_CTIME]
|
||||||
mtime = s[stat.ST_MTIME]
|
mtime = s[stat.ST_MTIME]
|
||||||
self._db.did_upload_file(filecap, path, mtime, ctime, size)
|
self._db.did_upload_file(filecap, path, mtime, ctime, size)
|
||||||
self._stats_provider.count('drop_upload.files_uploaded', 1)
|
self._stats_provider.count('magic_folder.files_uploaded', 1)
|
||||||
d2.addCallback(add_db_entry)
|
d2.addCallback(add_db_entry)
|
||||||
return d2
|
return d2
|
||||||
else:
|
else:
|
||||||
@ -238,12 +238,12 @@ class DropUploader(service.MultiService):
|
|||||||
d.addCallback(_maybe_upload)
|
d.addCallback(_maybe_upload)
|
||||||
|
|
||||||
def _succeeded(res):
|
def _succeeded(res):
|
||||||
self._stats_provider.count('drop_upload.objects_queued', -1)
|
self._stats_provider.count('magic_folder.objects_queued', -1)
|
||||||
self._stats_provider.count('drop_upload.objects_succeeded', 1)
|
self._stats_provider.count('magic_folder.objects_succeeded', 1)
|
||||||
return res
|
return res
|
||||||
def _failed(f):
|
def _failed(f):
|
||||||
self._stats_provider.count('drop_upload.objects_queued', -1)
|
self._stats_provider.count('magic_folder.objects_queued', -1)
|
||||||
self._stats_provider.count('drop_upload.objects_failed', 1)
|
self._stats_provider.count('magic_folder.objects_failed', 1)
|
||||||
self._log("%r while processing %r" % (f, path))
|
self._log("%r while processing %r" % (f, path))
|
||||||
return f
|
return f
|
||||||
d.addCallbacks(_succeeded, _failed)
|
d.addCallbacks(_succeeded, _failed)
|
||||||
@ -267,7 +267,7 @@ class DropUploader(service.MultiService):
|
|||||||
|
|
||||||
def finish(self, for_tests=False):
|
def finish(self, for_tests=False):
|
||||||
self._notifier.stopReading()
|
self._notifier.stopReading()
|
||||||
self._stats_provider.count('drop_upload.dirs_monitored', -1)
|
self._stats_provider.count('magic_folder.dirs_monitored', -1)
|
||||||
if for_tests and hasattr(self._notifier, 'wait_until_stopped'):
|
if for_tests and hasattr(self._notifier, 'wait_until_stopped'):
|
||||||
return self._notifier.wait_until_stopped()
|
return self._notifier.wait_until_stopped()
|
||||||
else:
|
else:
|
@ -4,7 +4,7 @@ from twisted.trial import unittest
|
|||||||
from twisted.application import service
|
from twisted.application import service
|
||||||
|
|
||||||
import allmydata
|
import allmydata
|
||||||
import allmydata.frontends.drop_upload
|
import allmydata.frontends.magic_folder
|
||||||
import allmydata.util.log
|
import allmydata.util.log
|
||||||
|
|
||||||
from allmydata.node import Node, OldConfigError, OldConfigOptionError, MissingConfigEntry, UnescapedHashError
|
from allmydata.node import Node, OldConfigError, OldConfigOptionError, MissingConfigEntry, UnescapedHashError
|
||||||
@ -302,20 +302,21 @@ class Basic(testutil.ReallyEqualMixin, testutil.NonASCIIPathMixin, unittest.Test
|
|||||||
_check("helper.furl = None", None)
|
_check("helper.furl = None", None)
|
||||||
_check("helper.furl = pb://blah\n", "pb://blah")
|
_check("helper.furl = pb://blah\n", "pb://blah")
|
||||||
|
|
||||||
def test_create_drop_uploader(self):
|
def test_create_magic_folder(self):
|
||||||
class MockDropUploader(service.MultiService):
|
class MockMagicFolder(service.MultiService):
|
||||||
name = 'drop-upload'
|
name = 'magic-folder'
|
||||||
|
|
||||||
def __init__(self, client, upload_dircap, parent_dircap, local_dir, dbfile, inotify=None,
|
def __init__(self, client, upload_dircap, collective_dircap, local_dir, dbfile, inotify=None,
|
||||||
pending_delay=1.0):
|
pending_delay=1.0):
|
||||||
service.MultiService.__init__(self)
|
service.MultiService.__init__(self)
|
||||||
self.client = client
|
self.client = client
|
||||||
self.upload_dircap = upload_dircap
|
self.upload_dircap = upload_dircap
|
||||||
|
self.collective_dircap = collective_dircap
|
||||||
self.local_dir = local_dir
|
self.local_dir = local_dir
|
||||||
self.dbfile = dbfile
|
self.dbfile = dbfile
|
||||||
self.inotify = inotify
|
self.inotify = inotify
|
||||||
|
|
||||||
self.patch(allmydata.frontends.drop_upload, 'DropUploader', MockDropUploader)
|
self.patch(allmydata.frontends.magic_folder, 'MagicFolder', MockMagicFolder)
|
||||||
|
|
||||||
upload_dircap = "URI:DIR2:blah"
|
upload_dircap = "URI:DIR2:blah"
|
||||||
local_dir_u = self.unicode_or_fallback(u"loc\u0101l_dir", u"local_dir")
|
local_dir_u = self.unicode_or_fallback(u"loc\u0101l_dir", u"local_dir")
|
||||||
@ -323,10 +324,10 @@ class Basic(testutil.ReallyEqualMixin, testutil.NonASCIIPathMixin, unittest.Test
|
|||||||
config = (BASECONFIG +
|
config = (BASECONFIG +
|
||||||
"[storage]\n" +
|
"[storage]\n" +
|
||||||
"enabled = false\n" +
|
"enabled = false\n" +
|
||||||
"[drop_upload]\n" +
|
"[magic_folder]\n" +
|
||||||
"enabled = true\n")
|
"enabled = true\n")
|
||||||
|
|
||||||
basedir1 = "test_client.Basic.test_create_drop_uploader1"
|
basedir1 = "test_client.Basic.test_create_magic_folder1"
|
||||||
os.mkdir(basedir1)
|
os.mkdir(basedir1)
|
||||||
|
|
||||||
fileutil.write(os.path.join(basedir1, "tahoe.cfg"),
|
fileutil.write(os.path.join(basedir1, "tahoe.cfg"),
|
||||||
@ -334,48 +335,49 @@ class Basic(testutil.ReallyEqualMixin, testutil.NonASCIIPathMixin, unittest.Test
|
|||||||
self.failUnlessRaises(MissingConfigEntry, client.Client, basedir1)
|
self.failUnlessRaises(MissingConfigEntry, client.Client, basedir1)
|
||||||
|
|
||||||
fileutil.write(os.path.join(basedir1, "tahoe.cfg"), config)
|
fileutil.write(os.path.join(basedir1, "tahoe.cfg"), config)
|
||||||
fileutil.write(os.path.join(basedir1, "private", "drop_upload_dircap"), "URI:DIR2:blah")
|
fileutil.write(os.path.join(basedir1, "private", "magic_folder_dircap"), "URI:DIR2:blah")
|
||||||
fileutil.write(os.path.join(basedir1, "private", "magic_folder_parent_dircap"), "URI:DIR2:meow")
|
fileutil.write(os.path.join(basedir1, "private", "collective_dircap"), "URI:DIR2:meow")
|
||||||
self.failUnlessRaises(MissingConfigEntry, client.Client, basedir1)
|
self.failUnlessRaises(MissingConfigEntry, client.Client, basedir1)
|
||||||
|
|
||||||
fileutil.write(os.path.join(basedir1, "tahoe.cfg"),
|
fileutil.write(os.path.join(basedir1, "tahoe.cfg"),
|
||||||
config + "upload.dircap = " + upload_dircap + "\n")
|
config.replace("[magic_folder]\n", "[drop_upload]\n"))
|
||||||
self.failUnlessRaises(OldConfigOptionError, client.Client, basedir1)
|
self.failUnlessRaises(OldConfigOptionError, client.Client, basedir1)
|
||||||
|
|
||||||
fileutil.write(os.path.join(basedir1, "tahoe.cfg"),
|
fileutil.write(os.path.join(basedir1, "tahoe.cfg"),
|
||||||
config + "local.directory = " + local_dir_utf8 + "\n")
|
config + "local.directory = " + local_dir_utf8 + "\n")
|
||||||
c1 = client.Client(basedir1)
|
c1 = client.Client(basedir1)
|
||||||
uploader = c1.getServiceNamed('drop-upload')
|
magicfolder = c1.getServiceNamed('magic-folder')
|
||||||
self.failUnless(isinstance(uploader, MockDropUploader), uploader)
|
self.failUnless(isinstance(magicfolder, MockMagicFolder), magicfolder)
|
||||||
self.failUnlessReallyEqual(uploader.client, c1)
|
self.failUnlessReallyEqual(magicfolder.client, c1)
|
||||||
self.failUnlessReallyEqual(uploader.upload_dircap, upload_dircap)
|
self.failUnlessReallyEqual(magicfolder.upload_dircap, upload_dircap)
|
||||||
self.failUnlessReallyEqual(os.path.basename(uploader.local_dir), local_dir_u)
|
self.failUnlessReallyEqual(os.path.basename(magicfolder.local_dir), local_dir_u)
|
||||||
self.failUnless(uploader.inotify is None, uploader.inotify)
|
self.failUnless(magicfolder.inotify is None, magicfolder.inotify)
|
||||||
self.failUnless(uploader.running)
|
self.failUnless(magicfolder.running)
|
||||||
|
|
||||||
class Boom(Exception):
|
class Boom(Exception):
|
||||||
pass
|
pass
|
||||||
def BoomDropUploader(client, upload_dircap, local_dir_utf8, inotify=None):
|
def BoomMagicFolder(self, client, upload_dircap, collective_dircap, local_dir, dbfile,
|
||||||
|
inotify=None, pending_delay=1.0):
|
||||||
raise Boom()
|
raise Boom()
|
||||||
|
|
||||||
logged_messages = []
|
logged_messages = []
|
||||||
def mock_log(*args, **kwargs):
|
def mock_log(*args, **kwargs):
|
||||||
logged_messages.append("%r %r" % (args, kwargs))
|
logged_messages.append("%r %r" % (args, kwargs))
|
||||||
self.patch(allmydata.util.log, 'msg', mock_log)
|
self.patch(allmydata.util.log, 'msg', mock_log)
|
||||||
self.patch(allmydata.frontends.drop_upload, 'DropUploader', BoomDropUploader)
|
self.patch(allmydata.frontends.magic_folder, 'MagicFolder', BoomMagicFolder)
|
||||||
|
|
||||||
basedir2 = "test_client.Basic.test_create_drop_uploader2"
|
basedir2 = "test_client.Basic.test_create_magic_folder2"
|
||||||
os.mkdir(basedir2)
|
os.mkdir(basedir2)
|
||||||
os.mkdir(os.path.join(basedir2, "private"))
|
os.mkdir(os.path.join(basedir2, "private"))
|
||||||
fileutil.write(os.path.join(basedir2, "tahoe.cfg"),
|
fileutil.write(os.path.join(basedir2, "tahoe.cfg"),
|
||||||
BASECONFIG +
|
BASECONFIG +
|
||||||
"[drop_upload]\n" +
|
"[magic_folder]\n" +
|
||||||
"enabled = true\n" +
|
"enabled = true\n" +
|
||||||
"local.directory = " + local_dir_utf8 + "\n")
|
"local.directory = " + local_dir_utf8 + "\n")
|
||||||
fileutil.write(os.path.join(basedir2, "private", "drop_upload_dircap"), "URI:DIR2:blah")
|
fileutil.write(os.path.join(basedir2, "private", "magic_folder_dircap"), "URI:DIR2:blah")
|
||||||
fileutil.write(os.path.join(basedir2, "private", "magic_folder_parent_dircap"), "URI:DIR2:meow")
|
fileutil.write(os.path.join(basedir2, "private", "collective_dircap"), "URI:DIR2:meow")
|
||||||
c2 = client.Client(basedir2)
|
c2 = client.Client(basedir2)
|
||||||
self.failUnlessRaises(KeyError, c2.getServiceNamed, 'drop-upload')
|
self.failUnlessRaises(KeyError, c2.getServiceNamed, 'magic-folder')
|
||||||
self.failUnless([True for arg in logged_messages if "Boom" in arg],
|
self.failUnless([True for arg in logged_messages if "Boom" in arg],
|
||||||
logged_messages)
|
logged_messages)
|
||||||
|
|
||||||
|
@ -13,13 +13,13 @@ from allmydata.test.no_network import GridTestMixin
|
|||||||
from allmydata.test.common_util import ReallyEqualMixin, NonASCIIPathMixin
|
from allmydata.test.common_util import ReallyEqualMixin, NonASCIIPathMixin
|
||||||
from allmydata.test.common import ShouldFailMixin
|
from allmydata.test.common import ShouldFailMixin
|
||||||
|
|
||||||
from allmydata.frontends import drop_upload
|
from allmydata.frontends import magic_folder
|
||||||
from allmydata.frontends.drop_upload import DropUploader
|
from allmydata.frontends.magic_folder import MagicFolder
|
||||||
from allmydata import backupdb
|
from allmydata import backupdb
|
||||||
from allmydata.util.fileutil import abspath_expanduser_unicode
|
from allmydata.util.fileutil import abspath_expanduser_unicode
|
||||||
|
|
||||||
|
|
||||||
class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonASCIIPathMixin):
|
class MagicFolderTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonASCIIPathMixin):
|
||||||
"""
|
"""
|
||||||
These tests will be run both with a mock notifier, and (on platforms that support it)
|
These tests will be run both with a mock notifier, and (on platforms that support it)
|
||||||
with the real INotify.
|
with the real INotify.
|
||||||
@ -29,7 +29,7 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
|
|||||||
GridTestMixin.setUp(self)
|
GridTestMixin.setUp(self)
|
||||||
temp = self.mktemp()
|
temp = self.mktemp()
|
||||||
self.basedir = abspath_expanduser_unicode(temp.decode(get_filesystem_encoding()))
|
self.basedir = abspath_expanduser_unicode(temp.decode(get_filesystem_encoding()))
|
||||||
self.uploader = None
|
self.magicfolder = None
|
||||||
self.dir_node = None
|
self.dir_node = None
|
||||||
|
|
||||||
def _get_count(self, name):
|
def _get_count(self, name):
|
||||||
@ -50,20 +50,20 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
|
|||||||
self.failUnless(IDirectoryNode.providedBy(n))
|
self.failUnless(IDirectoryNode.providedBy(n))
|
||||||
self.upload_dirnode = n
|
self.upload_dirnode = n
|
||||||
self.upload_dircap = n.get_uri()
|
self.upload_dircap = n.get_uri()
|
||||||
self.parent_dircap = "abc123"
|
self.collective_dircap = "abc123"
|
||||||
|
|
||||||
def _create_uploader(self, ign):
|
def _create_magicfolder(self, ign):
|
||||||
dbfile = abspath_expanduser_unicode(u"magicfolderdb.sqlite", base=self.basedir)
|
dbfile = abspath_expanduser_unicode(u"magicfolderdb.sqlite", base=self.basedir)
|
||||||
self.uploader = DropUploader(self.client, self.upload_dircap, self.parent_dircap, self.local_dir,
|
self.magicfolder = MagicFolder(self.client, self.upload_dircap, self.collective_dircap, self.local_dir,
|
||||||
dbfile, inotify=self.inotify, pending_delay=0.2)
|
dbfile, inotify=self.inotify, pending_delay=0.2)
|
||||||
self.uploader.setServiceParent(self.client)
|
self.magicfolder.setServiceParent(self.client)
|
||||||
self.uploader.upload_ready()
|
self.magicfolder.upload_ready()
|
||||||
|
|
||||||
# Prevent unclean reactor errors.
|
# Prevent unclean reactor errors.
|
||||||
def _cleanup(self, res):
|
def _cleanup(self, res):
|
||||||
d = defer.succeed(None)
|
d = defer.succeed(None)
|
||||||
if self.uploader is not None:
|
if self.magicfolder is not None:
|
||||||
d.addCallback(lambda ign: self.uploader.finish(for_tests=True))
|
d.addCallback(lambda ign: self.magicfolder.finish(for_tests=True))
|
||||||
d.addCallback(lambda ign: res)
|
d.addCallback(lambda ign: res)
|
||||||
return d
|
return d
|
||||||
|
|
||||||
@ -100,7 +100,7 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
|
|||||||
r = db.check_file(path)
|
r = db.check_file(path)
|
||||||
self.failUnless(r.was_uploaded())
|
self.failUnless(r.was_uploaded())
|
||||||
|
|
||||||
def test_uploader_start_service(self):
|
def test_magicfolder_start_service(self):
|
||||||
self.set_up_grid()
|
self.set_up_grid()
|
||||||
|
|
||||||
self.local_dir = abspath_expanduser_unicode(self.unicode_or_fallback(u"l\u00F8cal_dir", u"local_dir"),
|
self.local_dir = abspath_expanduser_unicode(self.unicode_or_fallback(u"l\u00F8cal_dir", u"local_dir"),
|
||||||
@ -112,10 +112,10 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
|
|||||||
|
|
||||||
d = self.client.create_dirnode()
|
d = self.client.create_dirnode()
|
||||||
d.addCallback(self._made_upload_dir)
|
d.addCallback(self._made_upload_dir)
|
||||||
d.addCallback(self._create_uploader)
|
d.addCallback(self._create_magicfolder)
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.dirs_monitored'), 1))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.dirs_monitored'), 1))
|
||||||
d.addBoth(self._cleanup)
|
d.addBoth(self._cleanup)
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.dirs_monitored'), 0))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.dirs_monitored'), 0))
|
||||||
return d
|
return d
|
||||||
|
|
||||||
def test_move_tree(self):
|
def test_move_tree(self):
|
||||||
@ -139,46 +139,46 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
|
|||||||
d = self.client.create_dirnode()
|
d = self.client.create_dirnode()
|
||||||
d.addCallback(self._made_upload_dir)
|
d.addCallback(self._made_upload_dir)
|
||||||
|
|
||||||
d.addCallback(self._create_uploader)
|
d.addCallback(self._create_magicfolder)
|
||||||
|
|
||||||
def _check_move_empty_tree(res):
|
def _check_move_empty_tree(res):
|
||||||
self.mkdir_nonascii(empty_tree_dir)
|
self.mkdir_nonascii(empty_tree_dir)
|
||||||
d2 = defer.Deferred()
|
d2 = defer.Deferred()
|
||||||
self.uploader.set_processed_callback(d2.callback, ignore_count=0)
|
self.magicfolder.set_processed_callback(d2.callback, ignore_count=0)
|
||||||
os.rename(empty_tree_dir, new_empty_tree_dir)
|
os.rename(empty_tree_dir, new_empty_tree_dir)
|
||||||
self.notify(to_filepath(new_empty_tree_dir), self.inotify.IN_MOVED_TO)
|
self.notify(to_filepath(new_empty_tree_dir), self.inotify.IN_MOVED_TO)
|
||||||
return d2
|
return d2
|
||||||
d.addCallback(_check_move_empty_tree)
|
d.addCallback(_check_move_empty_tree)
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_succeeded'), 1))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_succeeded'), 1))
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.files_uploaded'), 0))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.files_uploaded'), 0))
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_queued'), 0))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_queued'), 0))
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.directories_created'), 1))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.directories_created'), 1))
|
||||||
|
|
||||||
def _check_move_small_tree(res):
|
def _check_move_small_tree(res):
|
||||||
self.mkdir_nonascii(small_tree_dir)
|
self.mkdir_nonascii(small_tree_dir)
|
||||||
fileutil.write(abspath_expanduser_unicode(u"what", base=small_tree_dir), "say when")
|
fileutil.write(abspath_expanduser_unicode(u"what", base=small_tree_dir), "say when")
|
||||||
d2 = defer.Deferred()
|
d2 = defer.Deferred()
|
||||||
self.uploader.set_processed_callback(d2.callback, ignore_count=1)
|
self.magicfolder.set_processed_callback(d2.callback, ignore_count=1)
|
||||||
os.rename(small_tree_dir, new_small_tree_dir)
|
os.rename(small_tree_dir, new_small_tree_dir)
|
||||||
self.notify(to_filepath(new_small_tree_dir), self.inotify.IN_MOVED_TO)
|
self.notify(to_filepath(new_small_tree_dir), self.inotify.IN_MOVED_TO)
|
||||||
return d2
|
return d2
|
||||||
d.addCallback(_check_move_small_tree)
|
d.addCallback(_check_move_small_tree)
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_succeeded'), 3))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_succeeded'), 3))
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.files_uploaded'), 1))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.files_uploaded'), 1))
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_queued'), 0))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_queued'), 0))
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.directories_created'), 2))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.directories_created'), 2))
|
||||||
|
|
||||||
def _check_moved_tree_is_watched(res):
|
def _check_moved_tree_is_watched(res):
|
||||||
d2 = defer.Deferred()
|
d2 = defer.Deferred()
|
||||||
self.uploader.set_processed_callback(d2.callback, ignore_count=0)
|
self.magicfolder.set_processed_callback(d2.callback, ignore_count=0)
|
||||||
fileutil.write(abspath_expanduser_unicode(u"another", base=new_small_tree_dir), "file")
|
fileutil.write(abspath_expanduser_unicode(u"another", base=new_small_tree_dir), "file")
|
||||||
self.notify(to_filepath(abspath_expanduser_unicode(u"another", base=new_small_tree_dir)), self.inotify.IN_CLOSE_WRITE)
|
self.notify(to_filepath(abspath_expanduser_unicode(u"another", base=new_small_tree_dir)), self.inotify.IN_CLOSE_WRITE)
|
||||||
return d2
|
return d2
|
||||||
d.addCallback(_check_moved_tree_is_watched)
|
d.addCallback(_check_moved_tree_is_watched)
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_succeeded'), 4))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_succeeded'), 4))
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.files_uploaded'), 2))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.files_uploaded'), 2))
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_queued'), 0))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_queued'), 0))
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.directories_created'), 2))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.directories_created'), 2))
|
||||||
|
|
||||||
# Files that are moved out of the upload directory should no longer be watched.
|
# Files that are moved out of the upload directory should no longer be watched.
|
||||||
def _move_dir_away(ign):
|
def _move_dir_away(ign):
|
||||||
@ -192,10 +192,10 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
|
|||||||
return
|
return
|
||||||
d.addCallback(create_file)
|
d.addCallback(create_file)
|
||||||
d.addCallback(lambda ign: time.sleep(1))
|
d.addCallback(lambda ign: time.sleep(1))
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_succeeded'), 4))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_succeeded'), 4))
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.files_uploaded'), 2))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.files_uploaded'), 2))
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_queued'), 0))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_queued'), 0))
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.directories_created'), 2))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.directories_created'), 2))
|
||||||
|
|
||||||
d.addBoth(self._cleanup)
|
d.addBoth(self._cleanup)
|
||||||
return d
|
return d
|
||||||
@ -203,9 +203,9 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
|
|||||||
def test_persistence(self):
|
def test_persistence(self):
|
||||||
"""
|
"""
|
||||||
Perform an upload of a given file and then stop the client.
|
Perform an upload of a given file and then stop the client.
|
||||||
Start a new client and uploader... and verify that the file is NOT uploaded
|
Start a new client and magic-folder service... and verify that the file is NOT uploaded
|
||||||
a second time. This test is meant to test the database persistence along with
|
a second time. This test is meant to test the database persistence along with
|
||||||
the startup and shutdown code paths of the uploader.
|
the startup and shutdown code paths of the magic-folder service.
|
||||||
"""
|
"""
|
||||||
self.set_up_grid()
|
self.set_up_grid()
|
||||||
self.local_dir = abspath_expanduser_unicode(u"test_persistence", base=self.basedir)
|
self.local_dir = abspath_expanduser_unicode(u"test_persistence", base=self.basedir)
|
||||||
@ -215,18 +215,18 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
|
|||||||
self.stats_provider = self.client.stats_provider
|
self.stats_provider = self.client.stats_provider
|
||||||
d = self.client.create_dirnode()
|
d = self.client.create_dirnode()
|
||||||
d.addCallback(self._made_upload_dir)
|
d.addCallback(self._made_upload_dir)
|
||||||
d.addCallback(self._create_uploader)
|
d.addCallback(self._create_magicfolder)
|
||||||
|
|
||||||
def create_file(val):
|
def create_file(val):
|
||||||
d2 = defer.Deferred()
|
d2 = defer.Deferred()
|
||||||
self.uploader.set_processed_callback(d2.callback)
|
self.magicfolder.set_processed_callback(d2.callback)
|
||||||
test_file = abspath_expanduser_unicode(u"what", base=self.local_dir)
|
test_file = abspath_expanduser_unicode(u"what", base=self.local_dir)
|
||||||
fileutil.write(test_file, "meow")
|
fileutil.write(test_file, "meow")
|
||||||
self.notify(to_filepath(test_file), self.inotify.IN_CLOSE_WRITE)
|
self.notify(to_filepath(test_file), self.inotify.IN_CLOSE_WRITE)
|
||||||
return d2
|
return d2
|
||||||
d.addCallback(create_file)
|
d.addCallback(create_file)
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_succeeded'), 1))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_succeeded'), 1))
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_queued'), 0))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_queued'), 0))
|
||||||
d.addCallback(self._cleanup)
|
d.addCallback(self._cleanup)
|
||||||
|
|
||||||
def _restart(ign):
|
def _restart(ign):
|
||||||
@ -234,14 +234,14 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
|
|||||||
self.client = self.g.clients[0]
|
self.client = self.g.clients[0]
|
||||||
self.stats_provider = self.client.stats_provider
|
self.stats_provider = self.client.stats_provider
|
||||||
d.addCallback(_restart)
|
d.addCallback(_restart)
|
||||||
d.addCallback(self._create_uploader)
|
d.addCallback(self._create_magicfolder)
|
||||||
d.addCallback(lambda ign: time.sleep(3))
|
d.addCallback(lambda ign: time.sleep(3))
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_succeeded'), 0))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_succeeded'), 0))
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_queued'), 0))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_queued'), 0))
|
||||||
d.addBoth(self._cleanup)
|
d.addBoth(self._cleanup)
|
||||||
return d
|
return d
|
||||||
|
|
||||||
def test_drop_upload(self):
|
def test_magic_folder(self):
|
||||||
self.set_up_grid()
|
self.set_up_grid()
|
||||||
self.local_dir = os.path.join(self.basedir, self.unicode_or_fallback(u"loc\u0101l_dir", u"local_dir"))
|
self.local_dir = os.path.join(self.basedir, self.unicode_or_fallback(u"loc\u0101l_dir", u"local_dir"))
|
||||||
self.mkdir_nonascii(self.local_dir)
|
self.mkdir_nonascii(self.local_dir)
|
||||||
@ -252,7 +252,7 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
|
|||||||
d = self.client.create_dirnode()
|
d = self.client.create_dirnode()
|
||||||
|
|
||||||
d.addCallback(self._made_upload_dir)
|
d.addCallback(self._made_upload_dir)
|
||||||
d.addCallback(self._create_uploader)
|
d.addCallback(self._create_magicfolder)
|
||||||
|
|
||||||
# Write something short enough for a LIT file.
|
# Write something short enough for a LIT file.
|
||||||
d.addCallback(lambda ign: self._check_file(u"short", "test"))
|
d.addCallback(lambda ign: self._check_file(u"short", "test"))
|
||||||
@ -271,20 +271,20 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
|
|||||||
d.addCallback(lambda ign: self._check_file(name_u, "test"*100))
|
d.addCallback(lambda ign: self._check_file(name_u, "test"*100))
|
||||||
|
|
||||||
# TODO: test that causes an upload failure.
|
# TODO: test that causes an upload failure.
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.files_failed'), 0))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.files_failed'), 0))
|
||||||
|
|
||||||
d.addBoth(self._cleanup)
|
d.addBoth(self._cleanup)
|
||||||
return d
|
return d
|
||||||
|
|
||||||
def _check_file(self, name_u, data, temporary=False):
|
def _check_file(self, name_u, data, temporary=False):
|
||||||
previously_uploaded = self._get_count('drop_upload.objects_succeeded')
|
previously_uploaded = self._get_count('magic_folder.objects_succeeded')
|
||||||
previously_disappeared = self._get_count('drop_upload.objects_disappeared')
|
previously_disappeared = self._get_count('magic_folder.objects_disappeared')
|
||||||
|
|
||||||
d = defer.Deferred()
|
d = defer.Deferred()
|
||||||
|
|
||||||
# Note: this relies on the fact that we only get one IN_CLOSE_WRITE notification per file
|
# Note: this relies on the fact that we only get one IN_CLOSE_WRITE notification per file
|
||||||
# (otherwise we would get a defer.AlreadyCalledError). Should we be relying on that?
|
# (otherwise we would get a defer.AlreadyCalledError). Should we be relying on that?
|
||||||
self.uploader.set_processed_callback(d.callback)
|
self.magicfolder.set_processed_callback(d.callback)
|
||||||
|
|
||||||
path_u = abspath_expanduser_unicode(name_u, base=self.local_dir)
|
path_u = abspath_expanduser_unicode(name_u, base=self.local_dir)
|
||||||
path = to_filepath(path_u)
|
path = to_filepath(path_u)
|
||||||
@ -307,28 +307,28 @@ class DropUploadTestMixin(GridTestMixin, ShouldFailMixin, ReallyEqualMixin, NonA
|
|||||||
if temporary:
|
if temporary:
|
||||||
d.addCallback(lambda ign: self.shouldFail(NoSuchChildError, 'temp file not uploaded', None,
|
d.addCallback(lambda ign: self.shouldFail(NoSuchChildError, 'temp file not uploaded', None,
|
||||||
self.upload_dirnode.get, name_u))
|
self.upload_dirnode.get, name_u))
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_disappeared'),
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_disappeared'),
|
||||||
previously_disappeared + 1))
|
previously_disappeared + 1))
|
||||||
else:
|
else:
|
||||||
d.addCallback(lambda ign: self.upload_dirnode.get(name_u))
|
d.addCallback(lambda ign: self.upload_dirnode.get(name_u))
|
||||||
d.addCallback(download_to_data)
|
d.addCallback(download_to_data)
|
||||||
d.addCallback(lambda actual_data: self.failUnlessReallyEqual(actual_data, data))
|
d.addCallback(lambda actual_data: self.failUnlessReallyEqual(actual_data, data))
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_succeeded'),
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_succeeded'),
|
||||||
previously_uploaded + 1))
|
previously_uploaded + 1))
|
||||||
|
|
||||||
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('drop_upload.objects_queued'), 0))
|
d.addCallback(lambda ign: self.failUnlessReallyEqual(self._get_count('magic_folder.objects_queued'), 0))
|
||||||
return d
|
return d
|
||||||
|
|
||||||
|
|
||||||
class MockTest(DropUploadTestMixin, unittest.TestCase):
|
class MockTest(MagicFolderTestMixin, unittest.TestCase):
|
||||||
"""This can run on any platform, and even if twisted.internet.inotify can't be imported."""
|
"""This can run on any platform, and even if twisted.internet.inotify can't be imported."""
|
||||||
|
|
||||||
def setUp(self):
|
def setUp(self):
|
||||||
DropUploadTestMixin.setUp(self)
|
MagicFolderTestMixin.setUp(self)
|
||||||
self.inotify = fake_inotify
|
self.inotify = fake_inotify
|
||||||
|
|
||||||
def notify(self, path, mask):
|
def notify(self, path, mask):
|
||||||
self.uploader._notifier.event(path, mask)
|
self.magicfolder._notifier.event(path, mask)
|
||||||
|
|
||||||
def test_errors(self):
|
def test_errors(self):
|
||||||
self.set_up_grid()
|
self.set_up_grid()
|
||||||
@ -348,37 +348,37 @@ class MockTest(DropUploadTestMixin, unittest.TestCase):
|
|||||||
readonly_dircap = n.get_readonly_uri()
|
readonly_dircap = n.get_readonly_uri()
|
||||||
|
|
||||||
self.shouldFail(AssertionError, 'nonexistent local.directory', 'there is no directory',
|
self.shouldFail(AssertionError, 'nonexistent local.directory', 'there is no directory',
|
||||||
DropUploader, client, upload_dircap, '', doesnotexist, magicfolderdb, inotify=fake_inotify)
|
MagicFolder, client, upload_dircap, '', doesnotexist, magicfolderdb, inotify=fake_inotify)
|
||||||
self.shouldFail(AssertionError, 'non-directory local.directory', 'is not a directory',
|
self.shouldFail(AssertionError, 'non-directory local.directory', 'is not a directory',
|
||||||
DropUploader, client, upload_dircap, '', not_a_dir, magicfolderdb, inotify=fake_inotify)
|
MagicFolder, client, upload_dircap, '', not_a_dir, magicfolderdb, inotify=fake_inotify)
|
||||||
self.shouldFail(AssertionError, 'bad upload.dircap', 'does not refer to a directory',
|
self.shouldFail(AssertionError, 'bad upload.dircap', 'does not refer to a directory',
|
||||||
DropUploader, client, 'bad', '', errors_dir, magicfolderdb, inotify=fake_inotify)
|
MagicFolder, client, 'bad', '', errors_dir, magicfolderdb, inotify=fake_inotify)
|
||||||
self.shouldFail(AssertionError, 'non-directory upload.dircap', 'does not refer to a directory',
|
self.shouldFail(AssertionError, 'non-directory upload.dircap', 'does not refer to a directory',
|
||||||
DropUploader, client, 'URI:LIT:foo', '', errors_dir, magicfolderdb, inotify=fake_inotify)
|
MagicFolder, client, 'URI:LIT:foo', '', errors_dir, magicfolderdb, inotify=fake_inotify)
|
||||||
self.shouldFail(AssertionError, 'readonly upload.dircap', 'is not a writecap to a directory',
|
self.shouldFail(AssertionError, 'readonly upload.dircap', 'is not a writecap to a directory',
|
||||||
DropUploader, client, readonly_dircap, '', errors_dir, magicfolderdb, inotify=fake_inotify)
|
MagicFolder, client, readonly_dircap, '', errors_dir, magicfolderdb, inotify=fake_inotify)
|
||||||
|
|
||||||
def _not_implemented():
|
def _not_implemented():
|
||||||
raise NotImplementedError("blah")
|
raise NotImplementedError("blah")
|
||||||
self.patch(drop_upload, 'get_inotify_module', _not_implemented)
|
self.patch(magic_folder, 'get_inotify_module', _not_implemented)
|
||||||
self.shouldFail(NotImplementedError, 'unsupported', 'blah',
|
self.shouldFail(NotImplementedError, 'unsupported', 'blah',
|
||||||
DropUploader, client, upload_dircap, '', errors_dir, magicfolderdb)
|
MagicFolder, client, upload_dircap, '', errors_dir, magicfolderdb)
|
||||||
d.addCallback(_check_errors)
|
d.addCallback(_check_errors)
|
||||||
return d
|
return d
|
||||||
|
|
||||||
|
|
||||||
class RealTest(DropUploadTestMixin, unittest.TestCase):
|
class RealTest(MagicFolderTestMixin, unittest.TestCase):
|
||||||
"""This is skipped unless both Twisted and the platform support inotify."""
|
"""This is skipped unless both Twisted and the platform support inotify."""
|
||||||
|
|
||||||
def setUp(self):
|
def setUp(self):
|
||||||
DropUploadTestMixin.setUp(self)
|
MagicFolderTestMixin.setUp(self)
|
||||||
self.inotify = drop_upload.get_inotify_module()
|
self.inotify = magic_folder.get_inotify_module()
|
||||||
|
|
||||||
def notify(self, path, mask):
|
def notify(self, path, mask):
|
||||||
# Writing to the filesystem causes the notification.
|
# Writing to the filesystem causes the notification.
|
||||||
pass
|
pass
|
||||||
|
|
||||||
try:
|
try:
|
||||||
drop_upload.get_inotify_module()
|
magic_folder.get_inotify_module()
|
||||||
except NotImplementedError:
|
except NotImplementedError:
|
||||||
RealTest.skip = "Drop-upload support can only be tested for-real on an OS that supports inotify or equivalent."
|
RealTest.skip = "Magic Folder support can only be tested for-real on an OS that supports inotify or equivalent."
|
Loading…
Reference in New Issue
Block a user