ExternalVendorCode/tahoe-lafs
1
0
Fork 0
mirror of https://github.com/tahoe-lafs/tahoe-lafs.git synced 2025-01-04 12:14:11 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
7,340 Commits 185 Branches 90 Tags 89 MiB
cec3746682
Commit Graph

232 Commits

Author SHA1 Message Date
meejah
a50d0e5c8b WIP and debugging things 
Conflicts:
	src/allmydata/test/test_magic_folder.py
 2016-07-21 12:35:59 -07:00
Daira Hopwood
e67d23b8fa Fix negative counts for object_queued. fixes ticket:2731 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
meejah
378ce27cbd Make retry_download test work, and some bonus comments 2016-07-21 12:35:59 -07:00
Daira Hopwood
9e5b01b3ae Swallow the error from Downloader._when_queue_is_empty. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
meejah
e6104cd1a2 Replace _lazy_tail with iterative loop, cleanup tests (#2412) 
* uses @inlineCallbacks to turn the _lazy_tail recursion into
   a "real" looking loop;
 * remove the need for "immediate" vs delayed iteration of said loop;
 * make it easier for the unit-tests to control the behavior of the
   uploader/downloader;
 * consolidates (some) setup/teardown code into the setUp and tearDown
   hooks provided by unittest so unit-tests aren't doing that themselves
 * re-factors some of the unit-tests to use an @inlineCallbacks style
   so they're easier to follow and debug

This doesn't tackle the "how to know when our inotify events have arrived"
problem the unit-tests still have, nor does it eliminate the myriad bits
of state that get added to tests via all the MixIns.
 2016-07-21 12:35:59 -07:00
David Stainton
2963e9e50c Use os.path.dirname instead of split and rename var to initial_path_u 2016-07-21 12:35:59 -07:00
David Stainton
ee44732d03 Break out our chmod while loop into fielutils.py 2016-07-21 12:35:59 -07:00
David Stainton
51ccec7a33 Use chmod instead of changing umask 
Conflicts:
	src/allmydata/frontends/magic_folder.py
 2016-07-21 12:35:59 -07:00
David Stainton
3d909682a4 Add comment about FUDGE_SECONDS and refer to our design doc 2016-07-21 12:35:59 -07:00
meejah
bf3bf70814 add docstrings 2016-07-21 12:35:59 -07:00
meejah
8ba2912ff4 make history longer 2016-07-21 12:35:59 -07:00
meejah
7e84b0cf57 flesh out IQueuedItem 2016-07-21 12:35:59 -07:00
meejah
86abe56d91 Flesh out "tahoe magic-folder status" command 
Adds:

 - a JSON endpoint
 - CLI to display information
 - QueuedItem + IQueuedItem for uploader/downloader
 - IProgress interface + PercentProgress implementation
 - progress= args to many upload/download APIs
 2016-07-21 12:35:59 -07:00
David Stainton
71e3164e92 Further work in progress refinements to unit tests 
wip from pairing with Daira and Meejah
 2016-07-21 12:35:59 -07:00
David Stainton
38dd49ee5a Add logging for Downloader.stop. 2016-07-21 12:35:59 -07:00
David Stainton
b4153659d3 Minor cleanup and logging fix. 2016-07-21 12:35:59 -07:00
David Stainton
5380ff8f5a Clean up queue loging and add a clock advance in unit test 
not yet working
 2016-07-21 12:35:59 -07:00
David Stainton
e1822c0518 WIP 2016-07-21 12:35:59 -07:00
David Stainton
96b846f762 Add ignore pending argument to _periodic_full_scan 2016-07-21 12:35:59 -07:00
David Stainton
a5163c32cc Only perform full scan if pending set is empty 2016-07-21 12:35:59 -07:00
David Stainton
e9e74c43e1 Naive periodic full scan 2016-07-21 12:35:59 -07:00
David Stainton
9b4b43cf19 Fix uploader's _process to extend queue after scan and count stats properly 2016-07-21 12:35:59 -07:00
David Stainton
d2971fbabe Daira's excellent fixes to the uploader from our pairing session 2016-07-21 12:35:59 -07:00
David Stainton
70f66677be Teach uploader's _scan to use the deque 2016-07-21 12:35:59 -07:00
Daira Hopwood
2043bed3f1 Delay only the download scan not the turning of our event queue. 
Author: David Stainton <david@leastauthority.com>
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
David Stainton
0273034b0c Fix umask again 2016-07-21 12:35:59 -07:00
David Stainton
ae59b17e86 Fix umask usage bit fiddling and use try finally blocks 2016-07-21 12:35:59 -07:00
David Stainton
f2a7978b29 Add download.umask config option with default of 077 2016-07-21 12:35:59 -07:00
David Stainton
623ed3516e Add umask to Downloader 2016-07-21 12:35:59 -07:00
David Stainton
79f3c5d81a Remove magic-folder exclude stat 2016-07-21 12:35:59 -07:00
Daira Hopwood
6b4d3b8c6d Refactor is_new_file. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
02dacbc067 Minor simplification. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
David Stainton
e990483750 Simply conflict detection by removing nested if statements 2016-07-21 12:35:59 -07:00
David Stainton
6ed4480eb9 Scan our own dmd upon start only 2016-07-21 12:35:59 -07:00
meejah
ed4e4d053e Add a unit-test and correct the code for "already deleted" 
If a Downloader decides that it needs to delete a file, but that
file is already gone locally, the exeption is caugt and a log message
produced.
 2016-07-21 12:35:59 -07:00
Daira Hopwood
9d59ef493f WIP: Refactoring to get db fields in one query. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
42f535edf9 Fix pending upload conflict detection. 
Author: David Stainton <david@leastauthority.com>
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
e8b6b31373 Detect remote conflict by checking for pending upload. 
Author: David Stainton <david@leastauthority.com>
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
David Stainton
c7f38a496d Add debugging print statement for timestamp comparison 2016-07-21 12:35:59 -07:00
David Stainton
edc671f577 Add last uploaded timestamp comparison for remote conflict detection 2016-07-21 12:35:59 -07:00
Daira Hopwood
c7f298bce5 Fix some miscapture bugs. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
868c38d228 Workaround. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
b21a3dc070 Simplify _notify and improve logging. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
17384ac512 Simplify _scan_remote_* and remove Downloader._download_scan_batch attribute. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
d686575393 Cosmetics. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
4f41be0e70 Downloader doesn't need the pending set. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
017d63b47b Delete redundant is_ready attribute from MagicFolder. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
5f98d73216 Add precondition to Uploader._process. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
0c12162bc8 Debugging WIP. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
84af199d51 WIP: exclude own dirnode from scan. This is not quite right; we shouldn't exclude it on startup. refs #2553 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
ea14b1bebe Don't add subdirectory watches if the platform's notifier doesn't require them. refs #2559 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
6bb0963197 Improve reporting of assertion failures. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
2a79252a72 Fix some path Unixisms. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
1abc2c39c8 Don't download the deletion marker file unnecessarily. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
c6191257ea Distinguish deletion of directories. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
d6bb49b1a6 Rename deleted files to .backup rather than unlinking them. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
David Stainton
99d1339d38 Add basic bob upload test and fix conflict detect 2016-07-21 12:35:59 -07:00
David Stainton
9ed61f1eb4 Count conflicted objects 2016-07-21 12:35:59 -07:00
Daira Hopwood
7a3b35e29c Basic remote conflict detection based on ancestor uri 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
f75dda7635 Avoid .format, since it is inconsistent between Python 2.6 and 2.7 (and the rest of Tahoe-LAFS doesn't use it). 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
e4082a45d3 Add counter for uploader.objects_not_uploaded. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
meejah
c883977eb6 add the 'spurious' notifies 2016-07-21 12:35:59 -07:00
meejah
d61b8ed39e implement 'delete' functionality, with tests 2016-07-21 12:35:59 -07:00
Daira Hopwood
cec315da49 Add test that we don't write files outside the magic folder directory. refs ticket:2506 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
5d5d804719 More debug logging. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
8184ed4414 Watch for IN_CREATE events but filter them out for non-directories. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
cbec939d86 Implement creating local directories in downloader. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
60ab5a057a Decode names in the scanned remote. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
a3fa1617c2 Refactoring to allow logging from _write_downloaded_file and _rename_conflicted_file. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
6cb41cadf5 Simplify and fix non-existent-file handling. 
Also make the existent and non-existent cases as similar as possible,
with a view to merging them.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
cb905d8d97 Logging/debugging improvements. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
ac5730ea1c Correct a call to did_upload_version in the downloader. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
28587b5b2a Restore a call to increment files_uploaded that was mistakenly removed. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
David Stainton
2fcaa8c2b2 Teach uploader+downloader to use to db schema 
here we attempt to fix all the unit tests as well...
however two tests still fail
 2016-07-21 12:35:59 -07:00
Daira Hopwood
5754c01cb5 Add magicfolderdb.py. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
David Stainton
e2cee9ba2a WIP 2016-07-21 12:35:59 -07:00
David Stainton
f8bb229ab7 For all downloaded files ensure parent dir exists 2016-07-21 12:35:59 -07:00
meejah
94a2ef0095 Make downloader delay a class-variable 
This gives the integration-style CLI-based tests a chance
to set the delay to 0 before the first 3-second delayed
call is queued to _lazy_tail in the Downloader
 2016-07-21 12:35:59 -07:00
Daira Hopwood
e03a1e3361 WIP. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
262e238a7d Magic Folder file moves. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
3170cab0fc Prepare to move drop_upload.py to magic_folder.py. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
76a178969d Move backupdb.py to src/allmydata. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
5f143f7635 Enable Windows inotify support. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
61b3a9cef2 Add magic folder db. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
7b4067850f Unicode path fixes for drop-upload. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
a56a3adaae Teach StorageFarmBroker to fire a deferred when a connection threshold is reached. refs #1449 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-04-26 12:26:39 -06:00
Daira Hopwood
5d046c597e drop-upload.py: fix error messages. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2015-05-13 14:21:17 +01:00
Brian Warner
eda00a5c58 typo 2015-03-31 11:07:21 -07:00
Brian Warner
cc68a81326 bump Twisted dep to 11.1.0, thus simplify IntishPermissions 2015-03-31 11:06:14 -07:00
Brian Warner
26c8abadec Fix ftp 'ls' to work with current Twisted-15.0.0 
refs ticket:2394

It's kind of a hack, but Twisted changed the API and I couldn't find a
cleaner way to detect which form of "permissions" value the Twisted FTP
server wants.

I've manually tested it against 14.0.2 and 15.0.0.
 2015-03-31 11:06:14 -07:00
Daira Hopwood
b6be693cbe Add a test, add missing imports. refs #2388 
This tests ftpd, but not sftpd. Doing this sort of test on sftpd
requires the creation of a valid pubkey/privkey file pair, which is more
work than I want to do right now.

init_ftp/init_sftp were changed to interpret the configured
accounts.file as relative to the node's basedir, with
abspath_expanduser_unicode(accountfile, base=self.basedir).
This would happen naturally in a real node, since it os.chdir()s
to the basedir before doing anything. But tests don't do that.

Author: Brian Warner <warner@lothar.com>
Author: Daira Hopwood <daira@jacaranda.org>
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2015-03-24 17:10:00 +00:00
Daira Hopwood
3066039f0f Ensure that path parameters to SFTPServer and FTPServer constructors are unicode. refs #2388 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2015-03-24 16:46:40 +00:00
Daira Hopwood
14f783086f Change uses of os.path.expanduser and os.path.abspath. refs #2235 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2015-01-30 00:50:18 +00:00
Daira Hopwood
0e9f0da51f Reserve all keytypes starting with "ssh-" in an accounts file, rather than only "ssh-rsa" and "ssh-dsa". 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2015-01-06 19:09:04 +00:00
Daira Hopwood
6c756ba3e9 Simplify key checking code by inlining _allowedKey and _correctSignature. refs #1141 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2015-01-06 18:10:41 +00:00
Jean-Paul Calderone
102d581a90 Give out FTPAvatarID objects instead. 2015-01-04 09:48:38 -05:00
Jean-Paul Calderone
38aee94a3e Add the rest of the failure-case tests and a success-case test. Update the implementation to make them pass. 2015-01-04 09:44:56 -05:00
David-Sarah Hopwood
50c6562901 Improve SFTP error handling and remove use of IFinishableConsumer. fixes #1525 
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
 2013-03-19 05:37:02 +00:00
David-Sarah Hopwood
48a2989ee1 sftpd.py: switch from using bare asserts to assertutil. 
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
 2013-03-19 02:34:27 +00:00
Peter Le Bek
7f6ee7e918 ftpd file `size' attribute must be an integer 2012-03-22 13:18:06 +00:00