Commit Graph

3990 Commits

Author SHA1 Message Date
David Stainton
3d909682a4 Add comment about FUDGE_SECONDS and refer to our design doc 2016-07-21 12:35:59 -07:00
Daira Hopwood
807cfbf0dc Fix pyflakes warnings.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
meejah
9cfcb1b3aa remove humanize, use internal method, teach internal method to understand timedelta 2016-07-21 12:35:59 -07:00
meejah
573a5481d1 add Bob leaving to smoke-tests 2016-07-21 12:35:59 -07:00
meejah
c217d1f8f3 fix return-valid of leave 2016-07-21 12:35:59 -07:00
meejah
bf3bf70814 add docstrings 2016-07-21 12:35:59 -07:00
meejah
8ba2912ff4 make history longer 2016-07-21 12:35:59 -07:00
meejah
7e84b0cf57 flesh out IQueuedItem 2016-07-21 12:35:59 -07:00
meejah
05ec62c4b1 humanize doesn't report a proper version 2016-07-21 12:35:59 -07:00
meejah
5ed24913d2 add humanize dependency 2016-07-21 12:35:59 -07:00
meejah
658470ba6e Add some magicfolder database tests 2016-07-21 12:35:59 -07:00
meejah
2573cf18ef Add simple auth-token to get JSON data 2016-07-21 12:35:59 -07:00
meejah
86abe56d91 Flesh out "tahoe magic-folder status" command
Adds:

 - a JSON endpoint
 - CLI to display information
 - QueuedItem + IQueuedItem for uploader/downloader
 - IProgress interface + PercentProgress implementation
 - progress= args to many upload/download APIs
2016-07-21 12:35:59 -07:00
David Stainton
3df0a82a38 Add cli stub for magic-folder status command 2016-07-21 12:35:59 -07:00
David Stainton
af10ab43ee Daira's fix during session with David and Meejah. 2016-07-21 12:35:59 -07:00
David Stainton
71e3164e92 Further work in progress refinements to unit tests
wip from pairing with Daira and Meejah
2016-07-21 12:35:59 -07:00
David Stainton
9483d24f67 Fix cleanup after test_periodic_full_scan. 2016-07-21 12:35:59 -07:00
David Stainton
38dd49ee5a Add logging for Downloader.stop. 2016-07-21 12:35:59 -07:00
David Stainton
2dabd8a10b Work in progress on fixing test_periodic_full_scan. 2016-07-21 12:35:59 -07:00
David Stainton
b4153659d3 Minor cleanup and logging fix. 2016-07-21 12:35:59 -07:00
David Stainton
5380ff8f5a Clean up queue loging and add a clock advance in unit test
not yet working
2016-07-21 12:35:59 -07:00
David Stainton
e1822c0518 WIP 2016-07-21 12:35:59 -07:00
David Stainton
96b846f762 Add ignore pending argument to _periodic_full_scan 2016-07-21 12:35:59 -07:00
David Stainton
cf462d8b1f Attempt to test periodic uploader full scan 2016-07-21 12:35:59 -07:00
David Stainton
a5163c32cc Only perform full scan if pending set is empty 2016-07-21 12:35:59 -07:00
David Stainton
e9e74c43e1 Naive periodic full scan 2016-07-21 12:35:59 -07:00
David Stainton
9b4b43cf19 Fix uploader's _process to extend queue after scan and count stats properly 2016-07-21 12:35:59 -07:00
David Stainton
d2971fbabe Daira's excellent fixes to the uploader from our pairing session 2016-07-21 12:35:59 -07:00
David Stainton
70f66677be Teach uploader's _scan to use the deque 2016-07-21 12:35:59 -07:00
David Stainton
6b32841b10 Remove UNIQUE constraint from magicfolder db schema 2016-07-21 12:35:59 -07:00
Daira Hopwood
9c9a4e7aa2 Add get_pathinfo.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
278762200d Teach magic-folder join to use configutil.
Author: David Stainton <david@leastauthority.com>
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
David Stainton
cc721505e7 Add magic-folder test_scan_once_on_startup 2016-07-21 12:35:59 -07:00
Daira Hopwood
cb6291e93c Various test cleanups.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
David Stainton
3074b687be Teach join twice test to test for proper stdout and stderr values 2016-07-21 12:35:59 -07:00
David Stainton
74ad656e39 More thorough checks to the join leave join test 2016-07-21 12:35:59 -07:00
David Stainton
966c556562 Fix test_join_leave_join 2016-07-21 12:35:59 -07:00
David Stainton
1c3c9df492 Add more magic-folder join leave tests 2016-07-21 12:35:59 -07:00
David Stainton
24aa74bf53 Add unit test for join twice failure 2016-07-21 12:35:59 -07:00
Daira Hopwood
2043bed3f1 Delay only the download scan not the turning of our event queue.
Author: David Stainton <david@leastauthority.com>
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
David Stainton
9d7785f57b Fix magic folder constructor signature in test 2016-07-21 12:35:59 -07:00
David Stainton
68539e3fe0 Fix tests usage of umask 2016-07-21 12:35:59 -07:00
David Stainton
0273034b0c Fix umask again 2016-07-21 12:35:59 -07:00
David Stainton
ae59b17e86 Fix umask usage bit fiddling and use try finally blocks 2016-07-21 12:35:59 -07:00
David Stainton
f2a7978b29 Add download.umask config option with default of 077 2016-07-21 12:35:59 -07:00
David Stainton
623ed3516e Add umask to Downloader 2016-07-21 12:35:59 -07:00
David Stainton
7b9c89c6f4 remove magic_folder from config in test 2016-07-21 12:35:59 -07:00
David Stainton
189b1b2dc2 Remove magic_folder subsection from default tahoe.cfg 2016-07-21 12:35:59 -07:00
David Stainton
79f3c5d81a Remove magic-folder exclude stat 2016-07-21 12:35:59 -07:00
Daira Hopwood
cd1c166790 Improve the error reporting for 'tahoe magic-folder join/leave'. refs #2568
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
David Stainton
b4b5590b00 teach leave to: remove magic_folder section of config 2016-07-21 12:35:59 -07:00
David Stainton
d6e68129ab Add simple magic-folder leave command 2016-07-21 12:35:59 -07:00
David Stainton
8ff7642fba Teach join to also check for existing magicfolder db file 2016-07-21 12:35:59 -07:00
David Stainton
aea2cf2890 Prevent magic-folder join if already joined. 2016-07-21 12:35:59 -07:00
Daira Hopwood
6b4d3b8c6d Refactor is_new_file.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
02dacbc067 Minor simplification.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
David Stainton
e990483750 Simply conflict detection by removing nested if statements 2016-07-21 12:35:59 -07:00
David Stainton
6ed4480eb9 Scan our own dmd upon start only 2016-07-21 12:35:59 -07:00
meejah
0aaabe0ebe remove print 2016-07-21 12:35:59 -07:00
meejah
ed4e4d053e Add a unit-test and correct the code for "already deleted"
If a Downloader decides that it needs to delete a file, but that
file is already gone locally, the exeption is caugt and a log message
produced.
2016-07-21 12:35:59 -07:00
David Stainton
49b5661347 Fix test: previously we accounted for the propagation of the conflict
because alice scaned her own dmd... whereas now she does not.
2016-07-21 12:35:59 -07:00
David Stainton
eab24d022b Fix test helper _check_version_in_local_db 2016-07-21 12:35:59 -07:00
Daira Hopwood
9d59ef493f WIP: Refactoring to get db fields in one query.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
David Stainton
bca4c87e47 WIP 2016-07-21 12:35:59 -07:00
Daira Hopwood
42f535edf9 Fix pending upload conflict detection.
Author: David Stainton <david@leastauthority.com>
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
David Stainton
6eeab03b70 Create test for last upload uri conflict 2016-07-21 12:35:59 -07:00
Daira Hopwood
e8b6b31373 Detect remote conflict by checking for pending upload.
Author: David Stainton <david@leastauthority.com>
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
David Stainton
c7f38a496d Add debugging print statement for timestamp comparison 2016-07-21 12:35:59 -07:00
David Stainton
edc671f577 Add last uploaded timestamp comparison for remote conflict detection 2016-07-21 12:35:59 -07:00
Daira Hopwood
6402159106 Fix an unused import.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
c7f298bce5 Fix some miscapture bugs.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
868c38d228 Workaround.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
b21a3dc070 Simplify _notify and improve logging.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
17384ac512 Simplify _scan_remote_* and remove Downloader._download_scan_batch attribute.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
d686575393 Cosmetics.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
4f41be0e70 Downloader doesn't need the pending set.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
017d63b47b Delete redundant is_ready attribute from MagicFolder.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
3fab5caa02 test_encodingutil: fixes for Unix.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
5f98d73216 Add precondition to Uploader._process.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
583066b364 Fix test_errors.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
9646a3ce3e Fix test_move_tree.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
0c12162bc8 Debugging WIP.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
b53efdbf8b Fix a corner case for to_filepath on Windows to make it consistent with Unix.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
b9eb8932b6 test_encodingutil: add tests for FilePath-related functions.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
b6b638efd1 Depend on FilePath.asTextMode().
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
84af199d51 WIP: exclude own dirnode from scan. This is not quite right; we shouldn't exclude it on startup. refs #2553
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
ea14b1bebe Don't add subdirectory watches if the platform's notifier doesn't require them. refs #2559
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
6bb0963197 Improve reporting of assertion failures.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
a5b9cf36b8 Fix unused import.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
f77c593177 Add test for 'tahoe create-node/client/introducer' output. closes ticket:2556
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
d1eb64c01b bin\tahoe can't be run directly on Windows.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
3647ffc3a7 Correct type for Windows BOOL.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
9dafa2e759 Disable precondition that autoAdd == recursive.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
a6842946ca Fix a type error.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
0e19636146 Flush handling WIP.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
5e9792f8d7 Use fileutil.write for magic folder tests.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
f3803c058d Improve all of the Windows-specific error reporting.
Also make the Windows function declarations more readable and consistent.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
c20cf843eb Fix fileutil tests.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
dacf6ef633 More path fixes.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
f12ac3d94e Fix a test broken by the last commit.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
a622109a87 Don't include [magic_folder]enabled and local.directory fields by default.
Add a comment reminding to do the field modification properly.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
6fb7f86855 Don't use a long path for the [magic_folder]local.directory field.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
2a79252a72 Fix some path Unixisms.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
45adc30ed6 Fix test_alice_bob.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
e591c3aa81 Refactor _check_up/downloader_count.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
1abc2c39c8 Don't download the deletion marker file unnecessarily.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
c6191257ea Distinguish deletion of directories.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
d6bb49b1a6 Rename deleted files to .backup rather than unlinking them.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
David Stainton
a043a31860 Add file conflict unit test 2016-07-21 12:35:59 -07:00
David Stainton
99d1339d38 Add basic bob upload test and fix conflict detect 2016-07-21 12:35:59 -07:00
David Stainton
ee3d9b19be Fix bob's uploading test... 2016-07-21 12:35:59 -07:00
David Stainton
890e4622b9 Attempt to teach bob to upload a file 2016-07-21 12:35:59 -07:00
David Stainton
9ed61f1eb4 Count conflicted objects 2016-07-21 12:35:59 -07:00
Daira Hopwood
7a3b35e29c Basic remote conflict detection based on ancestor uri
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
f75dda7635 Avoid .format, since it is inconsistent between Python 2.6 and 2.7 (and the rest of Tahoe-LAFS doesn't use it).
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
5a2b2e61c7 Fix test_alice_bob.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
e4082a45d3 Add counter for uploader.objects_not_uploaded.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
95e2961a2e Advance Bob's clock after notifying.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
6e5e8ddd80 test_alice_bob: use magic= argument to notify, rather than self.magicfolder.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
meejah
0d466ec1f9 add excluded check 2016-07-21 12:35:59 -07:00
meejah
c883977eb6 add the 'spurious' notifies 2016-07-21 12:35:59 -07:00
Daira Hopwood
c156581b73 Fix a pyflakes warning and check existence of file in Bob's local dir.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
meejah
d61b8ed39e implement 'delete' functionality, with tests 2016-07-21 12:35:59 -07:00
meejah
946656b249 smoketest for magic-folder functionality 2016-07-21 12:35:59 -07:00
Daira Hopwood
cec315da49 Add test that we don't write files outside the magic folder directory. refs ticket:2506
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
567668eb8c Fix infinite loop in should_ignore_path for absolute paths.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
5d5d804719 More debug logging.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
57753a62bc Unicode fix for do_join.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
7d2f30643e Minor cleanups to tests.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
f754938cc9 Ensure that errors from Alice-and-Bob tests are reported correctly if setup fails.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
b56c3e06e4 Cosmetics.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
2a7bb6f444 Eliminate duplicate parsing of invite code.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
c28594268e Remaining test fixes.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
a5874070e3 Make sure that do_cli is only called with strs, and avoid unnecessary use of attributes in tests.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
ed2d27e712 Cosmetics.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
643a105991 URIs are strs.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
59eede8e88 Aliases and nicknames are Unicode.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
bb596b017d Fix call to argv_to_abspath. Also rename localdir to local_dir.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
David Stainton
b32b61e727 Fix magic-folder cli tests
convert path to abs path when matching
strings in the generated config file.
2016-07-21 12:35:59 -07:00
David Stainton
a1beef34a7 Attempt to fix cli tests 2016-07-21 12:35:59 -07:00
Daira Hopwood
b2d494837c Better but still broken tests.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
903c6f1d6c Fix check for initial '-' in argv_to_abspath.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
David Stainton
0789d295a6 Fix tests by submitting unicode args instead of str 2016-07-21 12:35:59 -07:00
David Stainton
16275cb8f6 Teach magic-folder join to use argv_to_abspath
- also we modify argv_to_abspath to through a usage error
if the name starts with a '-'

- add a test
currently the tests fail
2016-07-21 12:35:59 -07:00
David Stainton
941f67f3b0 Use argv_to_abspath for magic-folder join file path arg 2016-07-21 12:35:59 -07:00
Daira Hopwood
5f6783e382 Test creation of a subdirectory.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
8184ed4414 Watch for IN_CREATE events but filter them out for non-directories.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
ee5245ed2c Patch Downloader.REMOTE_SCAN_INTERVAL rather than setting it persistently.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
cbec939d86 Implement creating local directories in downloader.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00
Daira Hopwood
60ab5a057a Decode names in the scanned remote.
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
2016-07-21 12:35:59 -07:00