ExternalVendorCode/tahoe-lafs
1
0
Fork 0
mirror of https://github.com/tahoe-lafs/tahoe-lafs.git synced 2025-01-15 01:09:49 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
9,447 Commits 185 Branches 90 Tags 91 MiB
7be1883910
Commit Graph

357 Commits

Author SHA1 Message Date
meejah
87ad3cb8e9 fix 2880 
- remember upload URI so we don't re-download
- account for empty files in "should_download"
- fix 'conflicted' asserts in tests
 2017-12-05 12:02:16 -07:00
meejah
672475cb2b Multiple magic-folders 
This moves all magic-folder configs to a single YAML
file. We load legacy config fine and don't mess with
legacy config unless you use a magic-folder command that
changes the config.

Increase test coverage
 2017-12-05 10:34:48 -07:00
David Stainton
366fe039e8 magic-folder: change stopService to call MultiService.stopService 2017-04-10 11:36:58 -07:00
meejah
49ae2c5eb7 use @implementer instead of implements 
Also, derive some more things from 'object'
 2017-02-27 11:01:30 -07:00
meejah
e32b664b2b Adjust default poll_interval 
Also adds a --poll-interval option to both 'magic-folder join'
and 'magic-folder create' so that the integration tests can pass
something "very short".
 2016-12-14 20:29:41 -07:00
meejah
da4e1589bb Rename _turn_delay and friends to make things more clear 
This forces the Uploader and Downloader to implement a _scan_delay
method and makes the naming more consistent with what's actually
happening. Also, fix a few "bugs" in the names of args in the
mocks for some tests.
 2016-10-25 12:38:57 -07:00
meejah
ddaa29ce39 refactor/white-space MagicFolder creation 2016-10-25 12:38:57 -07:00
Brian Warner
203067810d magic_folder: announce debug on/off to log, not stdout 2016-10-24 09:37:35 -07:00
Brian Warner
ae5637dc9c test_magic_folder: turn off debugging when done 2016-10-24 09:23:16 -07:00
David Stainton
d6cbe44786 Enable debug logging for test_move_tree magic-folder unit test 2016-10-24 09:15:09 -07:00
meejah
de8e61ddf1 it's okay if we already cancelled it 2016-09-08 12:18:58 -07:00
meejah
622ae646b1 fix shutdown 2016-09-08 12:18:58 -07:00
meejah
0466522868 fix _should_download similar to other place 2016-08-30 16:44:52 -06:00
Daira Hopwood
bd62eba934 Work in progress on Magic Folder concurrent write problem. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-08-30 16:44:26 -06:00
meejah
054efe055c Fix file-operations/inotify testing 
This re-factors the magic-folder tests to abstract
the whole "do a file operation" so we can properly
send fake (or wait for real) inotify events to the
uploader/downloader. This speeds up the tests quite
a bit and makes test_alice_bob reasonable again (at
about 1.5s instead of over 30s).
 2016-08-18 10:41:40 -06:00
meejah
75a218d80f Show last time for successful scans too 2016-08-10 19:44:14 -06:00
meejah
8a33fc612a Add magic-folder indicator to main welcome page 
This doesn't reveal very much information, but does tell
you if magic-folder is currently working and if not it will
indicate when the last attempt to do a remote scan was.
 2016-08-09 18:07:00 -06:00
meejah
69b86ebe9a Improve startup error-reporting 
This keeps re-trying the initial magic-folder scan and alerts
the user (via logs only :/) until it succeeds at least once.

After this happens and the node has started up, it will continue
to re-try if enough storage servers go away later such that the
remote collection can't be retrieved.
 2016-08-09 14:16:11 -04:00
meejah
87acfe968e pyflakes warnings 2016-07-21 12:35:59 -07:00
meejah
4509c7dafd Fix magic-folder 'status' command 2016-07-21 12:35:59 -07:00
meejah
304da362f7 remove prints 2016-07-21 12:35:59 -07:00
meejah
3de2f9c756 get rid of a bunch of debug 2016-07-21 12:35:59 -07:00
meejah
fce5b789dc fixup rebase conflict mis-resolutions 2016-07-21 12:35:59 -07:00
Daira Hopwood
8a83cff14d Make _process return a boolean to indicate whether processing succeeded, 
and only increment objects_succeeded if that is the case.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
b949082f8e Change type of mtime and ctime stored in magicfolderdb to integer nanoseconds. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
meejah
a50d0e5c8b WIP and debugging things 
Conflicts:
	src/allmydata/test/test_magic_folder.py
 2016-07-21 12:35:59 -07:00
Daira Hopwood
e67d23b8fa Fix negative counts for object_queued. fixes ticket:2731 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
meejah
378ce27cbd Make retry_download test work, and some bonus comments 2016-07-21 12:35:59 -07:00
Daira Hopwood
9e5b01b3ae Swallow the error from Downloader._when_queue_is_empty. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
meejah
e6104cd1a2 Replace _lazy_tail with iterative loop, cleanup tests (#2412) 
* uses @inlineCallbacks to turn the _lazy_tail recursion into
   a "real" looking loop;
 * remove the need for "immediate" vs delayed iteration of said loop;
 * make it easier for the unit-tests to control the behavior of the
   uploader/downloader;
 * consolidates (some) setup/teardown code into the setUp and tearDown
   hooks provided by unittest so unit-tests aren't doing that themselves
 * re-factors some of the unit-tests to use an @inlineCallbacks style
   so they're easier to follow and debug

This doesn't tackle the "how to know when our inotify events have arrived"
problem the unit-tests still have, nor does it eliminate the myriad bits
of state that get added to tests via all the MixIns.
 2016-07-21 12:35:59 -07:00
David Stainton
2963e9e50c Use os.path.dirname instead of split and rename var to initial_path_u 2016-07-21 12:35:59 -07:00
David Stainton
ee44732d03 Break out our chmod while loop into fielutils.py 2016-07-21 12:35:59 -07:00
David Stainton
51ccec7a33 Use chmod instead of changing umask 
Conflicts:
	src/allmydata/frontends/magic_folder.py
 2016-07-21 12:35:59 -07:00
David Stainton
3d909682a4 Add comment about FUDGE_SECONDS and refer to our design doc 2016-07-21 12:35:59 -07:00
meejah
bf3bf70814 add docstrings 2016-07-21 12:35:59 -07:00
meejah
8ba2912ff4 make history longer 2016-07-21 12:35:59 -07:00
meejah
7e84b0cf57 flesh out IQueuedItem 2016-07-21 12:35:59 -07:00
meejah
86abe56d91 Flesh out "tahoe magic-folder status" command 
Adds:

 - a JSON endpoint
 - CLI to display information
 - QueuedItem + IQueuedItem for uploader/downloader
 - IProgress interface + PercentProgress implementation
 - progress= args to many upload/download APIs
 2016-07-21 12:35:59 -07:00
David Stainton
71e3164e92 Further work in progress refinements to unit tests 
wip from pairing with Daira and Meejah
 2016-07-21 12:35:59 -07:00
David Stainton
38dd49ee5a Add logging for Downloader.stop. 2016-07-21 12:35:59 -07:00
David Stainton
b4153659d3 Minor cleanup and logging fix. 2016-07-21 12:35:59 -07:00
David Stainton
5380ff8f5a Clean up queue loging and add a clock advance in unit test 
not yet working
 2016-07-21 12:35:59 -07:00
David Stainton
e1822c0518 WIP 2016-07-21 12:35:59 -07:00
David Stainton
96b846f762 Add ignore pending argument to _periodic_full_scan 2016-07-21 12:35:59 -07:00
David Stainton
a5163c32cc Only perform full scan if pending set is empty 2016-07-21 12:35:59 -07:00
David Stainton
e9e74c43e1 Naive periodic full scan 2016-07-21 12:35:59 -07:00
David Stainton
9b4b43cf19 Fix uploader's _process to extend queue after scan and count stats properly 2016-07-21 12:35:59 -07:00
David Stainton
d2971fbabe Daira's excellent fixes to the uploader from our pairing session 2016-07-21 12:35:59 -07:00
David Stainton
70f66677be Teach uploader's _scan to use the deque 2016-07-21 12:35:59 -07:00
Daira Hopwood
2043bed3f1 Delay only the download scan not the turning of our event queue. 
Author: David Stainton <david@leastauthority.com>
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
David Stainton
0273034b0c Fix umask again 2016-07-21 12:35:59 -07:00
David Stainton
ae59b17e86 Fix umask usage bit fiddling and use try finally blocks 2016-07-21 12:35:59 -07:00
David Stainton
f2a7978b29 Add download.umask config option with default of 077 2016-07-21 12:35:59 -07:00
David Stainton
623ed3516e Add umask to Downloader 2016-07-21 12:35:59 -07:00
David Stainton
79f3c5d81a Remove magic-folder exclude stat 2016-07-21 12:35:59 -07:00
Daira Hopwood
6b4d3b8c6d Refactor is_new_file. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
02dacbc067 Minor simplification. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
David Stainton
e990483750 Simply conflict detection by removing nested if statements 2016-07-21 12:35:59 -07:00
David Stainton
6ed4480eb9 Scan our own dmd upon start only 2016-07-21 12:35:59 -07:00
meejah
ed4e4d053e Add a unit-test and correct the code for "already deleted" 
If a Downloader decides that it needs to delete a file, but that
file is already gone locally, the exeption is caugt and a log message
produced.
 2016-07-21 12:35:59 -07:00
Daira Hopwood
9d59ef493f WIP: Refactoring to get db fields in one query. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
42f535edf9 Fix pending upload conflict detection. 
Author: David Stainton <david@leastauthority.com>
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
e8b6b31373 Detect remote conflict by checking for pending upload. 
Author: David Stainton <david@leastauthority.com>
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
David Stainton
c7f38a496d Add debugging print statement for timestamp comparison 2016-07-21 12:35:59 -07:00
David Stainton
edc671f577 Add last uploaded timestamp comparison for remote conflict detection 2016-07-21 12:35:59 -07:00
Daira Hopwood
c7f298bce5 Fix some miscapture bugs. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
868c38d228 Workaround. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
b21a3dc070 Simplify _notify and improve logging. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
17384ac512 Simplify _scan_remote_* and remove Downloader._download_scan_batch attribute. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
d686575393 Cosmetics. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
4f41be0e70 Downloader doesn't need the pending set. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
017d63b47b Delete redundant is_ready attribute from MagicFolder. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
5f98d73216 Add precondition to Uploader._process. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
0c12162bc8 Debugging WIP. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
84af199d51 WIP: exclude own dirnode from scan. This is not quite right; we shouldn't exclude it on startup. refs #2553 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
ea14b1bebe Don't add subdirectory watches if the platform's notifier doesn't require them. refs #2559 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
6bb0963197 Improve reporting of assertion failures. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
2a79252a72 Fix some path Unixisms. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
1abc2c39c8 Don't download the deletion marker file unnecessarily. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
c6191257ea Distinguish deletion of directories. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
d6bb49b1a6 Rename deleted files to .backup rather than unlinking them. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
David Stainton
99d1339d38 Add basic bob upload test and fix conflict detect 2016-07-21 12:35:59 -07:00
David Stainton
9ed61f1eb4 Count conflicted objects 2016-07-21 12:35:59 -07:00
Daira Hopwood
7a3b35e29c Basic remote conflict detection based on ancestor uri 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
f75dda7635 Avoid .format, since it is inconsistent between Python 2.6 and 2.7 (and the rest of Tahoe-LAFS doesn't use it). 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
e4082a45d3 Add counter for uploader.objects_not_uploaded. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
meejah
c883977eb6 add the 'spurious' notifies 2016-07-21 12:35:59 -07:00
meejah
d61b8ed39e implement 'delete' functionality, with tests 2016-07-21 12:35:59 -07:00
Daira Hopwood
cec315da49 Add test that we don't write files outside the magic folder directory. refs ticket:2506 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
5d5d804719 More debug logging. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
8184ed4414 Watch for IN_CREATE events but filter them out for non-directories. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
cbec939d86 Implement creating local directories in downloader. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
60ab5a057a Decode names in the scanned remote. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
a3fa1617c2 Refactoring to allow logging from _write_downloaded_file and _rename_conflicted_file. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
6cb41cadf5 Simplify and fix non-existent-file handling. 
Also make the existent and non-existent cases as similar as possible,
with a view to merging them.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
cb905d8d97 Logging/debugging improvements. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
ac5730ea1c Correct a call to did_upload_version in the downloader. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
28587b5b2a Restore a call to increment files_uploaded that was mistakenly removed. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
David Stainton
2fcaa8c2b2 Teach uploader+downloader to use to db schema 
here we attempt to fix all the unit tests as well...
however two tests still fail
 2016-07-21 12:35:59 -07:00
Daira Hopwood
5754c01cb5 Add magicfolderdb.py. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
David Stainton
e2cee9ba2a WIP 2016-07-21 12:35:59 -07:00
David Stainton
f8bb229ab7 For all downloaded files ensure parent dir exists 2016-07-21 12:35:59 -07:00
meejah
94a2ef0095 Make downloader delay a class-variable 
This gives the integration-style CLI-based tests a chance
to set the delay to 0 before the first 3-second delayed
call is queued to _lazy_tail in the Downloader
 2016-07-21 12:35:59 -07:00
Daira Hopwood
e03a1e3361 WIP. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
262e238a7d Magic Folder file moves. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
3170cab0fc Prepare to move drop_upload.py to magic_folder.py. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
76a178969d Move backupdb.py to src/allmydata. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
5f143f7635 Enable Windows inotify support. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
61b3a9cef2 Add magic folder db. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
7b4067850f Unicode path fixes for drop-upload. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-07-21 12:35:59 -07:00
Daira Hopwood
a56a3adaae Teach StorageFarmBroker to fire a deferred when a connection threshold is reached. refs #1449 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2016-04-26 12:26:39 -06:00
Daira Hopwood
5d046c597e drop-upload.py: fix error messages. 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2015-05-13 14:21:17 +01:00
Brian Warner
eda00a5c58 typo 2015-03-31 11:07:21 -07:00
Brian Warner
cc68a81326 bump Twisted dep to 11.1.0, thus simplify IntishPermissions 2015-03-31 11:06:14 -07:00
Brian Warner
26c8abadec Fix ftp 'ls' to work with current Twisted-15.0.0 
refs ticket:2394

It's kind of a hack, but Twisted changed the API and I couldn't find a
cleaner way to detect which form of "permissions" value the Twisted FTP
server wants.

I've manually tested it against 14.0.2 and 15.0.0.
 2015-03-31 11:06:14 -07:00
Daira Hopwood
b6be693cbe Add a test, add missing imports. refs #2388 
This tests ftpd, but not sftpd. Doing this sort of test on sftpd
requires the creation of a valid pubkey/privkey file pair, which is more
work than I want to do right now.

init_ftp/init_sftp were changed to interpret the configured
accounts.file as relative to the node's basedir, with
abspath_expanduser_unicode(accountfile, base=self.basedir).
This would happen naturally in a real node, since it os.chdir()s
to the basedir before doing anything. But tests don't do that.

Author: Brian Warner <warner@lothar.com>
Author: Daira Hopwood <daira@jacaranda.org>
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2015-03-24 17:10:00 +00:00
Daira Hopwood
3066039f0f Ensure that path parameters to SFTPServer and FTPServer constructors are unicode. refs #2388 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2015-03-24 16:46:40 +00:00
Daira Hopwood
14f783086f Change uses of os.path.expanduser and os.path.abspath. refs #2235 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2015-01-30 00:50:18 +00:00
Daira Hopwood
0e9f0da51f Reserve all keytypes starting with "ssh-" in an accounts file, rather than only "ssh-rsa" and "ssh-dsa". 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2015-01-06 19:09:04 +00:00
Daira Hopwood
6c756ba3e9 Simplify key checking code by inlining _allowedKey and _correctSignature. refs #1141 
Signed-off-by: Daira Hopwood <daira@jacaranda.org>
 2015-01-06 18:10:41 +00:00
Jean-Paul Calderone
102d581a90 Give out FTPAvatarID objects instead. 2015-01-04 09:48:38 -05:00
Jean-Paul Calderone
38aee94a3e Add the rest of the failure-case tests and a success-case test. Update the implementation to make them pass. 2015-01-04 09:44:56 -05:00
David-Sarah Hopwood
50c6562901 Improve SFTP error handling and remove use of IFinishableConsumer. fixes #1525 
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
 2013-03-19 05:37:02 +00:00
David-Sarah Hopwood
48a2989ee1 sftpd.py: switch from using bare asserts to assertutil. 
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>
 2013-03-19 02:34:27 +00:00
Peter Le Bek
7f6ee7e918 ftpd file `size' attribute must be an integer 2012-03-22 13:18:06 +00:00
Peter Le Bek
14a50f258a fix ftpd mtime retrieval 2012-03-30 23:41:19 +00:00
david-sarah
d9c7dfa4a0 Update more links from http: to https: in documentation and comments. 2012-03-13 20:26:54 +00:00
david-sarah
1b7b3bd0d4 SFTP: add a comment about a subtle interaction between OverwriteableFileConsumer and GeneralSFTPFile, and test the case it is commenting on. 2011-09-03 15:23:04 -07:00
Kevan Carstensen
a87fc8ffab frontends/sftpd: Resolve incompatibilities between SFTP frontend and MDMF changes 2011-08-01 19:12:07 -07:00
david-sarah
c102056ac1 test_client.py: add a test that the drop-uploader is initialized correctly by client.py. Also give the DropUploader service a name, which is necessary for the test. refs #1429 2011-08-09 20:05:38 -07:00
david-sarah
10ee22f50e drop-upload: rename 'start' method to 'startService', which is what you're supposed to use to start a Service. refs #1429 2011-08-09 20:03:45 -07:00
Zooko O'Whielacronx
612abca271 drop-upload: report the configured local directory being absent differently from it being a file 
refs #1429
 2011-08-09 15:09:30 -07:00
Zooko O'Whielacronx
b7683d9b83 drop-upload: rename the 'upload.uri' parameter to 'upload.dircap', and a couple of cleanups to error messages. refs #1429 
I rerecorded this patch, originally by David-Sarah, to use "darcs replace" instead of editing to do the renames. This uncovered one missed rename in Client.init_drop_uploader. (Which also means that code isn't exercised by the current unit tests.)
refs #1429
 2011-08-09 15:05:08 -07:00
david-sarah
bbed522033 Remove all trailing whitespace from .py files. 2011-08-08 17:11:17 -07:00
david-sarah
32a7717205 Drop-upload frontend, rerecorded for 1.9 beta (and correcting a minor mistake). Includes some fixes for Windows but not the Windows inotify implementation. fixes #1429 2011-08-08 16:40:49 -07:00
david-sarah
a2699ea6f6 SFTP: write an error message to standard error for unrecognized shell commands. Change the existing message for shell sessions to be written to standard error, and refactor some duplicated code. Also change the lines of the error messages to end in CRLF, and take into account Kevan's review comments. fixes #1442, #1446 2011-07-29 16:31:02 -07:00
david-sarah
0f83b76190 frontends/ftpd.py: remove the check for IWriteFile.close since we're now guaranteed to be using Twisted >= 10.1 which has it. 2011-07-21 17:03:20 -07:00
david-sarah
ed67df866c SFTP: report unknown sizes as "0" instead of "?", to satisfy some clients. fixes #1337 2011-01-27 22:22:49 -08:00
david-sarah
05022dca36 Move EncryptedTemporaryFile from SFTP frontend to allmydata.util.fileutil, and make the FTP frontend also use it (fixing #1083). 2010-07-11 14:37:21 -07:00
david-sarah
15ddab08ed SFTP: address some of the comments in zooko's review (#1106). 2010-07-11 19:55:37 -07:00
david-sarah
88169328be SFTP: don't call .stopProducing on the producer registered with OverwriteableFileConsumer (which breaks with warner's new downloader). 2010-06-28 16:19:26 -07:00
david-sarah
c1412c8a81 SFTP: get 'ctime' attribute from 'tahoe:linkmotime'. 2010-06-16 20:37:44 -07:00
david-sarah
a3acb2b773 SFTP: remove a dubious use of 'pragma: no cover'. 2010-06-13 09:43:56 -07:00
david-sarah
b67f8b66c8 SFTP: further small improvements to test coverage. Also ensure that after a test failure, later tests don't fail spuriously due to the checks for heisenfile leaks. 2010-06-11 20:07:37 -07:00
david-sarah
b4e074d250 SFTP: further improve test coverage (paths containing '.', bad data for posix-rename extension, and error in test of openShell). 2010-06-11 14:31:42 -07:00
david-sarah
52f87904ed SFTP: improve test coverage for no-write on mutable files, and check for heisenfile table leaks in all relevant tests. Delete test_memory_leak since it is now redundant. 2010-06-11 13:57:52 -07:00
david-sarah
3c44389440 SFTP: fix most significant memory leak described in #1045 (due to a file being added to all_heisenfiles under more than one direntry when renamed). 2010-06-09 01:00:03 -07:00
david-sarah
80a89d3931 stringutils.py, sftpd.py: Portability fixes for Python <= 2.5. 2010-06-08 18:33:02 -07:00
david-sarah
4b5927fdb8 SFTP: suppress NoSuchChildError if heisenfile attributes have been updated in setAttrs, in the case where the parent is available. 2010-06-07 23:37:53 -07:00
david-sarah
195b4afa8f SFTP: ignore permissions when opening a file (needed for sshfs interoperability). 2010-06-07 22:57:00 -07:00
Zooko O'Whielacronx
08058c1073 fix syntax of assertion handling that isn't portable to older versions of Python 2010-06-04 00:58:05 -07:00
david-sarah
a2cff856ff Resolve merge conflict for sftpd.py 2010-06-03 11:25:37 -07:00
david-sarah
2dcb62de90 SFTP: possible fix for metadata times being shown as the epoch. 2010-06-02 16:45:14 -07:00
david-sarah
c9984c40e9 SFTP: further improvements to test coverage. 2010-06-02 16:44:22 -07:00
david-sarah
2a791b0d05 SFTP: improve test coverage. Also make creating a directory fail when permissions are read-only (rather than ignoring the permissions). 2010-06-01 21:19:34 -07:00
david-sarah
db394671e9 SFTP: fix a bug that caused the temporary files underlying EncryptedTemporaryFiles not to be closed. 2010-05-31 22:53:10 -07:00
david-sarah
de95140b7b SFTP: changes for #1063 ('no-write' field) including comment:1 (clearing owner write permission diminishes to a read cap). Includes documentation changes, but not tests for the new behaviour. 2010-05-31 22:11:39 -07:00
david-sarah
9faa02deb5 SFTP: the same bug as in _sync_heisenfiles also occurred in two other places. 2010-05-29 23:01:27 -07:00
david-sarah
cccdd32806 SFTP: another try at fixing the _sync_heisenfiles bug. 2010-05-29 22:52:54 -07:00
david-sarah
51b0d8bde3 SFTP: fix silly bug in _sync_heisenfiles ('f is not ignore' vs 'not (f is ignore)'). 2010-05-29 22:38:07 -07:00
david-sarah
b9d3cf12b8 SFTP: log when a sync completes. 2010-05-29 22:18:40 -07:00
david-sarah
c4a8af4d27 SFTP: fix bug in previous logging patch. 2010-05-29 22:00:00 -07:00
david-sarah
3395a7e154 SFTP: more logging to track down OpenOffice hang. 2010-05-29 21:08:09 -07:00
david-sarah
4be24a89df SFTP: avoid blocking close on a heisenfile that has been abandoned or never changed. Also, improve the logging to help track down a case where OpenOffice hangs on opening a file with FXF_READ|FXF_WRITE. 2010-05-29 19:55:44 -07:00
david-sarah
8ed28a95e7 Move suppression of DeprecationWarning about BaseException.message from sftpd.py to main __init__.py. Also, remove the global suppression of the 'integer argument expected, got float' warning, which turned out to be a bug. 2010-05-28 22:05:37 -07:00
david-sarah
e867985539 SFTP: cater to clients that assume a file is created as soon as they have made an open request; also, fix some race conditions associated with closing a file at about the same time as renaming or removing it. 2010-05-28 21:52:53 -07:00
david-sarah
0b888f8201 SFTP: 'sync' any open files at a direntry before opening any new file at that direntry. This works around the sshfs misbehaviour of returning success to clients immediately on close. 2010-05-25 16:02:57 -07:00
david-sarah
a143b1297b SFTP: handle removing a file while it is open. Also some simplifications of the logout handling. 2010-05-25 11:42:10 -07:00
david-sarah
e81ce2bbd2 SFTP: a posix-rename response should actually return an FXP_STATUS reply, not an FXP_EXTENDED_REPLY as Twisted Conch assumes. Work around this by raising an SFTPError with code FX_OK. 2010-05-24 20:33:23 -07:00
david-sarah
8f86490d48 SFTP: fix problem with posix-rename code returning a Deferred for the renamed filenode, not for the result of the request (an empty string). 2010-05-24 19:02:09 -07:00
david-sarah
210578da63 SFTP: fix time handling to make sure floats are not passed into twisted.conch, and to print times in the future less ambiguously in directory listings. 2010-05-24 16:04:12 -07:00
david-sarah
549dccd608 SFTP: name of the POSIX rename extension should be 'posix-rename@openssh.com', not 'extposix-rename@openssh.com'. 2010-05-23 19:11:56 -07:00
david-sarah
6e114e44b5 SFTP: avoid race condition where .write could be called on an OverwriteableFileConsumer after it had been closed. 2010-05-23 16:38:30 -07:00
david-sarah
c6b414c815 SFTP: log tracebacks for RAISEd exceptions. 2010-05-23 15:15:35 -07:00
david-sarah
725daee788 SFTP: more logging to investigate behaviour of getAttrs(path). 2010-05-23 13:42:36 -07:00
david-sarah
75357fd2d5 SFTP: fix pyflakes warnings; drop 'noisy' versions of eventually_callback and eventually_errback; robustify conversion of exception messages to UTF-8. 2010-05-23 07:09:05 -07:00
david-sarah
38964fb35a SFTP: fixes and test cases for renaming of open files. 2010-05-22 20:25:49 -07:00
david-sarah
5974773969 SFTP: Fix error in support for getAttrs on an open file, to index open files by directory entry rather than path. Extend that support to renaming open files. Also, implement the extposix-rename@openssh.org extension, and some other minor refactoring. 2010-05-21 20:58:36 -07:00
david-sarah
ce3872d10e SFTP: allow getAttrs to succeed on a file that has been opened for creation but not yet uploaded or linked (part of #1050). 2010-05-19 20:56:13 -07:00
david-sarah
7f9e351bce SFTP: improve logging so that results of requests are (usually) logged. 2010-05-19 17:36:52 -07:00
david-sarah
a62733ed08 SFTP: add tests for more combinations of open flags. 2010-05-18 22:39:33 -07:00
david-sarah
f0945526ce SFTP: allow FXF_WRITE | FXF_TRUNC (#1050). 2010-05-18 21:32:40 -07:00
david-sarah
bde27cab57 SFTP: remove another case where we were logging data. 2010-05-18 18:27:13 -07:00
david-sarah
d8b096a054 SFTP: avoid logging all data passed to callbacks. 2010-05-18 17:06:51 -07:00
david-sarah
819eaa74c1 SFTP: fixes related to reporting of permissions (needed for sshfs). 2010-05-17 22:45:21 -07:00
david-sarah
8d118e28f1 SFTP: change error code returned for ExistingChildError to FX_FAILURE (fixes gvfs with some picky programs such as gedit). 2010-05-17 17:42:05 -07:00
david-sarah
57699fd1eb SFTP: fixed bugs that caused hangs during write (#1037). 2010-05-16 21:42:28 -07:00
david-sarah
5f9c10901b SFTP: work around a probable bug in twisted.conch.ssh.session:loseConnection(). Also some minor error handling cleanups. 2010-05-16 18:26:06 -07:00
david-sarah
6a2b0229f0 SFTP: Support statvfs extensions, avoid logging actual data, and decline shell sessions politely. 2010-05-16 08:43:47 -07:00
david-sarah
31994aaf74 SFTP: fix error in SFTPUserHandler arguments introduced by execCommand patch. 2010-05-15 18:40:45 -07:00
david-sarah
8028bf01bc SFTP: implement execCommand to interoperate with clients that issue a 'df -P -k /' command. Also eliminate use of Zope adaptation. 2010-05-15 18:27:54 -07:00
david-sarah
6ba6c97a7a sftpd.py: 'log.OPERATIONAL' should be just 'OPERATIONAL'. 2010-05-15 08:55:33 -07:00
david-sarah
5ab4a376b7 Attempt to fix #1040 by making SFTPUser implement ISession. 2010-05-14 17:57:19 -07:00
david-sarah
7409d1e019 Eliminate Windows newlines from sftpd.py. 2010-05-14 17:56:56 -07:00
david-sarah
14d0721246 Update SFTP implementation and tests: fix #1038 and switch to foolscap logging; also some code reorganization. 2010-05-13 21:31:13 -07:00
david-sarah
5248c808ac New SFTP implementation: mutable files, read/write support, streaming download, Unicode filenames, and more 2010-05-11 22:54:07 -07:00
david-sarah
e76092e16c Change relative imports to absolute 2010-02-26 01:14:33 -07:00
david-sarah
100548257a Use if instead of assert to check for twisted ftp patch 2010-01-26 17:55:29 -08:00
Zooko O'Whielacronx
4a4305e7e6 ftpd: clearer error message if Twisted needs a patch (by Nils Durner) 2010-01-26 06:34:11 -08:00
Brian Warner
96834da0a2 Simplify immutable download API: use just filenode.read(consumer, offset, size) 
* remove Downloader.download_to_data/download_to_filename/download_to_filehandle
* remove download.Data/FileName/FileHandle targets
* remove filenode.download/download_to_data/download_to_filename methods
* leave Downloader.download (the whole Downloader will go away eventually)
* add util.consumer.MemoryConsumer/download_to_data, for convenience
  (this is mostly used by unit tests, but it gets used by enough non-test
   code to warrant putting it in allmydata.util)
* update tests
* removes about 180 lines of code. Yay negative code days!

Overall plan is to rewrite immutable/download.py and leave filenode.read() as
the sole read-side API.
 2009-12-01 17:53:30 -05:00