ExternalVendorCode/tahoe-lafs
1
0
Fork 0
mirror of https://github.com/tahoe-lafs/tahoe-lafs.git synced 2024-12-27 00:21:07 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
5,216 Commits 185 Branches 90 Tags 89 MiB
370e6f271e
Commit Graph

2983 Commits

Author SHA1 Message Date
Brian Warner
797828f47f Rewrite immutable downloader (#798). This patch rearranges the rest of src/allmydata/immutable/ . 2010-08-04 00:26:39 -07:00
Brian Warner
22a07e9bbe Rewrite immutable downloader (#798). This patch adds the new downloader itself. 2010-08-04 00:26:29 -07:00
Brian Warner
88d7ec2d54 util/observer.py: add EventStreamObserver 2010-08-04 00:26:12 -07:00
Brian Warner
cbcb728e7e Add a byte-spans utility class, like perl's Set::IntSpan for .newsrc files. 
Also a data-spans class, which records a byte (instead of a bit) for each
index.
 2010-08-04 00:26:00 -07:00
david-sarah
f10f9a8646 test_storage.py: avoid spurious test failure by accepting either 'Next crawl in 59 minutes' or 'Next crawl in 60 minutes'. fixes #1140 2010-08-03 03:20:58 -07:00
david-sarah
a7c474a098 CLI: further improve consistency of basedir options and add tests. addresses #118 2010-08-03 01:54:16 -07:00
david-sarah
b730385ec8 CLI: make the synopsis for 'tahoe unlink' say unlink instead of rm. 2010-08-03 01:53:59 -07:00
david-sarah
a32f625db9 CLI: make all of the option descriptions imperative sentences. 2010-08-03 01:48:01 -07:00
david-sarah
51eeea900f test_cli.py: make 'tahoe mkdir' tests slightly less dumb (check for 'URI:' in the output). 2010-08-03 01:47:20 -07:00
david-sarah
b9174b5fe7 test_cli.py: use u-escapes instead of UTF-8. 2010-08-03 01:35:38 -07:00
david-sarah
d81e8acf88 test_runner.py: add test_run_with_python_options, which checks that the Windows script changes haven't broken 'python <options> bin/tahoe'. 2010-08-01 23:25:58 -07:00
david-sarah
3ddb4a9bbf test_runner.py: fix missing import of get_filesystem_encoding 2010-08-01 23:09:02 -07:00
david-sarah
3af6f19cb0 misc/build_helpers/run_trial.py: check that the root from which the module we are testing was loaded is the current directory. This version of the patch folds in later fixes to the logic for caculating the directories to compare, and improvements to error messages. addresses #1137 2010-08-01 21:55:35 -07:00
david-sarah
88b6c57a1c Skip option arguments to the python interpreter when reconstructing Unicode argv on Windows. 2010-07-27 23:27:31 -07:00
david-sarah
1a5a33886d windows/fixups.py: improve comments and reference some relevant Python bugs. 2010-07-27 11:19:21 -07:00
david-sarah
b5f052a0d7 windows/fixups.py: make errors reported to original_stderr have enough information to debug even if we can't see the traceback. 2010-07-26 15:19:04 -07:00
david-sarah
ec6a674a5c windows/fixups.py: fix paste-o in name of Unicode stderr wrapper. 2010-07-26 14:47:36 -07:00
david-sarah
54bbc5325f windows/fixups.py: Don't rely on buggy MSVCRT library for Unicode output, use the Win32 API instead. This should make it work on XP. Also, change how we handle the case where sys.stdout and sys.stderr are redirected, since the .encoding attribute isn't necessarily writeable. 2010-07-25 21:50:19 -07:00
david-sarah
1fad717aaa test_runner.py: change to code for locating the bin/tahoe script that was missed when rebasing the patch for #1074. 2010-07-25 11:20:08 -07:00
david-sarah
9d04b2a317 Add missing windows/fixups.py (for setting up Unicode args and output on Windows). 2010-07-25 02:28:49 -07:00
david-sarah
37b07a545f Changes to Tahoe needed to work with new zetuptoolz (that does not use .exe wrappers on Windows), and to support Unicode arguments and stdout/stderr -- v5 2010-07-25 01:32:16 -07:00
david-sarah
54a9ba8232 scripts/common.py: fix an error introduced when rebasing to the ticket798 branch, which caused base directories to be duplicated in self.basedirs. 2010-08-01 23:49:29 -07:00
david-sarah
f952532f08 Basedir/node directory option improvements for ticket798 branch. addresses #188, #706, #715, #772, #890 2010-08-01 21:30:04 -07:00
david-sarah
16647b49f6 scripts/create_node.py: allow nickname to be Unicode. Also ensure webport is validly encoded in config file. 2010-08-01 17:02:12 -07:00
david-sarah
9aae9208e3 test_runner.py: Fix error in message arguments to 'fail' calls. 2010-08-01 18:35:26 -07:00
david-sarah
02199db615 Additional Unicode basedir changes for ticket798 branch. 2010-08-01 18:05:52 -07:00
david-sarah
d3a8ef762f Unicode basedir changes for ticket798 branch. 2010-08-01 16:53:10 -07:00
david-sarah
013954c265 fileutil: change WindowsError to OSError in abspath_expanduser_unicode, because WindowsError might not exist. 2010-07-25 15:26:03 -07:00
david-sarah
2b33704d61 test_system: correct a failure in _test_runner caused by Unicode basedir patch on non-Unicode platforms. 2010-07-23 20:21:23 -07:00
david-sarah
f036dfaa4b Fix test failures due to Unicode basedir patches. 2010-07-24 18:03:18 -07:00
david-sarah
28e6ad51a7 util.encodingutil: change quote_output to do less unnecessary escaping, and to use double-quotes more consistently when needed. This version avoids u-escaping for characters that are representable in the output encoding, when double quotes are used, and includes tests. fixes #1135 2010-07-23 00:53:14 -07:00
david-sarah
618db4867c Replace uses of os.path.abspath with abspath_expanduser_unicode where necessary. This makes basedir paths consistently represented as Unicode. 2010-07-21 17:14:18 -07:00
david-sarah
11b18824c7 util.fileutil, test.test_util: add abspath_expanduser_unicode function, to work around <http://bugs.python.org/issue3426>. util.encodingutil: add a convenience function argv_to_abspath. 2010-07-21 16:15:07 -07:00
Zooko O'Whielacronx
24c72f2942 upload: tidy up logging messages 
reformat code to be less than 100 chars wide, refactor formatting of logging messages, add log levels to some logging messages, M-x whitespace-cleanup
 2010-08-02 00:02:12 -07:00
jacob.lyles
e276050d5a abbreviate time edge case python2.5 unit test 2010-07-29 14:06:38 -07:00
david-sarah
23c649c897 test_upload.py: rename test_problem_layout_ticket1124 to test_problem_layout_ticket_1124 -- fix .todo reference. 2010-07-29 08:29:27 -07:00
david-sarah
0ebe18cc73 test_upload.py: rename test_problem_layout_ticket1124 to test_problem_layout_ticket_1124 for consistency. 2010-07-29 07:22:50 -07:00
Zooko O'Whielacronx
28822dfb4e tests: remove debug print 2010-08-01 23:33:39 -07:00
Zooko O'Whielacronx
69c48ebde6 immutable: tidy-up some code by using a set instead of list to hold homeless_shares 2010-08-01 23:20:04 -07:00
Zooko O'Whielacronx
fd874d6669 setup: fix a couple instances of hard-coded 'allmydata-tahoe' in the scripts, tighten the tests (as suggested by David-Sarah) 2010-08-01 09:42:07 -07:00
Zooko O'Whielacronx
44edecb36c setup: replace hardcoded 'allmydata-tahoe' with allmydata.__appname__ 2010-08-01 09:05:17 -07:00
Zooko O'Whielacronx
0f94923f22 immutable: use PrefixingLogMixin to organize logging in Tahoe2PeerSelector and add more detailed messages about peer 2010-07-19 01:20:00 -07:00
Zooko O'Whielacronx
2ebe2c2ff3 benchmarking: update bench_dirnode to be correct and use the shiniest new pyutil.benchutil features concerning what units you measure in 2010-07-18 21:49:48 -07:00
Zooko O'Whielacronx
9b4f412e49 trivial: rename and add in-line doc to clarify "used_peers" => "upload_servers" 2010-07-18 21:47:44 -07:00
jacob.lyles
d3e6e15e23 web: don't use %d formatting on a potentially large negative float -- there is a bug in Python 2.5 in that case 
fixes #1055
 2010-07-30 15:05:50 -07:00
Zooko O'Whielacronx
1c62db0db8 docs: use current cap to Zooko's wiki page in example text 
fixes #1134
 2010-07-20 18:05:43 -07:00
david-sarah
94bec75dda __init__.py: silence DeprecationWarning about BaseException.message globally. fixes #1129 2010-07-19 18:19:39 -07:00
david-sarah
bdc6fd2f8b test_runner: test that 'tahoe --version' outputs no noise (e.g. DeprecationWarnings). 2010-07-19 18:13:45 -07:00
Zooko O'Whielacronx
c076c0b19b immutable: add test case of #1128, fix test case of #1118 2010-07-19 01:16:12 -07:00
Zooko O'Whielacronx
13b5e44fbc immutable: fix bug in which preexisting_shares and merged were shallowly referencing the same sets 
This bug had the effect of making uploads sometimes (rarely) appear to succeed when they had actually not distributed the shares well enough to achieve the desired servers-of-happiness level.
 2010-07-19 00:54:26 -07:00
david-sarah
461f946457 upload.py: fix #1118 by aborting newly-homeless buckets when reassignment runs. This makes a previously failing assert correct. This version refactors 'abort' into two methods, rather than using a default argument. 2010-07-18 21:46:55 -07:00
Zooko O'Whielacronx
1dbfcf753d immutable: extend the tests to check that the shares that got uploaded really do make a sufficiently Happy distribution 
This patch also renames some instances of "find_shares()" to "find_all_shares()" and other instances to "find_uri_shares()" as appropriate -- the conflation between those names confused me at first when writing these tests.
 2010-07-18 21:50:47 -07:00
Zooko O'Whielacronx
2e83f243c2 immutable: test for #1118 2010-07-18 15:15:37 -07:00
Zooko O'Whielacronx
0f46766a51 immutable: test for #1124 2010-07-18 15:29:07 -07:00
Zooko O'Whielacronx
8f201c75f0 trivial: fix unused import (sorry about that, pyflakes) 2010-07-18 14:51:33 -07:00
Zooko O'Whielacronx
16bb529339 tests, NEWS, CREDITS re: #1117 
Give Brian and Kevan promotions, move release date in NEWS to the 18th, commit Brian's test for #1117.
fixes #1117
 2010-07-18 13:32:25 -07:00
Kevan Carstensen
054c4553b5 test/test_upload.py: test to see that aborted buckets are ignored by the storage server 2010-07-15 17:10:46 -07:00
Kevan Carstensen
677f3b1f63 test/test_storage.py: test for the new remote_abort semantics. 2010-07-15 16:21:48 -07:00
Kevan Carstensen
6374f4307f storage/immutable.py: make remote_abort btell the storage server about aborted buckets. 2010-07-15 16:21:05 -07:00
Kevan Carstensen
5becaf0309 test/test_upload.py: changes to test plumbing for #1117 tests 
- Add a callRemoteOnly method to FakeBucketWriter.
    - Change the abort method in FakeBucketWriter to not return a
      RuntimeError.
 2010-07-15 16:18:20 -07:00
Kevan Carstensen
6d2c216b8f immutable/upload.py: abort buckets if peer selection fails 2010-07-15 16:17:14 -07:00
david-sarah
638a3b2af4 test_encodingutil: correct an error in the previous patch to StdlibUnicode.test_open_representable. 2010-07-18 08:14:20 -07:00
david-sarah
74c41ebb8b test_dirnode and test_web: don't use failUnlessReallyEqual in cases where the return type from simplejson.loads can vary between unicode and str. Use to_str when comparing URIs parsed from JSON. 2010-07-18 07:29:15 -07:00
david-sarah
9a4a864573 test_encodingutil: StdlibUnicode.test_open_representable no longer uses a mock. 2010-07-18 05:54:12 -07:00
Zooko O'Whielacronx
7cc98759bd docs: add comment clarifying #1051 2010-07-17 22:32:50 -07:00
david-sarah
d346e0853d Add tests of caps from the future that have non-ASCII characters in them (encoded as UTF-8). The changes to test_uri.py, test_client.py, and test_dirnode.py add tests of non-ASCII future caps in addition to the current tests. The changes to test_web.py just replace the tests of all-ASCII future caps with tests of non-ASCII future caps. We also change uses of failUnlessEqual to failUnlessReallyEqual, in order to catch cases where the type of a string is not as expected. 2010-07-11 13:02:52 -07:00
Zooko O'Whielacronx
3c4fe0706f M-x whitespace-cleanup 2010-07-17 20:27:39 -07:00
Zooko O'Whielacronx
6a7023efbe benchmarking: update bench_dirnode.py to reflect the new directory interfaces 2010-07-17 20:17:10 -07:00
david-sarah
bdb10553eb test_encodingutil: fix test_open_representable, which is only valid when run on a platform for which we know an unrepresentable filename. 2010-07-17 20:03:33 -07:00
francois
968edfda11 iputil.py: Add support for FreeBSD 7,8 and 9 
committed by: zooko@zooko.com
date: 2010-07-17
and I also patched: NEWS and CREDITS
 2010-07-17 19:28:32 -07:00
david-sarah
752897450d fileutil: docstrings for non-obvious usage restrictions on methods of EncryptedTemporaryFile. 2010-07-16 22:46:47 -07:00
david-sarah
05022dca36 Move EncryptedTemporaryFile from SFTP frontend to allmydata.util.fileutil, and make the FTP frontend also use it (fixing #1083). 2010-07-11 14:37:21 -07:00
david-sarah
a8161c915a Correct stringutils->encodingutil patch to be the newer version, rather than the old version that was committed in error. 2010-07-17 18:34:35 -07:00
david-sarah
0d79a4a7d1 test_cli.py: fix error that crept in when rebasing the patch for #1072. 2010-07-17 17:01:23 -07:00
david-sarah
c59f9b162f stringutils: add test for when sys.stdout has no encoding attribute (fixes #1099). 2010-07-16 21:58:16 -07:00
david-sarah
8090785716 CLI: add 'tahoe unlink' as an alias to 'tahoe rm', for forward-compatibility. 2010-07-17 15:04:11 -07:00
Zooko O'Whielacronx
6e8477114e minor code clean-up in dirnode.py 
Impose micro-POLA by passing only the writekey instead of the whole node object to {{{_encrypt_rw_uri()}}}. Remove DummyImmutableFileNode in nodemaker.py, which is obviated by this. Add micro-optimization by precomputing the netstring of the empty string and branching on whether the writekey is present or not outside of {{{_encrypt_rw_uri()}}}. Add doc about writekey to docstring.
fixes #967
 2010-07-13 23:02:55 -07:00
david-sarah
11077ea74d Rename stringutils to encodingutil, and drop listdir_unicode and open_unicode (since the Python stdlib functions work fine with Unicode paths). Also move some utility functions to fileutil. 2010-07-11 17:30:15 -07:00
david-sarah
fa0fd66e17 Allow URIs passed in the initial JSON for t=mkdir-with-children, t=mkdir-immutable to be Unicode. Also pass the name of each child into nodemaker.create_from_cap for error reporting. 2010-07-11 12:55:25 -07:00
david-sarah
e2ef2c2691 cli.py: make command descriptions consistently end with a full stop. 2010-07-13 18:45:38 -07:00
david-sarah
15ddab08ed SFTP: address some of the comments in zooko's review (#1106). 2010-07-11 19:55:37 -07:00
david-sarah
3b1b0147a8 test_storage.py: potential fix for failures when logging is enabled. 1970-07-12 21:05:46 -07:00
terrellrussell
d0381e679e upcase_since_on_welcome 2010-07-08 12:39:03 -07:00
freestorm77
35ec8f6ac2 server_version_on_welcome_page.dpatch.txt 
- The storage server version is 0 for all storage nodes in the Welcome Page
 2010-06-05 12:17:21 -07:00
freestorm77
496b91717a directory_html_top_banner.dpatch 
The div tag with the link "Return to Welcome page" on the directory.xhtml page is not correct
 2010-06-22 13:53:01 -07:00
freestorm77
eb63ba9a26 tahoe_css_toolbar.dpatch 
CSS modification to be correctly diplayed with Internet Explorer 8

The links on the top of page directory.xhtml are not diplayed in the same line as display with Firefox.
 2010-06-22 14:00:46 -07:00
freestorm77
97579af73c runnin_test_tahoe_css.dpatch 
Runnin test for changes in tahoe.css file
 2010-06-22 14:47:14 -07:00
freestorm77
0304918a77 runnin_test_directory_xhtml.dpatch 
Runnin test for diretory.xhtml top banner
 2010-06-22 13:14:03 -07:00
david-sarah
b8df25a8d1 stringutils.py: tolerate sys.stdout having no 'encoding' attribute. 2010-06-25 21:08:17 -07:00
david-sarah
88169328be SFTP: don't call .stopProducing on the producer registered with OverwriteableFileConsumer (which breaks with warner's new downloader). 2010-06-28 16:19:26 -07:00
Zooko O'Whielacronx
ab80719198 trivial: tiny update to in-line comment 
(I'm actually committing this patch as a test of the new eager-annotation-computation of trac-darcs.)
 2010-06-13 21:57:15 -07:00
david-sarah
6b2f99fa9a test_backupdb.py: skip test_unicode if we can't represent the test filenames. 2010-06-18 19:26:20 -07:00
david-sarah
a2caf6f4b2 test_web.py: correct a test that was missed in the change to not write ctime/mtime. 2010-06-18 19:17:18 -07:00
david-sarah
4712875193 dirnode.py: stop writing 'ctime' and 'mtime' fields. Includes documentation and test changes. 2010-06-18 16:01:19 -07:00
david-sarah
72e395d878 test_storage.py: print more information on test failures. 2010-06-16 20:46:23 -07:00
david-sarah
390fc78a9a test_cli.py: fix test failure in CLI.test_listdir_unicode_good due to filenames returned from listdir_unicode no longer being normalized. 2010-06-17 21:51:10 -07:00
david-sarah
178401eb4e tahoe backup: unicode tests. 2010-06-17 20:52:11 -07:00
david-sarah
1a0674bf37 CLI: allow Unicode patterns in exclude option to 'tahoe backup'. 2010-06-16 20:39:01 -07:00
david-sarah
a9fe3792de dirnodes: fix normalization hole where childnames in directories created by nodemaker.create_mutable/immutable_directory would not be normalized. Add a test that we normalize names coming out of a directory. 2010-06-17 17:02:49 -07:00
david-sarah
718870a796 dirnode.py: comments about normalization changes. 2010-06-16 21:14:11 -07:00
david-sarah
7e7644589a stringutils.py: remove unused import. 2010-06-16 20:44:40 -07:00
david-sarah
5ada31034b test_stringutils.py: take account of the output of listdir_unicode no longer being normalized. Also use Unicode escapes, not UTF-8. 2010-06-16 20:44:09 -07:00
david-sarah
025aede9e4 test_dirnode.py: partial tests for normalization changes. 2010-06-16 20:40:25 -07:00
david-sarah
c1412c8a81 SFTP: get 'ctime' attribute from 'tahoe:linkmotime'. 2010-06-16 20:37:44 -07:00
david-sarah
9f5488b2d1 stringutils.py: don't NFC-normalize the output of listdir_unicode. 2010-06-16 18:55:37 -07:00
david-sarah
e2c7ad1d88 stringutils.py: Add encoding argument to quote_output. Also work around a bug in locale.getpreferredencoding on older Pythons. 2010-06-15 21:20:12 -07:00
david-sarah
c8d99b77a3 Provisional patch to NFC-normalize filenames going in and out of Tahoe directories. 2010-06-15 20:14:50 -07:00
david-sarah
408b12a7e6 test_system.py: investigate failure in allmydata.test.test_system.SystemTest.test_upload_and_download_random_key due to bytes_sent not being an int 2010-06-15 17:16:48 -07:00
david-sarah
a3acb2b773 SFTP: remove a dubious use of 'pragma: no cover'. 2010-06-13 09:43:56 -07:00
david-sarah
11e88c340a SFTP: test that renaming onto a just-opened file fails. 2010-06-11 20:37:09 -07:00
david-sarah
b67f8b66c8 SFTP: further small improvements to test coverage. Also ensure that after a test failure, later tests don't fail spuriously due to the checks for heisenfile leaks. 2010-06-11 20:07:37 -07:00
david-sarah
b4e074d250 SFTP: further improve test coverage (paths containing '.', bad data for posix-rename extension, and error in test of openShell). 2010-06-11 14:31:42 -07:00
david-sarah
52f87904ed SFTP: improve test coverage for no-write on mutable files, and check for heisenfile table leaks in all relevant tests. Delete test_memory_leak since it is now redundant. 2010-06-11 13:57:52 -07:00
david-sarah
546c3d2ed4 SFTP: add test for extension of file opened with FXF_APPEND. 2010-06-10 11:26:47 -07:00
Zooko O'Whielacronx
f19c240ca8 tests: bump up the timeout on this iputil test from 2s to 4s 2010-06-09 07:30:17 -07:00
david-sarah
3c44389440 SFTP: fix most significant memory leak described in #1045 (due to a file being added to all_heisenfiles under more than one direntry when renamed). 2010-06-09 01:00:03 -07:00
david-sarah
5786874d38 test_stringutils.py: Fix test failure on CentOS builder, possibly Python 2.4.3-related. 2010-06-08 23:50:56 -07:00
david-sarah
ffd1c5aca5 Fix for Unicode-related test failures on Zooko's OS X 10.6 machine. 2010-06-08 22:54:48 -07:00
david-sarah
80a89d3931 stringutils.py, sftpd.py: Portability fixes for Python <= 2.5. 2010-06-08 18:33:02 -07:00
david-sarah
32d9deace3 Back out Windows-specific Unicode argument support for v1.7. 2010-06-08 17:08:03 -07:00
david-sarah
4b5927fdb8 SFTP: suppress NoSuchChildError if heisenfile attributes have been updated in setAttrs, in the case where the parent is available. 2010-06-07 23:37:53 -07:00
david-sarah
195b4afa8f SFTP: ignore permissions when opening a file (needed for sshfs interoperability). 2010-06-07 22:57:00 -07:00
david-sarah
a67e745b26 test_web.py: fix pyflakes warnings introduced by byterange patch. 2010-06-07 21:20:12 -07:00
Jeremy Fitzhardinge
63b28d707b Improve HTTP/1.1 byterange handling 
Fix parsing of a Range: header to support:
 - multiple ranges (parsed, but not returned)
 - suffix byte ranges ("-2139")
 - correct handling of incorrectly formatted range headers
   (correct behaviour is to ignore the header and return the full
    file)
 - return appropriate error for ranges outside the file

Multiple ranges are parsed, but only the first range is returned.
Returning multiple ranges requires using the multipart/byterange
content type.
 2010-03-09 20:59:13 -07:00
Zooko O'Whielacronx
d13784b649 tests: bump up the timeout on these tests; MM's buildslave is sometimes extremely slow on tests, but it will complete them if given enough time. MM is working on making that buildslave more predictable in how long it takes to run tests. 2010-06-07 20:37:54 -07:00
david-sarah
3c883e6e44 test_cli.py: remove invalid 'test_listdir_unicode_bad' test. 2010-06-07 11:37:30 -07:00
david-sarah
71f8757bd1 check_memory.py: adapt to servers-of-happiness changes. 2010-06-07 18:35:28 -07:00
david-sarah
7092de1b6f Remove the 'tahoe debug consolidate' subcommand. 2010-06-07 11:37:57 -07:00
david-sarah
731e3d68df common_http.py, tahoe_cp.py: Fix an error in calling the superclass constructor in HTTPError and MissingSourceError (introduced by the Unicode fixes). 2010-06-07 10:47:14 -07:00
Zooko O'Whielacronx
faade8b20a tests: drastically increase timeout of this very time-consuming test in honor of François's ARM box 2010-06-07 04:59:29 -07:00
david-sarah
8b014372b1 test_stringutils.py: trivial error in exception message for skipped test. 2010-06-06 23:14:55 -07:00
david-sarah
529c9f673a More Unicode test fixes. 2010-06-06 22:33:58 -07:00
david-sarah
65b6f4e3ce Unicode fixes for platforms with non-native-Unicode filesystems. 2010-06-06 21:32:38 -07:00
david-sarah
80252debcd Unicode fixes. 2010-06-06 18:02:15 -07:00
Zooko O'Whielacronx
5bcca5151e unicode tests: fix missing import 2010-06-04 07:26:30 -07:00
Zooko O'Whielacronx
db8a6f3aa6 unicode: make test_cli test a non-ascii argument, and make the fallback term encoding be locale.getpreferredencoding() 2010-06-04 07:12:51 -07:00
Zooko O'Whielacronx
442008a690 unicode: always decode json manifest as utf-8 then encode for stdout 
pyflakes pointed out that the exception handler fallback called an un-imported function, showing that the fallback wasn't being exercised.
I'm not 100% sure that this patch is right and would appreciate François or someone reviewing it.
 2010-06-04 01:48:40 -07:00
Zooko O'Whielacronx
952f3bf3ef fix flakes 2010-06-04 00:58:45 -07:00
Zooko O'Whielacronx
08058c1073 fix syntax of assertion handling that isn't portable to older versions of Python 2010-06-04 00:58:05 -07:00
Francois Deppierraz
0eb4d83937 test_stringutils.py: Skip test test_listdir_unicode_good if filesystem supports only ASCII filenames 2010-05-21 09:08:39 -07:00
Francois Deppierraz
01da7f06e6 test_stringutils.py: Skip test_listdir_unicode on mocked platform which cannot store non-ASCII filenames 2010-05-21 09:05:59 -07:00
Francois Deppierraz
58e9df9f5a test_stringutils.py: Add a test class for OpenBSD 4.1 with LANG=C 2010-05-21 07:00:53 -07:00
Francois Deppierraz
1aeff62161 test_stringutils.py: Mock the open() call in test_open_unicode 
This test ensure that open(a_unicode_string) is used on Unicode platforms
(Windows or MacOS X) and that open(a_correctly_encoded_bytestring) on other
platforms such as Unix.
 2010-05-21 06:58:17 -07:00
Francois Deppierraz
eecaaca6b7 test_stringutils.py: Fix a trivial Python 2.4 syntax incompatibility 2010-05-21 02:33:45 -07:00
Francois Deppierraz
6c70fb7b3c test_cli.py: Fix tests when sys.stdout.encoding=None and refactor this code into functions 2010-05-20 01:44:47 -07:00
Francois Deppierraz
496802420e Fix handling of correctly encoded unicode filenames (#534) 
Tahoe CLI commands working on local files, for instance 'tahoe cp' or 'tahoe
backup', have been improved to correctly handle filenames containing non-ASCII
characters.
  
In the case where Tahoe encounters a filename which cannot be decoded using the
system encoding, an error will be returned and the operation will fail.  Under
Linux, this typically happens when the filesystem contains filenames encoded
with another encoding, for instance latin1, than the system locale, for
instance UTF-8.  In such case, you'll need to fix your system with tools such
as 'convmv' before using Tahoe CLI.
  
All CLI commands have been improved to support non-ASCII parameters such as
filenames and aliases on all supported Operating Systems except Windows as of
now.
 2010-05-19 17:43:56 -07:00
Francois Deppierraz
d0ed14e1bb stringutils.py: Unicode helper functions + associated tests 
This file contains a bunch of helper functions which converts
unicode string from and to argv, filenames and stdout.
 2010-05-19 17:41:05 -07:00
david-sarah
a2cff856ff Resolve merge conflict for sftpd.py 2010-06-03 11:25:37 -07:00
david-sarah
2dcb62de90 SFTP: possible fix for metadata times being shown as the epoch. 2010-06-02 16:45:14 -07:00
david-sarah
c9984c40e9 SFTP: further improvements to test coverage. 2010-06-02 16:44:22 -07:00
david-sarah
2a791b0d05 SFTP: improve test coverage. Also make creating a directory fail when permissions are read-only (rather than ignoring the permissions). 2010-06-01 21:19:34 -07:00
david-sarah
29a06457d2 dirnode.py: fix a bug in the no-write change for Adder, and improve test coverage. Add a 'metadata' argument to create_subdirectory, with documentation. Also update some comments in test_dirnode.py made stale by the ctime/mtime change. 2010-06-01 20:26:41 -07:00
david-sarah
db394671e9 SFTP: fix a bug that caused the temporary files underlying EncryptedTemporaryFiles not to be closed. 2010-05-31 22:53:10 -07:00
david-sarah
de95140b7b SFTP: changes for #1063 ('no-write' field) including comment:1 (clearing owner write permission diminishes to a read cap). Includes documentation changes, but not tests for the new behaviour. 2010-05-31 22:11:39 -07:00
david-sarah
9faa02deb5 SFTP: the same bug as in _sync_heisenfiles also occurred in two other places. 2010-05-29 23:01:27 -07:00
david-sarah
cccdd32806 SFTP: another try at fixing the _sync_heisenfiles bug. 2010-05-29 22:52:54 -07:00
david-sarah
51b0d8bde3 SFTP: fix silly bug in _sync_heisenfiles ('f is not ignore' vs 'not (f is ignore)'). 2010-05-29 22:38:07 -07:00
david-sarah
b9d3cf12b8 SFTP: log when a sync completes. 2010-05-29 22:18:40 -07:00
david-sarah
c4a8af4d27 SFTP: fix bug in previous logging patch. 2010-05-29 22:00:00 -07:00
david-sarah
3395a7e154 SFTP: more logging to track down OpenOffice hang. 2010-05-29 21:08:09 -07:00
david-sarah
4be24a89df SFTP: avoid blocking close on a heisenfile that has been abandoned or never changed. Also, improve the logging to help track down a case where OpenOffice hangs on opening a file with FXF_READ|FXF_WRITE. 2010-05-29 19:55:44 -07:00
david-sarah
8ed28a95e7 Move suppression of DeprecationWarning about BaseException.message from sftpd.py to main __init__.py. Also, remove the global suppression of the 'integer argument expected, got float' warning, which turned out to be a bug. 2010-05-28 22:05:37 -07:00
david-sarah
e867985539 SFTP: cater to clients that assume a file is created as soon as they have made an open request; also, fix some race conditions associated with closing a file at about the same time as renaming or removing it. 2010-05-28 21:52:53 -07:00
david-sarah
0b888f8201 SFTP: 'sync' any open files at a direntry before opening any new file at that direntry. This works around the sshfs misbehaviour of returning success to clients immediately on close. 2010-05-25 16:02:57 -07:00
david-sarah
a143b1297b SFTP: handle removing a file while it is open. Also some simplifications of the logout handling. 2010-05-25 11:42:10 -07:00
david-sarah
e81ce2bbd2 SFTP: a posix-rename response should actually return an FXP_STATUS reply, not an FXP_EXTENDED_REPLY as Twisted Conch assumes. Work around this by raising an SFTPError with code FX_OK. 2010-05-24 20:33:23 -07:00
david-sarah
8f86490d48 SFTP: fix problem with posix-rename code returning a Deferred for the renamed filenode, not for the result of the request (an empty string). 2010-05-24 19:02:09 -07:00
david-sarah
210578da63 SFTP: fix time handling to make sure floats are not passed into twisted.conch, and to print times in the future less ambiguously in directory listings. 2010-05-24 16:04:12 -07:00
david-sarah
549dccd608 SFTP: name of the POSIX rename extension should be 'posix-rename@openssh.com', not 'extposix-rename@openssh.com'. 2010-05-23 19:11:56 -07:00
david-sarah
6e114e44b5 SFTP: avoid race condition where .write could be called on an OverwriteableFileConsumer after it had been closed. 2010-05-23 16:38:30 -07:00
david-sarah
c6b414c815 SFTP: log tracebacks for RAISEd exceptions. 2010-05-23 15:15:35 -07:00
david-sarah
725daee788 SFTP: more logging to investigate behaviour of getAttrs(path). 2010-05-23 13:42:36 -07:00
david-sarah
75357fd2d5 SFTP: fix pyflakes warnings; drop 'noisy' versions of eventually_callback and eventually_errback; robustify conversion of exception messages to UTF-8. 2010-05-23 07:09:05 -07:00
david-sarah
38964fb35a SFTP: fixes and test cases for renaming of open files. 2010-05-22 20:25:49 -07:00
david-sarah
42ecd77f4e SFTP: Increase test_sftp timeout to cater for francois' ARM buildslave. 2010-05-22 12:16:39 -07:00
david-sarah
5974773969 SFTP: Fix error in support for getAttrs on an open file, to index open files by directory entry rather than path. Extend that support to renaming open files. Also, implement the extposix-rename@openssh.org extension, and some other minor refactoring. 2010-05-21 20:58:36 -07:00
david-sarah
5545a678bf SFTP tests: fix test_openDirectory_and_attrs that was failing in timezones west of UTC. 2010-05-20 11:10:27 -07:00
david-sarah
ce3872d10e SFTP: allow getAttrs to succeed on a file that has been opened for creation but not yet uploaded or linked (part of #1050). 2010-05-19 20:56:13 -07:00
david-sarah
7f9e351bce SFTP: improve logging so that results of requests are (usually) logged. 2010-05-19 17:36:52 -07:00
david-sarah
a62733ed08 SFTP: add tests for more combinations of open flags. 2010-05-18 22:39:33 -07:00
david-sarah
f0945526ce SFTP: allow FXF_WRITE | FXF_TRUNC (#1050). 2010-05-18 21:32:40 -07:00
david-sarah
bde27cab57 SFTP: remove another case where we were logging data. 2010-05-18 18:27:13 -07:00
david-sarah
d8b096a054 SFTP: avoid logging all data passed to callbacks. 2010-05-18 17:06:51 -07:00
david-sarah
819eaa74c1 SFTP: fixes related to reporting of permissions (needed for sshfs). 2010-05-17 22:45:21 -07:00
david-sarah
8d118e28f1 SFTP: change error code returned for ExistingChildError to FX_FAILURE (fixes gvfs with some picky programs such as gedit). 2010-05-17 17:42:05 -07:00
david-sarah
57699fd1eb SFTP: fixed bugs that caused hangs during write (#1037). 2010-05-16 21:42:28 -07:00
david-sarah
5f9c10901b SFTP: work around a probable bug in twisted.conch.ssh.session:loseConnection(). Also some minor error handling cleanups. 2010-05-16 18:26:06 -07:00
david-sarah
6a2b0229f0 SFTP: Support statvfs extensions, avoid logging actual data, and decline shell sessions politely. 2010-05-16 08:43:47 -07:00
david-sarah
31994aaf74 SFTP: fix error in SFTPUserHandler arguments introduced by execCommand patch. 2010-05-15 18:40:45 -07:00
david-sarah
8028bf01bc SFTP: implement execCommand to interoperate with clients that issue a 'df -P -k /' command. Also eliminate use of Zope adaptation. 2010-05-15 18:27:54 -07:00
david-sarah
6ba6c97a7a sftpd.py: 'log.OPERATIONAL' should be just 'OPERATIONAL'. 2010-05-15 08:55:33 -07:00
david-sarah
5ab4a376b7 Attempt to fix #1040 by making SFTPUser implement ISession. 2010-05-14 17:57:19 -07:00
david-sarah
7409d1e019 Eliminate Windows newlines from sftpd.py. 2010-05-14 17:56:56 -07:00
david-sarah
14d0721246 Update SFTP implementation and tests: fix #1038 and switch to foolscap logging; also some code reorganization. 2010-05-13 21:31:13 -07:00
david-sarah
5fdd63d0ad Tests for new SFTP implementation 2010-05-11 23:05:52 -07:00
david-sarah
5248c808ac New SFTP implementation: mutable files, read/write support, streaming download, Unicode filenames, and more 2010-05-11 22:54:07 -07:00
david-sarah
9214dbda50 Add must_exist, must_be_directory, and must_be_file arguments to DirectoryNode.delete. This will be used to fixes a minor condition in the SFTP frontend. 2010-05-27 12:45:29 -07:00
david-sarah
4556702044 Fix test failures in test_web caused by changes to web page titles in #1062. Also, change a 'target' field to '_blank' instead of 'blank' in welcome.xhtml. 2010-06-03 16:21:05 -07:00
david-sarah
4f58355a0e dirnode.py: Fix bug that caused 'tahoe' fields, 'ctime' and 'mtime' not to be updated when new metadata is present. 2010-06-01 18:46:44 -07:00
david-sarah
53f7d2c7fe dirnode.py: Fix #1034 (MetadataSetter does not enforce restriction on setting 'tahoe' subkeys), and expose the metadata updater for use by SFTP. Also, support diminishing a child cap to read-only if 'no-write' is set in the metadata. 2010-05-31 21:54:28 -07:00
david-sarah
027e7701bd Change doc comments in interfaces.py to take into account unknown nodes. 2010-05-28 10:19:22 -07:00
david-sarah
6d91b1e00e Trivial whitespace changes. 2010-05-27 12:41:14 -07:00
david-sarah
aac94a8d89 Suppress 'integer argument expected, got float' DeprecationWarning everywhere 2010-05-23 15:11:57 -07:00
david-sarah
e706f043f0 Change shouldFail to avoid Unicode errors when converting Failure to str 2010-05-11 23:07:54 -07:00
david-sarah
b8641f8f9b allmydata.org -> tahoe-lafs.org in __init__.py 2010-06-02 23:35:30 -07:00
david-sarah
90e9fe28af Resolve conflict in patch to change imports to absolute. 2010-06-02 22:46:08 -07:00
freestorm77
59a77f82d7 title_rename_xhtml.dpatch.txt 
- Renamed xhtml Title from "Allmydata - Tahoe" to "Tahoe-LAFS"
- Renamed Tahoe to Tahoe-LAFS in page content
- Changed Tahoe-LAFS home page link to http://tahoe-lafs.org (added target="blank")
- Deleted commented css script in info.xhtml
 2010-05-29 10:25:42 -07:00
Zooko O'Whielacronx
9e2da05837 tests: refactor test_web.py to have less duplication of literal caps-from-the-future 
This is a prelude to a patch which will add tests of caps from the future which have non-ascii chars in them.
 2010-05-18 22:51:46 -07:00
david-sarah
a80f19a084 Correct harmless indentation errors found by pylint 2010-02-25 23:21:51 -07:00
david-sarah
e76092e16c Change relative imports to absolute 2010-02-26 01:14:33 -07:00
david-sarah
e880fc2ce4 Suppress deprecation warning for twisted.web.error.NoResource when using Twisted >= 9.0.0. 2010-05-16 13:56:25 -07:00
Zooko O'Whielacronx
d4f6d4092c test: show stdout and stderr in case of non-zero exit code from "tahoe" command 2010-05-24 00:33:48 -07:00
Zooko O'Whielacronx
294c0a6279 tests: fix pyflakes warnings in bench_dirnode.py 2010-05-21 13:25:11 -07:00
Zooko O'Whielacronx
35efdf04ca tests: don't let bench_dirnode.py do stuff and have side-effects at import time (unless __name__ == '__main__') 2010-05-21 05:20:52 -07:00
Zooko O'Whielacronx
a48979516e tests: increase timeout to give François's ARM buildslave a chance to complete the tests 2010-05-20 06:45:26 -07:00
Kevan Carstensen
d0706d27ea Hush pyflakes warnings 2010-05-15 11:43:44 -07:00
Kevan Carstensen
9179dd099a Improve code coverage of the Tahoe2PeerSelector tests. 2010-05-14 20:29:13 -07:00
Kevan Carstensen
55a2849f38 Remove a comment that no longer makes sense. 2010-05-14 13:35:16 -07:00
Kevan Carstensen
e225f573b9 Fix up the behavior of #778, per reviewers' comments 
- Make some important utility functions clearer and more thoroughly 
    documented.
  - Assert in upload.servers_of_happiness that the buckets attributes
    of PeerTrackers passed to it are mutually disjoint.
  - Get rid of some silly non-Pythonisms that I didn't see when I first
    wrote these patches.
  - Make sure that should_add_server returns true when queried about a 
    shnum that it doesn't know about yet.
  - Change Tahoe2PeerSelector.preexisting_shares to map a shareid to a set
    of peerids, alter dependencies to deal with that.
  - Remove upload.should_add_servers, because it is no longer necessary
  - Move upload.shares_of_happiness and upload.shares_by_server to a utility
    file.
  - Change some points in Tahoe2PeerSelector.
  - Compute servers_of_happiness using a bipartite matching algorithm that 
    we know is optimal instead of an ad-hoc greedy algorithm that isn't.
  - Change servers_of_happiness to just take a sharemap as an argument,
    change its callers to merge existing_shares and used_peers before 
    calling it.
  - Change an error message in the encoder to be more appropriate for 
    servers of happiness.
  - Clarify the wording of an error message in immutable/upload.py
  - Refactor a happiness failure message to happinessutil.py, and make
    immutable/upload.py and immutable/encode.py use it.
  - Move the word "only" as far to the right as possible in failure 
    messages.
  - Use a better definition of progress during peer selection.
  - Do read-only peer share detection queries in parallel, not sequentially.
  - Clean up logging semantics; print the query statistics whenever an
    upload is unsuccessful, not just in one case.
 2010-05-13 17:49:17 -07:00
Kevan Carstensen
9bc71d3da0 Alter the error message when an upload fails, per some comments in #778. 
When I first implemented #778, I just altered the error messages to refer to
servers where they referred to shares. The resulting error messages weren't
very good. These are a bit better.
 2009-12-30 15:03:44 -07:00
Kevan Carstensen
8bcc771e26 Change "UploadHappinessError" to "UploadUnhappinessError" 2009-12-04 22:30:37 -07:00
Kevan Carstensen
68fb556e93 Alter the error message returned when peer selection fails 
The Tahoe2PeerSelector returned either NoSharesError or NotEnoughSharesError
for a variety of error conditions that weren't informatively described by them.
This patch creates a new error, UploadHappinessError, replaces uses of 
NoSharesError and NotEnoughSharesError with it, and alters the error message
raised with the errors to be more in line with the new servers_of_happiness
behavior. See ticket #834 for more information.
 2009-11-22 18:24:05 -07:00
Kevan Carstensen
320582be5a Eliminate overcounting iof servers_of_happiness in Tahoe2PeerSelector; also reorganize some things. 2009-11-17 19:45:42 -07:00
Kevan Carstensen
4e29060847 Change stray "shares_of_happiness" to "servers_of_happiness" 2009-11-16 15:24:59 -07:00
Kevan Carstensen
a816de3f23 Alter Tahoe2PeerSelector to make sure that it recognizes existing shares on readonly servers, fixing an issue in #778 2009-11-16 13:28:05 -07:00
Kevan Carstensen
8c71df53f9 Alter 'immutable/encode.py' and 'immutable/upload.py' to use servers_of_happiness instead of shares_of_happiness. 2009-11-04 05:12:22 -07:00
Kevan Carstensen
b2d8a7cec2 Alter the signature of set_shareholders in IEncoder to add a 'servermap' parameter, which gives IEncoders enough information to perform a sane check for servers_of_happiness. 2009-11-03 21:32:41 -07:00
Kevan Carstensen
89a8da9dd2 Alter CiphertextDownloader to work with servers_of_happiness 2009-09-23 21:19:32 -07:00
Kevan Carstensen
5683112a02 Revisions of the #778 tests, per reviewers' comments 
- Fix comments and confusing naming.
- Add tests for the new error messages suggested by David-Sarah
  and Zooko.
- Alter existing tests for new error messages.
- Make sure that the tests continue to work with the trunk.
- Add a test for a mutual disjointedness assertion that I added to
  upload.servers_of_happiness.
- Fix the comments to correctly reflect read-onlyness
- Add a test for an edge case in should_add_server
- Add an assertion to make sure that share redistribution works as it 
  should
- Alter tests to work with revised servers_of_happiness semantics
- Remove tests for should_add_server, since that function no longer exists.
- Alter tests to know about merge_peers, and to use it before calling 
  servers_of_happiness.
- Add tests for merge_peers.
- Add Zooko's puzzles to the tests.
- Edit encoding tests to expect the new kind of failure message.
- Edit tests to expect error messages with the word "only" moved as far
  to the right as possible.
- Extended and cleaned up some helper functions.
- Changed some tests to call more appropriate helper functions.
- Added a test for the failing redistribution algorithm
- Added a test for the progress message
- Added a test for the upper bound on readonly peer share discovery.
 2010-05-13 18:25:42 -07:00
Kevan Carstensen
d360ee0d2e Alter various unit tests to work with the new happy behavior 2010-01-07 12:13:25 -07:00
Kevan Carstensen
953c9555ca Replace "UploadHappinessError" with "UploadUnhappinessError" in tests. 2009-12-04 22:34:53 -07:00
Kevan Carstensen
c0f3dbb919 Add tests for the behavior described in #834. 2009-11-22 19:20:08 -07:00
Kevan Carstensen
c3b11dedea Re-work 'test_upload.py' to be more readable; add more tests for #778 2009-11-16 13:23:34 -07:00
Kevan Carstensen
9590690450 Test Tahoe2PeerSelector to make sure that it recognizeses existing shares on readonly servers 2009-11-08 18:37:35 -07:00
Kevan Carstensen
324a72e4a3 Add more tests for comment:53 in ticket #778 2009-11-04 05:28:49 -07:00
Kevan Carstensen
697f1cfae8 Add a test for upload.shares_by_server 2009-11-04 05:13:24 -07:00
Kevan Carstensen
0d8e70bfe5 Minor tweak to an existing test -- make the first server read-write, instead of read-only 2009-11-03 21:42:32 -07:00
Kevan Carstensen
5ad29d4e8b Alter tests to use the new form of set_shareholders 2009-11-03 21:36:02 -07:00
"Kevan Carstensen"
ee9690b357 Refactor some behavior into a mixin, and add tests for the behavior described in #778 2009-10-30 02:19:08 -07:00
Kevan Carstensen
362f204075 Alter NoNetworkGrid to allow the creation of readonly servers for testing purposes. 2009-10-17 18:30:13 -07:00
"Kevan Carstensen"
5fe125ed74 Alter wording in 'interfaces.py' to be correct wrt #778 2009-12-04 21:40:05 -07:00
Zooko O'Whielacronx
6037df55b3 web: add a simple test that you can load directory.xhtml 2010-05-09 23:37:29 -07:00
Francois Deppierraz
dd95e8e71b CLI: Support for https url in option --node-url 
This patch modifies the regular expression used for verifying of '--node-url'
parameter.  Support for accessing a Tahoe gateway over HTTPS was already
present, thanks to Python's urllib.
 2010-04-30 11:56:09 -07:00
Brian Warner
e2031aff0f backupdb.did_create_directory: use REPLACE INTO, not INSERT INTO + ignore error 
This handles the case where we upload a new tahoe directory for a
previously-processed local directory, possibly creating a new dircap (if the
metadata had changed). Now we replace the old dirhash->dircap record. The
previous behavior left the old record in place (with the old dircap and
timestamps), so we'd never stop creating new directories and never converge
on a null backup.
 2010-04-27 22:08:03 -07:00
Brian Warner
f6f8256b35 "tahoe webopen": add --info flag, to get ?t=info 
Also fix some trailing whitespace.
 2010-04-24 16:30:03 -07:00
Zooko O'Whielacronx
80ae6f2706 setup: fix error in licensing declaration from Shawn Willden, as noted by the FSF compliance division 2010-03-09 08:37:36 -08:00
Zooko O'Whielacronx
34a50b857a docs: a few small edits to try to guide newcomers through the docs 
These edits were suggested by my watching over Jake Appelbaum's shoulder as he completely ignored/skipped/missed install.html and also as he decided that debian.txt wouldn't help him with basic installation. Then I threw in a few docs edits that have been sitting around in my sandbox asking to be committed for months.
 2010-03-03 15:19:02 -08:00
david-sarah
40edf8f419 Change code that gives a base32 SI or an empty string to be more straightforward. (#948) 2010-02-26 22:55:51 -08:00
david-sarah
a2ed17f2a0 Additional test for DIR2-LIT directories in test_web.py, fixed version (#948) 2010-02-24 20:18:24 -08:00
david-sarah
d7b50a3b86 Additional fixes for DIR2-LIT More Info page and deep-check/manifest operations (#948) 2010-02-24 00:02:20 -08:00
david-sarah
e6aee33bb7 directories: add DIR2-LIT directories to test_deepcheck.py (#948) 2010-02-23 23:54:33 -08:00
david-sarah
187d837c1d dirnode: add tests of literal dirnodes (current and fix for #948) 2010-02-23 20:33:45 -08:00
david-sarah
fec9185f2a Additional fix for abbrev_si, with test 2010-02-21 19:36:52 -08:00
david-sarah
d29ec184a6 Additions to test_web.py for #948 2010-02-21 18:53:52 -08:00
david-sarah
973f0afdd3 Change direct accesses to an_uri.storage_index to calls to .get_storage_index() (fixes #948) 2010-02-21 18:45:04 -08:00
david-sarah
6a7feea455 Tweak to 'tahoe ls --help' output (#837) 2010-02-23 19:02:31 -08:00
david-sarah
9741b9655f Test behaviour of 'tahoe ls' for unknown objects (#837) 2010-02-23 18:59:13 -08:00
david-sarah
03134eedb5 Improve behaviour of 'tahoe ls' for unknown objects, addressing kevan's comments 2010-02-19 22:13:13 -08:00
Brian Warner
1de4d2c594 test_util.py: improve coverage of util.time_format 2010-02-23 23:46:37 -05:00
Kevan Carstensen
81ad52d6eb Change OphandleTable to use a deterministic clock, so we can test it 
To test the changes for #577, we need a deterministic way to simulate
the passage of long periods of time. twisted.internet.task.Clock seems,
from my Googling, to be the way to go for this functionality. I changed
a few things so that OphandleTable would use twisted.internet.task.Clock
when testing:

  * WebishServer.__init___ now takes an optional 'clock' parameter,
  * which it passes to the root.Root instance it creates.
  * root.Root.__init__ now takes an optional 'clock' parameter, which it
    passes to the OphandleTable.__init__ method.
  * OphandleTable.__init__ now takes an optional 'clock' parameter. If
    it is provided, and it isn't None, its callLater method will be used
    to schedule ophandle expirations (as opposed to using
    reactor.callLater, which is what OphandleTable does normally).
  * The WebMixin object in test_web.py now sets a self.clock parameter,
    which is a twisted.internet.task.Clock that it feeds to the 
    WebishServer it creates. 

Tests using the WebMixin can control the passage of time in
OphandleTable by accessing self.clock.
 2010-02-20 13:07:13 -08:00
Kevan Carstensen
bcdc78bd44 Add tests for the ophandle expiration behavior in #577 2010-02-20 17:04:55 -08:00
Kevan Carstensen
e9b59a4949 Increase ophandle expiration times, per #577 2010-02-20 17:05:12 -08:00
david-sarah
85a50feeaa More cleanups to test_cli using new utilities for reading and writing files. 2010-02-05 17:38:55 -08:00
david-sarah
c984a09fe7 Fix race conditions and missing callback in allmydata.test.test_cli.Cp.test_copy_using_filecap, add utilities for one-liner reading and writing of files, and fix cases in test_cli where files were not being closed after writing. 2010-02-05 17:37:27 -08:00
Kevan Carstensen
c00a62a2e7 Add tests for #939 2010-02-11 22:21:37 -08:00
Kevan Carstensen
63c8c65747 Alter CLI utilities to handle nonexistent aliases better 2010-02-10 18:43:18 -08:00
Brian Warner
4040b1000b web/storage.py: display total-seen on the last-complete-cycle line. For #940. 2010-02-07 16:20:10 -08:00
Brian Warner
880f824103 code coverage: replace figleaf with coverage.py, should work on py2.6 now. 
It still lacks the right HTML report (the builtin report is very pretty, but
lacks the "lines uncovered" numbers that I want), and the half-finished
delta-from-last-run measurements.
 2010-02-03 08:54:21 -08:00
Zooko O'Whielacronx
3e4342ecb3 immutable: downloader accepts notifications of buckets even if those notifications arrive after he has begun downloading shares. 
This can be useful if one of the ones that he has already begun downloading fails. See #287 for discussion. This fixes part of #287 which part was a regression caused by #928, namely this fixes fail-over in case a share is corrupted (or the server returns an error or disconnects). This does not fix the related issue mentioned in #287 if a server hangs and doesn't reply to requests for blocks.
 2010-01-31 22:16:10 -08:00
Zooko O'Whielacronx
e4e2599017 tests: don't require tahoe to run with no noise if we are using an old twisted that emits DeprecationWarnings 2010-01-31 21:23:23 -08:00
david-sarah
100548257a Use if instead of assert to check for twisted ftp patch 2010-01-26 17:55:29 -08:00
Zooko O'Whielacronx
7d32fafc44 tests: stop being surprised that Nevow no longer prints out warnings when it tries to find its static files 
Unless we are using a sufficiently new version of Nevow, in which case if it prints out warnings then this is a hard test failure. :-)
 2010-01-31 20:11:44 -08:00
david-sarah
6215ebd95e cli: suppress DeprecationWarnings emitted from importing nevow and twisted. Fixes #859 2010-01-31 16:44:29 -08:00
david-sarah
ea3954372a Improvements to test_unknownnode to cover invalid cap URIs with known prefixes 2010-01-29 22:39:08 -08:00
david-sarah
37a242e01a Improvements to test_hung_server, and fix for status updates in download.py 2010-01-29 22:43:03 -08:00
Zooko O'Whielacronx
d62428c1e6 immutable: fix bug in tests, change line-endings to unix style, add comment 2010-01-29 10:42:37 -08:00
david-sarah
baa11a0ad4 New tests for #928 2010-01-29 04:38:45 -08:00
Zooko O'Whielacronx
2bd9dfa5bd immutable: download from the first servers which provide at least K buckets instead of waiting for all servers to reply 
This should put an end to the phenomenon I've been seeing that a single hung server can cause all downloads on a grid to hang.  Also it should speed up all downloads by (a) not-waiting for responses to queries that it doesn't need, and (b) downloading shares from the servers which answered the initial query the fastest.
Also, do not count how many buckets you've gotten when deciding whether the download has enough shares or not -- instead count how many buckets to *unique* shares that you've gotten.  This appears to improve a slightly weird behavior in the current download code in which receiving >= K different buckets all to the same sharenumber would make it think it had enough to download the file when in fact it hadn't.
This patch needs tests before it is actually ready for trunk.
 2010-01-27 15:34:17 -08:00
david-sarah
14280b009c Eliminate 'foo if test else bar' syntax that isn't supported by Python 2.4 2010-01-28 19:52:10 -08:00
david-sarah
3e35959e9b Add mutable field to t=json output for unknown nodes, when mutability is known 2010-01-28 19:14:24 -08:00
david-sarah
4560e021a9 Show -IMM and -RO suffixes for types of immutable and read-only unknown nodes in directory listings 2010-01-28 14:08:00 -08:00
david-sarah
1bb627405f Fix inaccurate comment in test_mutant_dirnodes_are_omitted 2010-01-28 12:24:56 -08:00
david-sarah
b9eda4de6a Address comments by Kevan on 833 and add test for stripping spaces 2010-01-27 15:06:42 -08:00
david-sarah
56c00cb381 Miscellaneous documentation, test, and code formatting tweaks. 2010-01-26 23:03:09 -08:00
david-sarah
6057bc02cc Prevent mutable objects from being retrieved from an immutable directory, and associated forward-compatibility improvements. 2010-01-26 22:44:30 -08:00
Brian Warner
3880486f91 test_runner: cleanup, refactor common code into a non-executable method 
Having both test_node() and test_client() (one of which calls the other) felt
confusing to me, so I changed it to have test_node(), test_client(), and a
common do_create() helper method.
 2010-01-27 14:40:40 -08:00
Brian Warner
5045a8721a scripts/runner.py: simplify David-Sarah's clever grouped-commands usage trick 2010-01-27 14:37:58 -08:00
Brian Warner
e769bbb6dd tahoe backup: skip all symlinks, with warning. Fixes #850, addresses #641. 2010-01-27 14:35:17 -08:00
Brian Warner
52185053c3 "tahoe backup": fix --exclude-vcs docs to include Git 2010-01-27 12:10:44 -08:00
david-sarah
5c5a6fe610 Patch to accept t=set-children as well as t=set_children 2010-01-23 19:00:20 -08:00
Zooko O'Whielacronx
4a4305e7e6 ftpd: clearer error message if Twisted needs a patch (by Nils Durner) 2010-01-26 06:34:11 -08:00
francois
b03406af9d tahoe_backup.py: display warnings on errors instead of stopping the whole backup. Fix #729. 
This patch displays a warning to the user in two cases:
  
  1. When special files like symlinks, fifos, devices, etc. are found in the
     local source.
  
  2. If files or directories are not readables by the user running the 'tahoe
     backup' command.

In verbose mode, the number of skipped files and directories is printed at the
end of the backup.

Exit status returned by 'tahoe backup':

  - 0 everything went fine
  - 1 the backup failed
  - 2 files were skipped during the backup
 2010-01-20 01:42:49 -08:00
david-sarah
174d356692 Message saying that we couldn't find bin/tahoe should say where we looked 2010-01-16 12:45:56 -08:00
Zooko O'Whielacronx
a1444d9367 cli: merge the better version of David-Sarah's split-usage-and-help patch with the earlier version that I mistakenly committed 2010-01-25 20:45:59 -08:00
david-sarah
b079f32da2 Split tahoe --help options into groups. 2010-01-11 20:39:35 -08:00
Zooko O'Whielacronx
b94b9af189 cli: split usage strings into groups (patch by David-Sarah Hopwood) 2010-01-25 20:39:21 -08:00
david-sarah
87f1bae7fe Add create-node CLI command, and make create-client equivalent to create-node --no-storage (fixes #760) 2010-01-15 21:20:55 -08:00
david-sarah
26ab58e006 Remove replace= parameter to mkdir-immutable and mkdir-with-children 2010-01-24 14:43:25 -08:00
Brian Warner
de14791caf Fix webapi t=mkdir with multpart/form-data, as on the Welcome page. Closes #919. 2010-01-20 22:50:52 -08:00
Brian Warner
bb4883bbd4 tahoe_add_alias.py: minor refactoring 2010-01-14 22:42:20 -08:00
Brian Warner
9ab7524f0d test_dirnode.py: reduce scope of a Client instance, suggested by Kevan. 2010-01-14 22:27:13 -08:00
Brian Warner
01a7630334 test_provisioning: STAN is not always a list. Fix by David-Sarah Hopwood. 2010-01-14 17:46:32 -08:00
Brian Warner
2e098e2a4d web/directory.py mkdir-immutable: hush pyflakes, add TODO for #903 behavior 2010-01-14 14:28:04 -08:00
Brian Warner
e9223fcd0e hush pyflakes-0.4.0 warnings: slightly less-trivial fixes. Closes #900. 
This includes one fix (in test_web) which was testing the wrong thing.
 2010-01-14 14:17:19 -08:00
Brian Warner
731d15e56f hush pyflakes-0.4.0 warnings: remove trivial unused variables. For #900. 2010-01-14 14:15:29 -08:00
Brian Warner
874a979a8e tahoe add-alias/create-alias: don't corrupt non-newline-terminated alias 
file. Closes #741.
 2010-01-14 13:02:46 -08:00
Brian Warner
d3d1293d2f change docs and --help to use "grid" instead of "virtual drive": closes #892. 
Thanks to David-Sarah Hopwood for the patch.
 2010-01-14 12:11:19 -08:00
Brian Warner
fb879ddea4 client.py: fix/update comments on KeyGenerator 2010-01-12 16:42:26 -08:00
Brian Warner
d888bf3377 Clean up log.err calls, for one of the issues in #889. 
allmydata.util.log.err() either takes a Failure as the first positional
argument, or takes no positional arguments and must be invoked in an
exception handler. Fixed its signature to match both foolscap.logging.log.err
and twisted.python.log.err . Included a brief unit test.
 2010-01-11 17:33:43 -08:00
Brian Warner
bacb6fe5aa tidy up DeadReferenceError handling, ignore them in add_lease calls 
Stop checking separately for ConnectionDone/ConnectionLost, since those have
been folded into DeadReferenceError since foolscap-0.3.1 . Write
rrefutil.trap_deadref() in terms of rrefutil.trap_and_discard() to improve
code coverage.
 2010-01-11 16:07:23 -08:00
Brian Warner
1ed5bbdcb6 test_repairer: add (commented-out) test_each_byte, to see exactly what the 
Verifier misses

The results (described in #819) match our expectations: it misses corruption
in unused share fields and in most container fields (which are only visible
to the storage server, not the client). 1265 bytes of a 2753 byte
share (hosting a 56-byte file with an artifically small segment size) are
unused, mostly in the unused tail of the overallocated UEB space (765 bytes),
and the allocated-but-unwritten plaintext_hash_tree (480 bytes).
 2010-01-10 12:35:52 -08:00
Zooko O'Whielacronx
e7a9c000bf repairer: fix some wrong offsets in the randomized verifier tests, debugged by Brian 
fixes #819
 2010-01-10 12:37:21 -08:00
Brian Warner
1ff49a15bb test_repairer: fix colliding basedir names, which caused test inconsistencies 2010-01-10 00:46:19 -08:00
Zooko O'Whielacronx
876c4a153b repairer: add deterministic test for #819, mark as TODO 2010-01-09 17:36:19 -08:00
Brian Warner
419a076f59 mutable/publish: don't loop() right away upon DeadReferenceError. Closes #877 
The bug was that a disconnected server could cause us to re-enter the initial
loop() call, sending multiple queries to a single server, provoking an
incorrect UCWE. To fix it, stall the loop() with an eventual.fireEventually()
 2010-01-02 14:08:41 -08:00
Brian Warner
db19b62702 immutable/checker.py: oops, forgot some imports. Also hush pyflakes. 2009-12-29 15:39:09 -08:00
Brian Warner
ba0690c9d7 mutable repair: return successful=False when numshares<k (thus repair fails), 
instead of weird errors. Closes #874 and #786.

Previously, if the file had 0 shares, this would raise TypeError as it tried
to call download_version(None). If the file had some shares but fewer than
'k', it would incorrectly raise MustForceRepairError.

Added get_successful() to the IRepairResults API, to give repair() a place to
report non-code-bug problems like this.
 2009-12-29 15:37:46 -08:00
Brian Warner
acd211765c node.py/interfaces.py: minor docs fixes 2009-12-29 15:04:09 -08:00
Brian Warner
794e32738f checker: don't let failures in add-lease affect checker results. Closes #875. 
Mutable servermap updates and the immutable checker, when run with
add_lease=True, send both the do-you-have-block and add-lease commands in
parallel, to avoid an extra round trip time. Many older servers have problems
with add-lease and raise various exceptions, which don't generally matter.
The client-side code was catching+ignoring some of them, but unrecognized
exceptions were passed through to the DYHB code, concealing the DYHB results
from the checker, making it think the server had no shares.

The fix is to separate the code paths. Both commands are sent at the same
time, but the errback path from add-lease is handled separately. Known
exceptions are ignored, the others (both unknown-remote and all-local) are
logged (log.WEIRD, which will trigger an Incident), but neither will affect
the DYHB results.

The add-lease message is sent first, and we know that the server handles them
synchronously. So when the checker is done, we can be sure that all the
add-lease messages have been retired. This makes life easier for unit tests.
 2009-12-29 15:01:08 -08:00
Brian Warner
a50fdca6ab test_cli: verify fix for "tahoe get" not creating empty file on error (#121) 2009-12-27 18:54:44 -05:00
Brian Warner
00d0ca3902 addendum to "Fix 'tahoe ls' on files (#771)" 
tahoe_ls.py: tolerate missing metadata
web/filenode.py: minor cleanups
test_cli.py: test 'tahoe ls FILECAP'
 2009-12-27 18:21:49 -05:00
Brian Warner
a8a768ef9d Fix 'tahoe ls' on files (#771). Patch adapted from Kevan Carstensen. 
web/filenode.py: also serve edge metadata when using t=json on a
                 DIRCAP/childname object.
tahoe_ls.py: list file objects as if we were listing one-entry directories.
             Show edge metadata if we have it, which will be true when doing
             'tahoe ls DIRCAP/filename' and false when doing 'tahoe ls
             FILECAP'
 2009-12-27 17:54:43 -05:00
Brian Warner
931ab76588 tahoe_get: don't create the output file on error. Closes #121. 2009-12-27 17:04:04 -05:00
Brian Warner
499add09e6 webapi: don't accept zero-length childnames during traversal. Closes #358, #676. 
This forbids operations that would implicitly create a directory with a
zero-length (empty string) name, like what you'd get if you did "tahoe put
local /oops/blah" (#358) or "POST /uri/CAP//?t=mkdir" (#676). The error
message is fairly friendly too.

Also added code to "tahoe put" to catch this error beforehand and suggest the
correct syntax (i.e. without the leading slash).
 2009-12-27 15:10:43 -05:00
Brian Warner
2e0a61a953 CLI: send 'Accept:' header to ask for text/plain tracebacks. Closes #646. 
The webapi has been looking for an Accept header since 1.4.0, but it treats a
missing header as equal to */* (to honor RFC2616). This change finally
modifies our CLI tools to ask for "text/plain, application/octet-stream",
which seems roughly correct (we either want a plain-text traceback or error
message, or an uninterpreted chunk of binary data to save to disk). Some day
we'll figure out how JSON fits into this scheme.
 2009-12-27 14:58:28 -05:00
Brian Warner
a4a6c02ef8 more #859: avoid deprecation warning for unit tests too, hush pyflakes 
* factor maybe-import-sha logic into util.hashutil
 2009-12-14 16:01:47 -08:00
Zooko O'Whielacronx
b69e1c600d use hashlib module if available, thus avoiding a DeprecationWarning for importing the old sha module; fixes #859 2009-12-14 13:27:03 -08:00
Brian Warner
47cba4b51a mutable/retrieve.py: stop reaching into private MutableFileNode attributes 2009-12-08 09:29:21 -08:00
Brian Warner
aa3caf812d mutable/servermap.py: stop reaching into private MutableFileNode attributes 2009-12-08 09:26:08 -08:00
Brian Warner
eb18686284 mutable/servermap.py: oops, query N+e servers in MODE_WRITE, not k+e 
under normal conditions, this wouldn't cause any problems, but if the shares
are really sparse (perhaps because new servers were added), then
file-modifies might stop looking too early and leave old shares in place
 2009-12-08 09:11:56 -08:00
Brian Warner
696cc000d0 control.py: fix speedtest: use download_best_version (not read) on mutable nodes 2009-12-06 22:05:12 -08:00
Zooko O'Whielacronx
4706859967 setup: ignore _darcs in the "test-clean" test and make the "clean" step remove all .egg's in the root dir 2009-12-06 10:48:35 -08:00
Brian Warner
a980ea337b remove MutableFileNode.download(), prefer download_best_version() instead 2009-12-01 17:54:38 -05:00
Brian Warner
96834da0a2 Simplify immutable download API: use just filenode.read(consumer, offset, size) 
* remove Downloader.download_to_data/download_to_filename/download_to_filehandle
* remove download.Data/FileName/FileHandle targets
* remove filenode.download/download_to_data/download_to_filename methods
* leave Downloader.download (the whole Downloader will go away eventually)
* add util.consumer.MemoryConsumer/download_to_data, for convenience
  (this is mostly used by unit tests, but it gets used by enough non-test
   code to warrant putting it in allmydata.util)
* update tests
* removes about 180 lines of code. Yay negative code days!

Overall plan is to rewrite immutable/download.py and leave filenode.read() as
the sole read-side API.
 2009-12-01 17:53:30 -05:00
Brian Warner
1059db51f2 server.py: undo my bogus 'correction' of David-Sarah's comment fix 
and move it to a better line
 2009-11-30 21:46:07 -05:00
Brian Warner
74974b27fe Implement more coherent behavior when copying with dircaps/filecaps (closes #761). Patch by Kevan Carstensen. 2009-11-30 13:10:09 -08:00
Brian Warner
91e7cb484b storage.py: update comment 2009-11-30 11:59:13 -08:00
david-sarah
ef002c935a storage server: detect disk space usage on Windows too (fixes #637) 2009-11-20 21:56:44 -08:00
david-sarah
c4d38ad4c5 make status of finished operations consistently "Finished" 2009-11-20 22:15:43 -08:00
Brian Warner
9901a08fd3 Add dirnodes to backupdb and "tahoe backup", closes #606. 
* backups now share dirnodes with any previous backup, in any location,
  so renames and moves are handled very efficiently
* "tahoe backup" no longer bothers reading the previous snapshot
* if you switch grids, you should delete ~/.tahoe/private/backupdb.sqlite,
  to force new uploads of all files and directories
 2009-11-26 15:42:57 -08:00
Brian Warner
366a309795 webapi: fix t=check for DIR2-LIT (i.e. empty immutable directories) 2009-11-26 15:27:31 -08:00
Brian Warner
41bcc9f39e PipelineError: fix str() on python2.4 . Closes #842. 2009-11-24 13:25:12 -08:00
Brian Warner
d918ab407e test_uri.py: s/NewDirnode/Dirnode/ , now that they aren't "new" anymore 2009-11-19 23:55:53 -08:00
Brian Warner
0cf320c2ab interface name cleanups: IFileNode, IImmutableFileNode, IMutableFileNode 
The proper hierarchy is:
 IFilesystemNode
 +IFileNode
 ++IMutableFileNode
 ++IImmutableFileNode
 +IDirectoryNode

Also expand test_client.py (NodeMaker) to hit all IFilesystemNode types.
 2009-11-19 23:52:55 -08:00
Brian Warner
d2badbea78 class name cleanups: s/FileNode/ImmutableFileNode/ 
also fix test/bench_dirnode.py for recent dirnode changes
 2009-11-19 23:22:39 -08:00
Brian Warner
6e7fb1006d Use DIR-IMM and t=mkdir-immutable for "tahoe backup", for #828 2009-11-18 11:28:13 -08:00
Brian Warner
834b20210a web/directory.py: use "DIR-IMM" to describe immutable directories, not DIR-RO 2009-11-18 11:18:32 -08:00
Brian Warner
e5452290f7 web/info.py: hush pyflakes 2009-11-18 11:17:36 -08:00
Brian Warner
e046744f40 make get_size/get_current_size consistent for all IFilesystemNode classes 
* stop caching most_recent_size in dirnode, rely upon backing filenode for it
* start caching most_recent_size in MutableFileNode
* return None when you don't know, not "?"
* only render None as "?" in the web "more info" page
* add get_size/get_current_size to UnknownNode
 2009-11-18 11:16:24 -08:00
Brian Warner
512fe3ad62 ImmutableDirectoryURIVerifier: fix verifycap handling 2009-11-18 08:42:38 -08:00