Commit Graph

2061 Commits

Author SHA1 Message Date
david-sarah
5ba0529b87 test/test_runner.py: BinTahoe.test_path has rare nondeterministic failures; this patch probably fixes a problem where the actual cause of failure is masked by a string conversion error. 2011-09-27 15:53:36 -07:00
kevan
2187f27ad0 test/test_mutable: reenable mdmf publish surprise test 2011-09-24 16:54:15 -07:00
david-sarah
4af626a798 test_mutable.py: update SkipTest message for test_publish_surprise_mdmf to reference the right ticket number. refs #1540. 2011-09-23 14:16:22 -07:00
david-sarah
5d3d0dc336 test_mutable.py: skip test_publish_surprise_mdmf, which is causing an error. refs #1534, #393 2011-09-20 11:33:19 -07:00
kevan
d93b2965c3 test/test_mutable: write publish surprise test for MDMF, rename existing test_publish_surprise to clarify that it is for SDMF 2011-09-17 17:36:57 -07:00
kevan
19cf58f2aa test/test_mutable: refactor publish surprise test into common test fixture, rewrite test_publish_surprise to use test fixture 2011-09-17 17:35:33 -07:00
Brian Warner
8e69b94588 Make platform-detection code tolerate linux-3.0, patch by zooko.
Otherwise address-autodetection can't find ifconfig. refs #1536
2011-09-15 13:26:20 -07:00
david-sarah
3f756e9429 test_web.py: fix a bug in _count_leases that was causing us to check only the lease count of one share file, not of all share files as intended. 2011-09-15 11:51:26 -07:00
Zooko O'Whielacronx
2025e42ec2 tests: bump up the timeout in this test that fails on FreeStorm's CentOS in order to see if it is just very slow 2011-09-12 19:42:55 -07:00
Zooko O'Whielacronx
942c5e5162 storage: test that the storage server ignores requests to extend shares by sending a new_length, and that the storage server fills exposed holes with 0 to avoid "palimpsest" exposure of previous contents
ref. #1528
2011-09-12 15:25:54 -07:00
Zooko O'Whielacronx
65de17245d storage: test that the storage server does *not* have a "remote_cancel_lease" function
We're removing this function because it is currently unused, because it is dangerous, and because the bug described in #1528 leaks the cancellation secret, which allows anyone who knows a file's storage index to abuse this function to delete shares of that file.
ref. #1528
2011-09-12 15:23:24 -07:00
Zooko O'Whielacronx
cffc987804 immutable: test whether the server allows clients to read past the end of share data, which would allow them to learn the cancellation secret
Also test whether the server explicitly declares that it prevents this problem.
ref #1528
2011-09-12 15:12:01 -07:00
Brian Warner
a15ce96846 Retrieve: implement/test stopProducing 2011-09-09 11:11:50 -07:00
Brian Warner
748e419a9b move DownloadStopped from download.common to interfaces 2011-09-09 11:11:50 -07:00
Brian Warner
2edfb1a334 Retrieve: remove the initial prefix-is-still-good check
This check needs to be done with each fetch from the storage server, to
detect when someone has changed the share (i.e. our servermap goes stale).
Doing it just once at the beginning of retrieve isn't enough: a write might
occur after the first segment but before the second, etc.

_try_to_validate_prefix() was not removed: it will be used by the future
check-with-each-fetch code.

test_mutable.Roundtrip.test_corrupt_all_seqnum_late was disabled, since it
fails until this check is brought back. (the corruption it applies only
touches the prefix, not the block data, so the check-less retrieve actually
tolerates it). Don't forget to re-enable it once the check is brought back.
2011-09-09 11:11:50 -07:00
Brian Warner
2b4f2b7fa3 MDMFSlotReadProxy: remove the queue
This is a neat trick to reduce Foolscap overhead, but the need for an
explicit flush() complicates the Retrieve path and makes it prone to
lost-progress bugs.

Also change test_mutable.FakeStorageServer to tolerate multiple reads of the
same share in a row, a limitation exposed by turning off the queue.
2011-09-09 11:11:50 -07:00
david-sarah
6b97e6f407 CLI: test_cli.py -- remove an unnecessary call in test_mkdir_mutable_type. refs #1527 2011-09-06 11:37:30 -07:00
david-sarah
1e59930fda CLI: improve test for 'tahoe mkdir --mutable-type='. refs #1527 2011-09-06 11:30:20 -07:00
david-sarah
23f46b758e CLI: make the --mutable-type option value for 'tahoe put' and 'tahoe mkdir' case-insensitive, and change --help for these commands accordingly. fixes #1527 2011-09-04 19:09:22 -07:00
Kevan Carstensen
64996a913d cli: make --mutable-type imply --mutable in 'tahoe put' 2011-09-03 12:09:20 -07:00
david-sarah
1b7b3bd0d4 SFTP: add a comment about a subtle interaction between OverwriteableFileConsumer and GeneralSFTPFile, and test the case it is commenting on. 2011-09-03 15:23:04 -07:00
Brian Warner
7191b0b32e test_mutable.Version: consolidate some tests, reduce runtime from 19s to 15s 2011-08-30 22:04:51 -07:00
Kevan Carstensen
32e30c9023 mutable/retrieve: handle the case where self._read_length is 0.
Note that the downloader will still fetch a segment for a zero-length
read, which is wasteful. Fixing that isn't specifically required to fix
#1512, but it should probably be fixed before 1.9.
2011-08-30 14:01:41 -07:00
Brian Warner
980eb778c1 test_mutable.Update: only upload the files needed for each test. refs #1500
This first step shaves 15% off the runtime: from 139s to 119s on my laptop.
It also fixes a couple of places where a Deferred was being dropped, which
would cause two tests to run in parallel and also confuse error reporting.
2011-08-29 00:27:17 -07:00
Brian Warner
fd676a5846 Let Uploader retain History instead of passing it into upload(). Fixes #1079.
This consistently records all immutable uploads in the Recent Uploads And
Downloads page, regardless of code path. Previously, certain webapi upload
operations (like PUT /uri/$DIRCAP/newchildname) failed to pass the History
object and were left out.
2011-08-28 23:32:46 -07:00
Brian Warner
d575ccba28 Teach 'tahoe debug catalog-shares about MDMF. Closes #1507. 2011-08-28 01:09:31 -07:00
Brian Warner
a3d6fd4992 teach 'tahoe debug dump-share' about MDMF and offsets. refs #1507 2011-08-28 00:38:34 -07:00
Brian Warner
101ca8dfe4 test_mutable.Version.test_debug: use splitlines() to fix buildslaves
Any slave running in a directory with spaces in the name was miscounting
shares, causing the test to fail.
2011-08-27 23:47:28 -07:00
Brian Warner
7d48f902fc test_mutable.Version: exercise 'tahoe debug find-shares' on MDMF. refs #1507
Also changes NoNetworkGrid to put shares in storage/shares/ .
2011-08-27 17:55:42 -07:00
Brian Warner
80bdc0f5b7 test_mutable.py: oops, missed a .todo 2011-08-27 17:21:18 -07:00
Brian Warner
e0eaeff7f5 test_mutable: merge davidsarah's patch with my Version refactorings 2011-08-27 16:57:07 -07:00
david-sarah
88989a4ea2 Additional tests for MDMF URIs and for zero-length files. refs #393 2011-08-22 18:15:32 -07:00
david-sarah
ac7b8400d4 Additional tests for zero-length partial reads and updates to mutable versions. refs #393 2011-08-21 18:41:11 -07:00
Brian Warner
03ba8065ce test_mutable.Version: factor out some expensive uploads, save 25% runtime 2011-08-27 16:27:37 -07:00
Brian Warner
370e6f271e SDMF: update filenode with correct k/N after Retrieve. Fixes #1510.
Without this, we get a regression when modifying a mutable file that was
created with more shares (larger N) than our current tahoe.cfg . The
modification attempt creates new versions of the (0,1,..,newN-1) shares, but
leaves the old versions of the (newN,..,oldN-1) shares alone (and throws a
assertion error in SDMFSlotWriteProxy.finish_publishing in the process).

The mixed versions that result (some shares with e.g. N=10, some with N=20,
such that both versions are recoverable) cause problems for the Publish code,
even before MDMF landed. Might be related to refs #1390 and refs #1042.
2011-08-27 15:50:31 -07:00
Brian Warner
482a7dd3f1 Add 'tahoe debug dump-cap' support for MDMF, DIR2-CHK, DIR2-MDMF. refs #1507.
This also adds tests for all those cases, and fixes an omission in uri.py
that broke parsing of DIR2-MDMF-Verifier and DIR2-CHK-Verifier.
2011-08-27 12:50:48 -07:00
Brian Warner
522868d5e9 MDMF: more writable/writeable consistentifications 2011-08-27 12:06:02 -07:00
Brian Warner
48544a251d MDMF: s/Writable/Writeable/g, for consistency with existing SDMF code 2011-08-27 11:33:57 -07:00
Brian Warner
349093fa2e test_mutable.Update: increase timeout from 120s to 400s, slaves are failing 2011-08-25 16:01:40 -07:00
Zooko O'Whielacronx
5bf5404ba1 tests: fix check_memory test
fixes #1503
2011-08-25 13:11:16 -07:00
Brian Warner
9e613ee9b3 test_mutable.Update: remove .timeout overrides, otherwise tests ERROR 2011-08-24 19:24:55 -07:00
david-sarah
3d7a32647c Implementation, tests and docs for blacklists. This version allows listing directories containing a blacklisted child. Inclusion of blacklist.py fixed. fixes #1425 2011-08-24 08:59:28 -07:00
Kevan Carstensen
b49986f801 test/test_mutable: test for incorrect div_ceil equations 2011-08-13 11:39:36 -07:00
Brian Warner
fa6a768bc1 now that tests pass with full-size keys, return test-keys to normal (522bit) 2011-08-11 10:54:18 -07:00
david-sarah
dd23f265bf More idiomatic resolution of the conflict between ticket393-MDMF-2 and trunk. refs #393 2011-08-10 13:29:42 -07:00
david-sarah
c55c44e70a Replace the hard-coded 522-bit RSA key size used for tests with a TEST_RSA_KEY_SIZE constant defined in test/common.py (part 2). refs #393 2011-08-10 13:23:10 -07:00
david-sarah
fbe0e42bbf Replace the hard-coded 522-bit RSA key size used for tests with a TEST_RSA_KEY_SIZE constant defined in test/common.py (part 1). refs #393 2011-08-10 13:22:43 -07:00
Zooko O'Whielacronx
95a7551586 merge some minor conflicts in test code from the 393-2 branch and trunk 2011-08-10 10:21:39 -07:00
david-sarah
b60506b0ba test_web.py: fix a test failure dependent on whether simplejson.loads returns a unicode or str object. 2011-08-08 14:39:25 -07:00
Kevan Carstensen
47eab03c6c test: fix assorted tests broken by MDMF changes 2011-08-06 17:44:59 -07:00
Kevan Carstensen
a7fb55f986 uri: add MDMF and MDMF directory caps, add extension hint support 2011-08-06 17:44:36 -07:00
Kevan Carstensen
52e9512b49 test/test_mutable: tests for MDMF
These are their own patch because they cut across a lot of the changes
I've made in implementing MDMF in such a way as to make it difficult to
split them up into the other patches.
2011-08-06 17:44:14 -07:00
Kevan Carstensen
a9cada2e03 webapi changes for MDMF
- Learn how to create MDMF files and directories through the
      mutable-type argument.
    - Operate with the interface changes associated with MDMF and #993.
    - Learn how to do partial updates of mutable files.
2011-08-06 17:43:48 -07:00
Kevan Carstensen
cde17ec041 dirnode: teach dirnode to make MDMF directories 2011-08-06 17:42:24 -07:00
david-sarah
50c692d1c5 Fix some test failures caused by #393 patch. 2011-08-01 20:28:10 -07:00
Kevan Carstensen
4d09ef3d83 cli: teach CLI how to create MDMF mutable files
Specifically, 'tahoe mkdir' and 'tahoe put' now take a --mutable-type
argument.
2011-08-01 19:16:13 -07:00
Kevan Carstensen
a87fc8ffab frontends/sftpd: Resolve incompatibilities between SFTP frontend and MDMF changes 2011-08-01 19:12:07 -07:00
Kevan Carstensen
b1b77d3b89 mutable/layout: Define MDMF share format, write tools for working with MDMF share format
The changes in layout.py are mostly concerned with the MDMF share
format. In particular, we define read and write proxy objects used by
retrieval, publishing, and other code to write and read the MDMF share
format. We create equivalent proxies for SDMF objects so that these
objects can be suitably general.
2011-08-01 19:11:20 -07:00
Kevan Carstensen
edf9858fb8 immutable/filenode: implement unified filenode interface 2011-08-01 19:09:05 -07:00
Kevan Carstensen
3e996e0c94 immutable/literal.py: Implement interface changes in literal nodes. 2011-08-01 19:08:14 -07:00
Kevan Carstensen
8ae564d8ad test/common: Alter common test code to work with MDMF.
This mostly has to do with making the test code implement the new
unified filenode interfaces.
2011-08-01 18:56:43 -07:00
david-sarah
a48aee11b3 allmydata/__init__.py, test_version.py: make version parsing understand '<=', with test. refs #1435 2011-08-14 20:51:53 -07:00
david-sarah
6c2670641e Add test for webopen. fixes #1149 2011-07-24 14:16:59 -07:00
david-sarah
ab9eb12f70 test_client.py: relax a check in test_create_drop_uploader so that it should pass on Python 2.4.x. refs #1429 2011-08-09 22:25:04 -07:00
david-sarah
0dc56daaf9 test/common_util.py: correct fix to mkdir_nonascii. refs #1472 2011-08-09 22:19:06 -07:00
david-sarah
2deba3319c test/common_util.py: fix a typo. refs #1472 2011-08-09 21:42:35 -07:00
david-sarah
0104dbab92 test_client.py, test_drop_upload.py: fix pyflakes warnings. 2011-08-09 20:45:05 -07:00
david-sarah
db22fdc20d Factor out methods dealing with non-ASCII directories and filenames from test_drop_upload.py into common_util.py. refs #1429, #1472 2011-08-09 20:15:58 -07:00
david-sarah
c102056ac1 test_client.py: add a test that the drop-uploader is initialized correctly by client.py. Also give the DropUploader service a name, which is necessary for the test. refs #1429 2011-08-09 20:05:38 -07:00
david-sarah
10ee22f50e drop-upload: rename 'start' method to 'startService', which is what you're supposed to use to start a Service. refs #1429 2011-08-09 20:03:45 -07:00
david-sarah
f157b73367 test_drop_upload.py: add comment explaining why we don't use FilePath.setContent. refs #1429 2011-08-09 19:59:42 -07:00
david-sarah
369e30b1df test_drop_upload.py: fix some grammatical and spelling nits. refs #1429 2011-08-09 15:12:31 -07:00
Zooko O'Whielacronx
5633375d26 drop-upload test for non-existent local dir separately from test for non-directory local dir
A candidate patch for #1429 has a bug when it is using FilePath.is_dir() to detect whether the configured local dir exists and is a directory. FilePath.is_dir() raises exception, instead of returning False, if the thing doesn't exist. This test is to make sure that DropUploader.__init__ raise different exceptions for those two cases.
refs #1429
2011-08-09 15:01:15 -07:00
Zooko O'Whielacronx
720bc2433b drop-upload: unit tests for the configuration options being named "cap" instead of "uri"
This is a subset of a patch that David-Sarah attached to #1429. This is just the unit-tests part of that patch, and uses darcs record instead of hunks to change the names.
refs #1429
2011-08-09 14:59:13 -07:00
david-sarah
4c592f1505 test_storage.py: test that we are using the filesystem of storage/shares/, rather than storage/, to calculate remaining space, and that the HTML status output reflects the values returned by fileutil.get_disk_stats. This version works with older versions of the mock library. refs #1384 2011-08-09 12:07:22 -07:00
david-sarah
70f03bd615 Work around ref #1472 by having test_drop_upload delete the non-ASCII directories it creates. 2011-08-08 18:23:34 -07:00
david-sarah
bbed522033 Remove all trailing whitespace from .py files. 2011-08-08 17:11:17 -07:00
david-sarah
08af9cea50 test_drop_upload.py: fix unused imports. refs #1429 2011-08-08 16:54:22 -07:00
david-sarah
32a7717205 Drop-upload frontend, rerecorded for 1.9 beta (and correcting a minor mistake). Includes some fixes for Windows but not the Windows inotify implementation. fixes #1429 2011-08-08 16:40:49 -07:00
david-sarah
b9121e045f test_runner.py: fix a race condition in the test when NODE_URL_FILE is written before PORTNUM_FILE. refs #1469 2011-08-06 16:18:42 -07:00
david-sarah
743cb898e7 test_runner.py: cleanups of HOTLINE_FILE writing and removal. 2011-08-06 16:16:52 -07:00
david-sarah
e3c600d047 test_runner.py: remove an unused constant. 2011-08-06 15:14:16 -07:00
david-sarah
531758df19 test_runner.py: test that client and introducer nodes record their port number and use that port on the next restart. This tests for a regression caused by ref #1385. 2011-08-06 15:06:35 -07:00
david-sarah
b9eb0235ea test_runner.py: fix a bug in CreateNode.do_create introduced in changeset [5114] when the tahoe.cfg file has been written with CRLF line endings. refs #1385 2011-08-03 17:30:32 -07:00
david-sarah
b6cfbbeb23 test_client.py: repair Basic.test_error_on_old_config_files. refs #1385 2011-08-03 16:50:36 -07:00
david-sarah
1c77c5f5dc test_checker.py: increase timeout for TooParallel.test_immutable again. The ARM buildslave took 38 seconds, so 40 seconds is too close to the edge; make it 80. 2011-08-03 14:40:42 -07:00
david-sarah
521754b506 test_runner.py: fix RunNode.test_introducer to not rely on the mtime of introducer.furl to detect when the node has restarted. Instead we detect when node.url has been written. refs #1385 2011-08-03 11:09:17 -07:00
david-sarah
ed3dbe4985 test_checker.py: increase timeout for TooParallel.test_immutable (was consistently failing on ARM buildslave). 2011-08-03 09:32:13 -07:00
david-sarah
e74387f4f1 test_runner.py: modify RunNode.test_introducer to test that starting an introducer works when the introducer.furl file already exists. refs #1385 2011-08-02 18:27:04 -07:00
david-sarah
9f8d34e63a test_repairer: add a deterministic test of share data corruption that always flips the bits of the last byte of the share data. refs #1395 2011-08-02 10:58:41 -07:00
Zooko O'Whielacronx
f426e82287 verifier: serialize the fetching of blocks within a share so that we don't use too much RAM
Shares are still verified in parallel, but within a share, don't request a
block until the previous block has been verified and the memory we used to hold
it has been freed up.

Patch originally due to Brian. This version has a mockery-patchery-style test
which is "low tech" (it implements the patching inline in the test code instead
of using an extension of the mock.patch() function from the mock library) and
which unpatches in case of exception.

fixes #1395
2011-08-01 23:37:03 -07:00
Brian Warner
3668cb3d06 remove nodeid from WriteBucketProxy classes and customers
refs #1363
2011-08-01 15:43:17 -07:00
Brian Warner
550d67f51f remove get_serverid() from ReadBucketProxy and customers, including Checker
and debug.py dump-share commands
refs #1363
2011-08-01 15:43:07 -07:00
Zooko O'Whielacronx
e5c4e83f4c reject old-style (pre-Tahoe-LAFS-v1.3) configuration files
Check for the existence of any of them and if any are found raise exception which will abort the startup of the node.
This is a backwards-incompatible change for anyone who is still using old-style configuration files.
fixes #1385
2011-08-01 16:24:23 -07:00
Zooko O'Whielacronx
420e3aa293 whitespace-cleanup 2011-07-24 18:55:46 -07:00
Zooko O'Whielacronx
3505a8a31c tests: use fileutil.write() instead of open() to ensure timely close even without CPython-style reference counting
Some of these already had an explicit close() but I went ahead and replaced them with fileutil.write() as well for the sake of uniformity.
2011-03-31 07:54:27 -07:00
david-sarah
06a5d0c1a3 cleanup: implement rm as a synonym for unlink rather than vice-versa. refs #776 2011-08-01 15:01:08 -07:00
david-sarah
07ecac1d83 WUI: change the label of the button to unlink a file from 'del' to 'unlink'. Also change some internal names to 'unlink', and allow 't=unlink' as a synonym for 't=delete' in the web-API interface. Incidentally, improve a test to check for the rename button as well as the unlink button. fixes #1104 2011-07-12 17:12:18 -07:00
Brian Warner
294dc5d0c7 fix typo introduced during rebasing of 'remove get_serverid from
DownloadStatus.add_dyhb_request and customers' patch, to fix test failure.
2011-08-01 13:03:41 -07:00
Zooko O'Whielacronx
6b2e798595 remove get_serverid from DownloadStatus.add_dyhb_request and customers
This patch is a rebase of a patch originally written by Brian. I didn't change any of the intent of Brian's patch, just ported it to current trunk.
refs #1363
2011-08-01 11:54:01 -07:00
Brian Warner
0605c77f08 test_immutable.Test: rewrite to use NoNetworkGrid, now takes 2.7s not 97s
remove now-unused ShareManglingMixin
refs #1363
2011-08-01 10:44:44 -07:00
Brian Warner
0f11d35f85 replace IServer.name() with get_name(), and get_longname()
This patch was originally written by Brian, but was re-recorded by Zooko to use
darcs replace instead of hunks for any file in which it would result in fewer
total hunks.
refs #1363
2011-08-01 10:44:28 -07:00
david-sarah
00fefeba49 test_cli.py: Test for error message when 'tahoe rm' is invoked without a path. refs #1292 2011-01-04 04:51:08 -07:00
david-sarah
16fd14a78a test_cli.py: use to_str on fields loaded using simplejson.loads in new tests. refs #1304 2011-07-29 20:25:21 -07:00
Kevan Carstensen
448278e807 cli: make 'tahoe cp' overwrite mutable files in-place 2011-07-29 13:20:39 -07:00
david-sarah
a2699ea6f6 SFTP: write an error message to standard error for unrecognized shell commands. Change the existing message for shell sessions to be written to standard error, and refactor some duplicated code. Also change the lines of the error messages to end in CRLF, and take into account Kevan's review comments. fixes #1442, #1446 2011-07-29 16:31:02 -07:00
david-sarah
b978378392 Fix the help synopses of CLI commands to include [options] in the right place. fixes #1359, fixes #636 2011-07-24 15:54:40 -07:00
david-sarah
f9d218c673 encodingutil: argv and output encodings are always the same on all platforms. Lose the unnecessary generality of them being different. fixes #1120 2011-06-29 11:53:56 -07:00
david-sarah
5fcd3f8803 test_sftp.py cleanup: remove a redundant definition of failUnlessReallyEqual. 2011-07-16 11:18:13 -07:00
Zooko O'Whielacronx
4f8e3e5ae8 contrib: remove the contributed fuse modules and the entire contrib/ directory, which is now empty
Also remove a couple of vestigial references to figleaf, which is long gone.
fixes #1409 (remove contrib/fuse)
2011-07-12 08:32:29 -07:00
Brian Warner
d8358f2863 add more download-status data, fix tests 2011-06-29 15:25:55 -07:00
Brian Warner
fc5c2208fb prepare for viz: improve DownloadStatus events
consolidate IDownloadStatusHandlingConsumer stuff into DownloadNode
2011-06-29 15:25:42 -07:00
Brian Warner
fb4dd43310 Rename test_package_initialization.py to (much shorter) test_import.py .
The former name was making my 'ls' listings hard to read, by forcing them
down to just two columns.
2011-06-11 12:02:34 -07:00
Zooko O'Whielacronx
be1ebde6e5 tests: fix tests to accomodate [20110611153758-92b7f-0ba5e4726fb6318dac28fb762a6512a003f4c430]
Apparently none of the two authors (stercor, terrell), three reviewers (warner, davidsarah, terrell), or one committer (me) actually ran the tests. This is presumably due to #20.
fixes #1412
2011-06-11 09:37:41 -07:00
wilcoxjg
67ad0175cd server.py: get_latencies now reports percentiles _only_ if there are sufficient observations for the interpretation of the percentile to be unambiguous.
interfaces.py:  modified the return type of RIStatsProvider.get_stats to allow for None as a return value
NEWS.rst, stats.py: documentation of change to get_latencies
stats.rst: now documents percentile modification in get_latencies
test_storage.py:  test_latencies now expects None in output categories that contain too few samples for the associated percentile to be unambiguously reported.
fixes #1392
2011-05-27 05:01:35 -07:00
david-sarah
a817163cc6 docs: convert NEWS to NEWS.rst and change all references to it. 2011-05-16 18:02:55 -07:00
david-sarah
a09d7300e0 tests: add test for including the ImportError message and traceback entry in the summary of errors from importing dependencies. refs #1389 2011-04-10 08:58:44 -07:00
david-sarah
f3246a9ca0 allmydata/__init__.py: Nicer reporting of unparseable version numbers in dependencies. fixes #1388 2011-04-01 13:27:50 -07:00
Brian Warner
9ae026d9f4 immutable/downloader/share.py: reduce get_serverid(), one left, update ext deps
test_download.py: create+check MyShare instances better, make sure they share
Server objects, now that finder.py cares
2011-02-26 19:11:50 -07:00
Brian Warner
2f63d9b522 immutable/upload.py: reduce use of get_serverid() 2011-02-26 19:11:38 -07:00
Brian Warner
9672b34880 add remaining get_* methods to storage_client.Server, NoNetworkServer, and
MockIServer stubs
2011-02-26 19:11:32 -07:00
Brian Warner
ee14f2ad67 happinessutil.py: finally rename merge_peers to merge_servers 2011-02-26 19:11:24 -07:00
Brian Warner
13d284b087 test_upload.py: factor out FakeServerTracker 2011-02-26 19:11:20 -07:00
Brian Warner
9fc55bad25 test_upload.py: server-vs-tracker cleanup 2011-02-26 19:11:15 -07:00
Brian Warner
ebfcb649f9 refactor: s/peer/server/ in immutable/upload, happinessutil.py, test_upload
No behavioral changes, just updating variable/method names and log messages.
The effects outside these three files should be minimal: some exception
messages changed (to say "server" instead of "peer"), and some internal class
names were changed. A few things still use "peer" to minimize external
changes, like UploadResults.timings["peer_selection"] and
happinessutil.merge_peers, which can be changed later.
2011-02-26 19:11:00 -07:00
Brian Warner
9b9ea3f739 storage_client.py: clean up test_add_server/test_add_descriptor, remove .test_servers 2011-02-26 19:10:56 -07:00
Brian Warner
cba3de04a4 test_client.py, upload.py:: remove KiB/MiB/etc constants, and other dead code 2011-02-26 19:10:51 -07:00
Zooko O'Whielacronx
578a117c5b test: increase timeout on a network test because Francois's ARM machine hit that timeout
I'm skeptical that the test was proceeding correctly but ran out of time. It seems more likely that it had gotten hung. But if we raise the timeout to an even more extravagant number then we can be even more certain that the test was never going to finish.
2011-03-17 09:59:09 -07:00
david-sarah
787d12165a Add unit tests for cross_check_pkg_resources_versus_import, and a regression test for ref #1355. This requires a little refactoring to make it testable. 2011-02-20 17:58:17 -08:00
Brian Warner
ffd296fc5a Refactor StorageFarmBroker handling of servers
Pass around IServer instance instead of (peerid, rref) tuple. Replace
"descriptor" with "server". Other replacements:

 get_all_servers -> get_connected_servers/get_known_servers
 get_servers_for_index -> get_servers_for_psi (now returns IServers)

This change still needs to be pushed further down: lots of code is now
getting the IServer and then distributing (peerid, rref) internally.
Instead, it ought to distribute the IServer internally and delay
extracting a serverid or rref until the last moment.

no_network.py was updated to retain parallelism.
2011-02-20 17:58:04 -08:00
david-sarah
ed67df866c SFTP: report unknown sizes as "0" instead of "?", to satisfy some clients. fixes #1337 2011-01-27 22:22:49 -08:00
david-sarah
d318122cd2 Change tests for check_requirement to ensure that it accepts None in the comment field of a (version, location, comment) triple, and add a test that it raises ImportError when a module couldn't be imported. Also fix typo in a comment. refs #1339 2011-01-28 11:24:52 -08:00
Zooko O'Whielacronx
6f84f1b869 setup: adjust tests to use the new interface of check_requirement which has a 3-tuples instead of a 2-tuple
fixes #1339
2011-01-28 06:15:46 -08:00
david-sarah
f84f2aa987 tests: use verlib to compare versions, and get the versions of Twisted and Nevow from get_package_versions() instead of pkg_resources. refs #1287 2011-01-21 17:42:56 -08:00
david-sarah
8587c63bdc Make bb-freeze (and probably other static packaging tools) work. This updates various places where we assumed that the tahoe process was executed via the Python interpreter. It also allows tests to recursively invoke the same tahoe.exe, rather than bin/tahoe. refs #585 2011-01-21 00:04:29 -08:00
david-sarah
29336a0916 Refactor _auto_deps.py and __init__.py, adding more robust checking of dependency versions, and not trusting pkg_resources to get the versions right. refs #1258, #1287 2011-01-20 21:36:10 -08:00
david-sarah
6dd8b6f471 Eliminate dependencies on pywin32, even via Twisted. refs #1274 2011-01-19 20:32:38 -08:00
david-sarah
fd6cdc48ae src/allmydata/test/test_runner.py: add test_import_from_repl, which checks that we are running the right code in a bin/tahoe subprocess. refs #1258 2011-01-19 00:21:45 -08:00
david-sarah
a9fc4668c0 docs/frontends/CLI.rst, src/allmydata/test/trialtest.py: add trailing newlines. refs #1296 2011-01-19 00:19:55 -08:00
david-sarah
8f0af33ba6 src/allmydata/test/test_cli.py: add test for 'tahoe debug trial' options help. refs #1296 2011-01-18 18:42:24 -08:00
david-sarah
1819c25c88 Add src/allmydata/test/trialtest.py needed by tests for 'tahoe debug trial'. refs #1296 2011-01-18 18:02:39 -08:00
david-sarah
0d6df9c9fc Tests for 'tahoe debug trial' (rebased and fixed to work with Twisted 10.2). refs #1296 2011-01-18 17:38:59 -08:00
david-sarah
5a3acc05b7 CLI: tests for ref #1305 (v2, remove spurious extra arg to create-alias in test) 2011-01-13 20:03:27 -08:00
Brian Warner
b9fb74d2a0 Improve 'tahoe ln' help text. Patch by David-Sarah. Closes #1230. 2011-01-17 00:14:21 -08:00
Brian Warner
09a2241471 Tolerate Twisted-10.2's endpoints, patch by David-Sarah. Closes #1286.
The service generated by strports.service() changed in 10.2, and the ugly
private-attribute-reading hack we used to glean a kernel-allocated port
number (e.g. when using "tcp:0", especially during unit tests) broke, causing
Tahoe to be completely unusable with Twisted-10.2 . The new ugly
private-attribute-reading hack starts by figuring out what sort of service
was generated, then reads different attributes accordingly.

This also hushes a warning when using schemeless strports strings like "0" or
"3456", by quietly prepending a "tcp:" scheme, since 10.2 complains about
those. It also adds getURL() and getPortnum() accessors to the "webish"
service, rather than having unit tests dig through _url and _portnum and such
to find out what they are.
2011-01-16 23:47:51 -08:00
david-sarah
8a3cb60bf7 Test changes to take account of ref #1311. 2011-01-16 22:05:40 -08:00
Brian Warner
45212d8000 Set "reserved_space=1G" in newly-created storage nodes. Closes #1208. 2011-01-16 12:58:22 -08:00
david-sarah
e62a69e7ff test_runner: avoid unnecessary use of non-ASCII. 2011-01-01 02:01:01 -08:00
david-sarah
5027753e23 test_storage.py: fix a pyflakes unused import warning. 2010-12-31 14:07:56 -08:00
david-sarah
0da10d4c37 test_storage.py: leave at least 512 MiB free when running test_large_share. refs #1195 2010-12-31 12:32:15 -08:00
Zooko O'Whielacronx
123a1a3f04 storage: use fileutil's version of get_disk_stats() and get_available_space(), use mockery/fakery in tests, enable large share test on platforms with sparse files and if > 4 GiB of disk space is currently available 2010-09-10 08:36:29 -08:00
Zooko O'Whielacronx
d5e71c2940 fileutil: copy in the get_disk_stats() and get_available_space() functions from storage/server.py 2010-09-10 08:35:20 -08:00
david-sarah
9ceb527787 Change another doc reference from .txt to .rst in a text string. 2010-11-28 09:45:07 -08:00
Brian Warner
69b42c6cb7 remove --multiple/-m option from all CLI commands: closes #1262
I personally used "tahoe start/restart -m ../MY-TESTNET/node*" all the time,
to spin up or update a local testgrid while iterating over new code. However,
with the recent switch from "subprocess.Popen(/bin/twistd)" to "import and
call twistd.run()" in scripts/startstop_node.py (yay fewer processes!),
"start -m" broke, and fixing it requires os.fork, which is unavailable on
windows (boo windows!). And I was probably the only one using -m. So in the
interests of uniformity among platforms and simpler code (yay negative code
days!), we're just removing -m from everything. I will start using a little
shell script or something to simulate the removed functionality.

This patch also cleans up CLI-function calling a bit: get the basedir from
the config dict (instead of sometimes from a separate argument), and always
return a numeric exit code.
2010-11-27 00:38:09 -08:00
Brian Warner
f36bda2780 Revert previous commit: there's an ugly corner-case on windows that fails tests.
Specifically, test_runner.CreateNode.test_client failed, because the
os.fork-is-present test decided that --multiple should not be allowed on
windows, even though --multiple works just fine for 'tahoe create-client'.
The only restriction on --multiple is for 'tahoe start' and 'tahoe restart'.

This needs a different approach, probably by cleaning up BasedirMixin. We
should only be withholding --multiple on windows for "start" and
"restart". (we should continue withholding --multiple on all platforms for
"run").

This reverts (git) commit f3adb037ae:
 "startstop_node.py: fix "tahoe start -m" by forking before non-final targets"
2010-11-26 16:44:11 -08:00
Brian Warner
f3adb037ae startstop_node.py: fix "tahoe start -m" by forking before non-final targets
* don't advertise -m flag on tahoe start/restart/run unless os.fork is
  available (i.e. windows)
* test_runner.py: add test to exercise "start/stop/restart -m"
2010-11-26 14:45:38 -08:00
david-sarah
fdf6946669 test_cli.py: test that 'tahoe mv' reports errors from the DELETE operation. refs #1255 2010-11-11 17:46:53 -08:00
Zooko O'Whielacronx
93dc740f01 tests: bump up the timeout on test_dirnode.Dirnode from 240s to 480s since it apparently took longer than 240s just now on François's ARM buildslave 2010-11-15 01:21:19 -08:00
david-sarah
76371ae5a9 test_cli.py: fix a stale comment that incorrectly implied that test_cli runs CLI commands in subprocesses (it actually runs them using deferToThread). 2010-11-09 18:28:19 -08:00
david-sarah
01a5365051 test_system.py: test for 'tahoe mv' with the http_proxy and HTTP_PROXY variables set. refs #1253 2010-11-09 19:36:49 -08:00
david-sarah
edec9c9869 test_client.py: change overzealous use of failUnlessReallyEqual on integers back to failUnlessEqual. 2010-11-09 15:08:16 -08:00
david-sarah
56f4c1218b test_runner.py: fix test failure in test_the_right_code after applying zooko's change to test-with-fake-pkg.py 2010-11-01 20:59:05 -07:00
david-sarah
6eaa7f2356 tahoe backup: perform tilde expansion in exclude-from filename (version 2). fixes #1241 2010-10-30 20:52:31 -07:00
david-sarah
10a5f23df9 test_runner.py: fix error in BinTahoe.test_version_no_noise introduced by last patch. refs #1235 2010-10-29 15:11:23 -07:00
david-sarah
d249589667 test_runner.py: also allow 'from pkg_resources import load_entry_point' as noise. refs #1235. 2010-10-29 13:42:46 -07:00
david-sarah
8c95f88b4f test_runner.py: if the only noise is 'UserWarning: Unbuilt egg for setuptools', skip instead of failing the no_noise tests. This version avoids 'any' to be compatible with Python < 2.5. refs #1235. 2010-10-29 12:18:04 -07:00
david-sarah
4061258c85 make ResponseCache smarter to avoid memory leaks: don't record timestamps, use DataSpans to merge entries, and clear the cache when we see a new seqnum. refs #1045, #1229 2010-10-26 21:33:02 -07:00
Brian Warner
c88eb097d9 scripts/tahoe_backup.py: oops, fix missing import, thanks pyflakes
test_cli.py: hush minor pyflakes complaint
2010-10-29 02:42:23 -07:00
Francois Deppierraz
14ee763c54 tahoe_cp.py: Don't call urllib.quote with an Unicode argument, fix #1224
tahoe_backup.py: Fix another (potential) occurrence of calling urllib.quote()
with an Unicode parameter
2010-10-29 01:45:20 -07:00
Brian Warner
c18953c169 fix #1223, crash+inefficiency during repair due to read overrun
* repairer (really the uploader) reads beyond end of input file (Uploadable)
* new-downloader does not tolerate overreads
* uploader does lots of tiny reads (inefficient)

This fixes the last two. The uploader still does a single overread at the end
of the input file, but now that's ok so we can leave it in place. The
uploader now expects the Uploadable to behave like a normal disk
file (reading beyond EOF will return less data than was asked for), and now
the new-downloadable behaves that way.
2010-10-29 01:20:36 -07:00
david-sarah
59e80b2ec9 test_mutable.py: add test for ResponseCache memory leak. refs #1045, #1129 2010-10-24 12:34:09 -07:00
david-sarah
daf2b8314a test_encodingutil.py: test_argv_to_unicode modified the wrong encoding variable. fixes #1214 2010-10-22 20:58:10 -07:00
Kevan Carstensen
ec4f87a98c immutable/repairer.py: don't use the default happiness setting when repairing 2010-09-27 13:01:02 -07:00
david-sarah
6c4ba62c5d test_download.py: simplify and possibly fix test_lost_servers; refs #1197 2010-09-11 17:32:51 -07:00
david-sarah
b23a7fcc10 test_runner.py: fix spurious failures of test_path on case-insensitive filesystems, and restore the check that the version is not 'unknown'. 2010-09-11 15:19:13 -07:00
Zooko O'Whielacronx
fd17d63c95 test: make tests stop relying on pyutil version class accepting the string 'unknown' for its version, and make them forward-compatible with the future Python Rational Version Numbering standard 2010-09-10 08:41:35 -07:00
Zooko O'Whielacronx
56a3258ff7 immutable downloader: add a test specifically of whether the finder sometimes announces "no more shares ever" and then announces a newly found share
(The current code fails this test, ref #1191.)
2010-09-08 21:16:54 -07:00
Zooko O'Whielacronx
9d421cb022 tests: assign the storage servers to a fixed order which triggers a bug in new downloader every time this test is run (formerly this test would detect the bug in new-downloader only sporadically)
If you are investigating the bug in new-downloader, one way to investigate might be to change this ordering to a different fixed order (e.g. rotate by 4 instead of rotate by 5) and observe how the behavior of new-downloader differs in that case.
2010-09-03 21:15:15 -07:00
francois
485bfc0fd6 DownloadStatus: show active immutable downloads in Active Operations, Fix #1172 2010-09-02 03:17:28 -07:00
Zooko O'Whielacronx
cdc625e0f8 tests: bump up the allowed number of reads
Kyle's OpenBSD buildslave used 41 reads when doing this test. The fact that I'm blindly bumping this number up to match the observed behavior probably means this isn't a good criterion to be testing for anyway. But perhaps someone else (Brian) could investigate why that run on Kyle's OpenBSD box took four more reads than we expected, and whether the fact that it took 41 reads to do this operation is indicative of an actual problem.
2010-09-01 22:38:01 -07:00
Brian Warner
00e9e4e676 SegmentFetcher: use new diversity-seeking share-selection algorithm, and
deliver all shares at once instead of feeding them out one-at-a-time.

Also fix distribution of real-number-of-segments information: now all
CommonShares (not just the ones used for the first segment) get a
correctly-sized hashtree. Previously, the late ones might not, which would
make them crash and get dropped (causing the download to fail if the initial
set were insufficient, perhaps because one of their servers went away).

Update tests, add some TODO notes, improve variable names and comments.
Improve logging: add logparents, set more appropriate levels.
2010-08-31 18:37:02 -07:00
Brian Warner
c89a464510 Share: drop received data after each block finishes. Quick fix for the #1170 spans.py complexity bug. 2010-08-31 18:35:58 -07:00
francois
f026927f86 web: refactor rate computation, fixes #1166 2010-08-15 07:19:33 -07:00
Brian Warner
a3cf3331d2 DownloadStatus: put real numbers in progress/status rows, not placeholders.
Improve tests.
2010-08-09 23:06:03 -07:00
Brian Warner
dc1afc81bb web download-status: tolerate DYHBs that haven't retired yet. Fixes #1160.
Also add a better unit test for it.
2010-08-09 15:51:00 -07:00
david-sarah
fd9a2205de test_runner.py: correct and simplify normalization of package directory for case-insensitive filesystems. 2010-08-08 11:50:05 -07:00
david-sarah
f88e4f0df6 test_runner.py: make test_path work for test-from-installdir. 2010-08-08 10:13:40 -07:00
david-sarah
dc53d4f5ef test_runner.py: another try at calculating the rootdir correctly for test-from-egg and test-from-prefixdir. 2010-08-08 08:43:07 -07:00
david-sarah
39fa9a81d1 test_runner.py: calculate the location of bin/tahoe correctly for test-from-prefixdir (by copying code from misc/build_helpers/run_trial.py). Also fix the false-positive check for Unicode paths in test_the_right_code, which was causing skips that should have been failures. 2010-08-07 21:28:17 -07:00
david-sarah
a79ec1c6cd test_util.py: use SHA-256 from pycryptopp instead of MD5 from hashlib (for uses in which any hash will do), since hashlib was only added to the stdlib in Python 2.5. 2010-08-05 22:00:51 -07:00
david-sarah
600cb95fe2 test_runner.py: increase timeout to cater for Francois' ARM buildslave. 2010-08-05 21:26:01 -07:00
david-sarah
8e90255349 test_util.py: remove use of 'a if p else b' syntax that requires Python 2.5. 2010-08-05 21:16:16 -07:00
Brian Warner
a0124e95ee test_download.DownloadTest.test_simultaneous_goodguess: enable some disabled
checks that used to fail but work now.
2010-08-05 11:55:07 -07:00
Brian Warner
f6f9a97627 DownloadNode: fix lost-progress in fetch_failed, tolerate cancel when no segment-fetch is active. Fixes #1154.
The lost-progress bug occurred when two simultanous read() calls fetched
different segments, and the first one failed (due to corruption, or the other
bugs in #1154): the second read() would never complete. While in this state,
cancelling the second read by having its consumer call stopProducing) would
trigger the cancel-intolerance bug. Finally, in downloader.node.Cancel,
prevent late cancels by adding an 'active' flag
2010-08-05 11:55:07 -07:00
Brian Warner
43c5032105 util/spans.py: __nonzero__ cannot return a long either. for #1154 2010-08-05 11:55:07 -07:00
david-sarah
cd8d41584e test_storage.py: change skip note for test_large_share to say that Windows doesn't support sparse files. refs #569 2010-08-04 19:26:12 -07:00
Brian Warner
8844655705 One fix for bug #1154: webapi GETs with a 'Range' header broke new-downloader.
The Range header causes n.read() to be called with an offset= of type 'long',
which eventually got used in a Spans/DataSpans object's __len__ method.
Apparently python doesn't permit __len__() to return longs, only ints.
Rewrote Spans/DataSpans to use s.len() instead of len(s) aka s.__len__() .
Added a test in test_download. Note that test_web didn't catch this because
it uses mock FileNodes for speed: it's probably time to rewrite that.

There is still an unresolved error-recovery problem in #1154, so I'm not
closing the ticket quite yet.
2010-08-04 11:45:49 -07:00
Brian Warner
2bd8749849 test_download: minor cleanup 2010-08-04 10:55:55 -07:00
Brian Warner
2a05aa2d91 lazily create DownloadNode upon first read()/get_segment() 2010-08-04 00:28:08 -07:00
Brian Warner
abcd6e0e96 test_hung_server: update comments, remove dead "stage_4_d" code 2010-08-04 00:28:00 -07:00
Brian Warner
919938dd95 copy the rest of David-Sarah's changes to make my tree match 1.8.0beta 2010-08-04 00:27:52 -07:00
Brian Warner
20847dd876 ShareFinder: add 10s OVERDUE timer, send new requests to replace overdue ones
The fixed 10-second timer will eventually be replaced with a per-server
value, calculated based on observed response times.

test_hung_server.py: enhance to exercise DYHB=OVERDUE state. Split existing
mutable+immutable tests into two pieces for clarity. Reenabled several tests.
Deleted the now-obsolete "test_failover_during_stage_4".
2010-08-04 00:27:41 -07:00
Brian Warner
63b61ce7bd Rewrite immutable downloader (#798). This patch adds and updates unit tests. 2010-08-04 00:27:10 -07:00
Brian Warner
cbcb728e7e Add a byte-spans utility class, like perl's Set::IntSpan for .newsrc files.
Also a data-spans class, which records a byte (instead of a bit) for each
index.
2010-08-04 00:26:00 -07:00
david-sarah
f10f9a8646 test_storage.py: avoid spurious test failure by accepting either 'Next crawl in 59 minutes' or 'Next crawl in 60 minutes'. fixes #1140 2010-08-03 03:20:58 -07:00
david-sarah
a7c474a098 CLI: further improve consistency of basedir options and add tests. addresses #118 2010-08-03 01:54:16 -07:00
david-sarah
51eeea900f test_cli.py: make 'tahoe mkdir' tests slightly less dumb (check for 'URI:' in the output). 2010-08-03 01:47:20 -07:00
david-sarah
b9174b5fe7 test_cli.py: use u-escapes instead of UTF-8. 2010-08-03 01:35:38 -07:00
david-sarah
d81e8acf88 test_runner.py: add test_run_with_python_options, which checks that the Windows script changes haven't broken 'python <options> bin/tahoe'. 2010-08-01 23:25:58 -07:00
david-sarah
3ddb4a9bbf test_runner.py: fix missing import of get_filesystem_encoding 2010-08-01 23:09:02 -07:00
david-sarah
3af6f19cb0 misc/build_helpers/run_trial.py: check that the root from which the module we are testing was loaded is the current directory. This version of the patch folds in later fixes to the logic for caculating the directories to compare, and improvements to error messages. addresses #1137 2010-08-01 21:55:35 -07:00
david-sarah
1fad717aaa test_runner.py: change to code for locating the bin/tahoe script that was missed when rebasing the patch for #1074. 2010-07-25 11:20:08 -07:00
david-sarah
37b07a545f Changes to Tahoe needed to work with new zetuptoolz (that does not use .exe wrappers on Windows), and to support Unicode arguments and stdout/stderr -- v5 2010-07-25 01:32:16 -07:00
david-sarah
f952532f08 Basedir/node directory option improvements for ticket798 branch. addresses #188, #706, #715, #772, #890 2010-08-01 21:30:04 -07:00
david-sarah
9aae9208e3 test_runner.py: Fix error in message arguments to 'fail' calls. 2010-08-01 18:35:26 -07:00
david-sarah
2b33704d61 test_system: correct a failure in _test_runner caused by Unicode basedir patch on non-Unicode platforms. 2010-07-23 20:21:23 -07:00
david-sarah
f036dfaa4b Fix test failures due to Unicode basedir patches. 2010-07-24 18:03:18 -07:00
david-sarah
28e6ad51a7 util.encodingutil: change quote_output to do less unnecessary escaping, and to use double-quotes more consistently when needed. This version avoids u-escaping for characters that are representable in the output encoding, when double quotes are used, and includes tests. fixes #1135 2010-07-23 00:53:14 -07:00
david-sarah
618db4867c Replace uses of os.path.abspath with abspath_expanduser_unicode where necessary. This makes basedir paths consistently represented as Unicode. 2010-07-21 17:14:18 -07:00
david-sarah
11b18824c7 util.fileutil, test.test_util: add abspath_expanduser_unicode function, to work around <http://bugs.python.org/issue3426>. util.encodingutil: add a convenience function argv_to_abspath. 2010-07-21 16:15:07 -07:00
jacob.lyles
e276050d5a abbreviate time edge case python2.5 unit test 2010-07-29 14:06:38 -07:00
david-sarah
23c649c897 test_upload.py: rename test_problem_layout_ticket1124 to test_problem_layout_ticket_1124 -- fix .todo reference. 2010-07-29 08:29:27 -07:00
david-sarah
0ebe18cc73 test_upload.py: rename test_problem_layout_ticket1124 to test_problem_layout_ticket_1124 for consistency. 2010-07-29 07:22:50 -07:00
Zooko O'Whielacronx
28822dfb4e tests: remove debug print 2010-08-01 23:33:39 -07:00
Zooko O'Whielacronx
fd874d6669 setup: fix a couple instances of hard-coded 'allmydata-tahoe' in the scripts, tighten the tests (as suggested by David-Sarah) 2010-08-01 09:42:07 -07:00
Zooko O'Whielacronx
44edecb36c setup: replace hardcoded 'allmydata-tahoe' with allmydata.__appname__ 2010-08-01 09:05:17 -07:00
Zooko O'Whielacronx
0f94923f22 immutable: use PrefixingLogMixin to organize logging in Tahoe2PeerSelector and add more detailed messages about peer 2010-07-19 01:20:00 -07:00
Zooko O'Whielacronx
2ebe2c2ff3 benchmarking: update bench_dirnode to be correct and use the shiniest new pyutil.benchutil features concerning what units you measure in 2010-07-18 21:49:48 -07:00
Zooko O'Whielacronx
9b4f412e49 trivial: rename and add in-line doc to clarify "used_peers" => "upload_servers" 2010-07-18 21:47:44 -07:00
david-sarah
bdc6fd2f8b test_runner: test that 'tahoe --version' outputs no noise (e.g. DeprecationWarnings). 2010-07-19 18:13:45 -07:00
Zooko O'Whielacronx
c076c0b19b immutable: add test case of #1128, fix test case of #1118 2010-07-19 01:16:12 -07:00
Zooko O'Whielacronx
1dbfcf753d immutable: extend the tests to check that the shares that got uploaded really do make a sufficiently Happy distribution
This patch also renames some instances of "find_shares()" to "find_all_shares()" and other instances to "find_uri_shares()" as appropriate -- the conflation between those names confused me at first when writing these tests.
2010-07-18 21:50:47 -07:00
Zooko O'Whielacronx
2e83f243c2 immutable: test for #1118 2010-07-18 15:15:37 -07:00
Zooko O'Whielacronx
0f46766a51 immutable: test for #1124 2010-07-18 15:29:07 -07:00
Zooko O'Whielacronx
8f201c75f0 trivial: fix unused import (sorry about that, pyflakes) 2010-07-18 14:51:33 -07:00
Zooko O'Whielacronx
16bb529339 tests, NEWS, CREDITS re: #1117
Give Brian and Kevan promotions, move release date in NEWS to the 18th, commit Brian's test for #1117.
fixes #1117
2010-07-18 13:32:25 -07:00
Kevan Carstensen
054c4553b5 test/test_upload.py: test to see that aborted buckets are ignored by the storage server 2010-07-15 17:10:46 -07:00
Kevan Carstensen
677f3b1f63 test/test_storage.py: test for the new remote_abort semantics. 2010-07-15 16:21:48 -07:00
Kevan Carstensen
5becaf0309 test/test_upload.py: changes to test plumbing for #1117 tests
- Add a callRemoteOnly method to FakeBucketWriter.
    - Change the abort method in FakeBucketWriter to not return a
      RuntimeError.
2010-07-15 16:18:20 -07:00
david-sarah
638a3b2af4 test_encodingutil: correct an error in the previous patch to StdlibUnicode.test_open_representable. 2010-07-18 08:14:20 -07:00
david-sarah
74c41ebb8b test_dirnode and test_web: don't use failUnlessReallyEqual in cases where the return type from simplejson.loads can vary between unicode and str. Use to_str when comparing URIs parsed from JSON. 2010-07-18 07:29:15 -07:00
david-sarah
9a4a864573 test_encodingutil: StdlibUnicode.test_open_representable no longer uses a mock. 2010-07-18 05:54:12 -07:00
Zooko O'Whielacronx
7cc98759bd docs: add comment clarifying #1051 2010-07-17 22:32:50 -07:00
david-sarah
d346e0853d Add tests of caps from the future that have non-ASCII characters in them (encoded as UTF-8). The changes to test_uri.py, test_client.py, and test_dirnode.py add tests of non-ASCII future caps in addition to the current tests. The changes to test_web.py just replace the tests of all-ASCII future caps with tests of non-ASCII future caps. We also change uses of failUnlessEqual to failUnlessReallyEqual, in order to catch cases where the type of a string is not as expected. 2010-07-11 13:02:52 -07:00
Zooko O'Whielacronx
3c4fe0706f M-x whitespace-cleanup 2010-07-17 20:27:39 -07:00
Zooko O'Whielacronx
6a7023efbe benchmarking: update bench_dirnode.py to reflect the new directory interfaces 2010-07-17 20:17:10 -07:00
david-sarah
bdb10553eb test_encodingutil: fix test_open_representable, which is only valid when run on a platform for which we know an unrepresentable filename. 2010-07-17 20:03:33 -07:00
david-sarah
a8161c915a Correct stringutils->encodingutil patch to be the newer version, rather than the old version that was committed in error. 2010-07-17 18:34:35 -07:00
david-sarah
0d79a4a7d1 test_cli.py: fix error that crept in when rebasing the patch for #1072. 2010-07-17 17:01:23 -07:00
david-sarah
c59f9b162f stringutils: add test for when sys.stdout has no encoding attribute (fixes #1099). 2010-07-16 21:58:16 -07:00
david-sarah
8090785716 CLI: add 'tahoe unlink' as an alias to 'tahoe rm', for forward-compatibility. 2010-07-17 15:04:11 -07:00
Zooko O'Whielacronx
6e8477114e minor code clean-up in dirnode.py
Impose micro-POLA by passing only the writekey instead of the whole node object to {{{_encrypt_rw_uri()}}}. Remove DummyImmutableFileNode in nodemaker.py, which is obviated by this. Add micro-optimization by precomputing the netstring of the empty string and branching on whether the writekey is present or not outside of {{{_encrypt_rw_uri()}}}. Add doc about writekey to docstring.
fixes #967
2010-07-13 23:02:55 -07:00
david-sarah
11077ea74d Rename stringutils to encodingutil, and drop listdir_unicode and open_unicode (since the Python stdlib functions work fine with Unicode paths). Also move some utility functions to fileutil. 2010-07-11 17:30:15 -07:00
david-sarah
3b1b0147a8 test_storage.py: potential fix for failures when logging is enabled. 1970-07-12 21:05:46 -07:00
freestorm77
97579af73c runnin_test_tahoe_css.dpatch
Runnin test for changes in tahoe.css file
2010-06-22 14:47:14 -07:00
freestorm77
0304918a77 runnin_test_directory_xhtml.dpatch
Runnin test for diretory.xhtml top banner
2010-06-22 13:14:03 -07:00
david-sarah
6b2f99fa9a test_backupdb.py: skip test_unicode if we can't represent the test filenames. 2010-06-18 19:26:20 -07:00
david-sarah
a2caf6f4b2 test_web.py: correct a test that was missed in the change to not write ctime/mtime. 2010-06-18 19:17:18 -07:00
david-sarah
4712875193 dirnode.py: stop writing 'ctime' and 'mtime' fields. Includes documentation and test changes. 2010-06-18 16:01:19 -07:00
david-sarah
72e395d878 test_storage.py: print more information on test failures. 2010-06-16 20:46:23 -07:00
david-sarah
390fc78a9a test_cli.py: fix test failure in CLI.test_listdir_unicode_good due to filenames returned from listdir_unicode no longer being normalized. 2010-06-17 21:51:10 -07:00
david-sarah
178401eb4e tahoe backup: unicode tests. 2010-06-17 20:52:11 -07:00
david-sarah
a9fe3792de dirnodes: fix normalization hole where childnames in directories created by nodemaker.create_mutable/immutable_directory would not be normalized. Add a test that we normalize names coming out of a directory. 2010-06-17 17:02:49 -07:00
david-sarah
5ada31034b test_stringutils.py: take account of the output of listdir_unicode no longer being normalized. Also use Unicode escapes, not UTF-8. 2010-06-16 20:44:09 -07:00
david-sarah
025aede9e4 test_dirnode.py: partial tests for normalization changes. 2010-06-16 20:40:25 -07:00
david-sarah
408b12a7e6 test_system.py: investigate failure in allmydata.test.test_system.SystemTest.test_upload_and_download_random_key due to bytes_sent not being an int 2010-06-15 17:16:48 -07:00
david-sarah
11e88c340a SFTP: test that renaming onto a just-opened file fails. 2010-06-11 20:37:09 -07:00
david-sarah
b67f8b66c8 SFTP: further small improvements to test coverage. Also ensure that after a test failure, later tests don't fail spuriously due to the checks for heisenfile leaks. 2010-06-11 20:07:37 -07:00
david-sarah
b4e074d250 SFTP: further improve test coverage (paths containing '.', bad data for posix-rename extension, and error in test of openShell). 2010-06-11 14:31:42 -07:00
david-sarah
52f87904ed SFTP: improve test coverage for no-write on mutable files, and check for heisenfile table leaks in all relevant tests. Delete test_memory_leak since it is now redundant. 2010-06-11 13:57:52 -07:00
david-sarah
546c3d2ed4 SFTP: add test for extension of file opened with FXF_APPEND. 2010-06-10 11:26:47 -07:00
Zooko O'Whielacronx
f19c240ca8 tests: bump up the timeout on this iputil test from 2s to 4s 2010-06-09 07:30:17 -07:00
david-sarah
3c44389440 SFTP: fix most significant memory leak described in #1045 (due to a file being added to all_heisenfiles under more than one direntry when renamed). 2010-06-09 01:00:03 -07:00
david-sarah
5786874d38 test_stringutils.py: Fix test failure on CentOS builder, possibly Python 2.4.3-related. 2010-06-08 23:50:56 -07:00
david-sarah
ffd1c5aca5 Fix for Unicode-related test failures on Zooko's OS X 10.6 machine. 2010-06-08 22:54:48 -07:00
david-sarah
32d9deace3 Back out Windows-specific Unicode argument support for v1.7. 2010-06-08 17:08:03 -07:00
david-sarah
195b4afa8f SFTP: ignore permissions when opening a file (needed for sshfs interoperability). 2010-06-07 22:57:00 -07:00
david-sarah
a67e745b26 test_web.py: fix pyflakes warnings introduced by byterange patch. 2010-06-07 21:20:12 -07:00
Jeremy Fitzhardinge
63b28d707b Improve HTTP/1.1 byterange handling
Fix parsing of a Range: header to support:
 - multiple ranges (parsed, but not returned)
 - suffix byte ranges ("-2139")
 - correct handling of incorrectly formatted range headers
   (correct behaviour is to ignore the header and return the full
    file)
 - return appropriate error for ranges outside the file

Multiple ranges are parsed, but only the first range is returned.
Returning multiple ranges requires using the multipart/byterange
content type.
2010-03-09 20:59:13 -07:00
Zooko O'Whielacronx
d13784b649 tests: bump up the timeout on these tests; MM's buildslave is sometimes extremely slow on tests, but it will complete them if given enough time. MM is working on making that buildslave more predictable in how long it takes to run tests. 2010-06-07 20:37:54 -07:00
david-sarah
3c883e6e44 test_cli.py: remove invalid 'test_listdir_unicode_bad' test. 2010-06-07 11:37:30 -07:00
david-sarah
71f8757bd1 check_memory.py: adapt to servers-of-happiness changes. 2010-06-07 18:35:28 -07:00
david-sarah
7092de1b6f Remove the 'tahoe debug consolidate' subcommand. 2010-06-07 11:37:57 -07:00
Zooko O'Whielacronx
faade8b20a tests: drastically increase timeout of this very time-consuming test in honor of François's ARM box 2010-06-07 04:59:29 -07:00
david-sarah
8b014372b1 test_stringutils.py: trivial error in exception message for skipped test. 2010-06-06 23:14:55 -07:00
david-sarah
529c9f673a More Unicode test fixes. 2010-06-06 22:33:58 -07:00
david-sarah
65b6f4e3ce Unicode fixes for platforms with non-native-Unicode filesystems. 2010-06-06 21:32:38 -07:00
david-sarah
80252debcd Unicode fixes. 2010-06-06 18:02:15 -07:00
Zooko O'Whielacronx
5bcca5151e unicode tests: fix missing import 2010-06-04 07:26:30 -07:00
Zooko O'Whielacronx
db8a6f3aa6 unicode: make test_cli test a non-ascii argument, and make the fallback term encoding be locale.getpreferredencoding() 2010-06-04 07:12:51 -07:00
Zooko O'Whielacronx
442008a690 unicode: always decode json manifest as utf-8 then encode for stdout
pyflakes pointed out that the exception handler fallback called an un-imported function, showing that the fallback wasn't being exercised.
I'm not 100% sure that this patch is right and would appreciate François or someone reviewing it.
2010-06-04 01:48:40 -07:00
Zooko O'Whielacronx
952f3bf3ef fix flakes 2010-06-04 00:58:45 -07:00
Francois Deppierraz
0eb4d83937 test_stringutils.py: Skip test test_listdir_unicode_good if filesystem supports only ASCII filenames 2010-05-21 09:08:39 -07:00
Francois Deppierraz
01da7f06e6 test_stringutils.py: Skip test_listdir_unicode on mocked platform which cannot store non-ASCII filenames 2010-05-21 09:05:59 -07:00
Francois Deppierraz
58e9df9f5a test_stringutils.py: Add a test class for OpenBSD 4.1 with LANG=C 2010-05-21 07:00:53 -07:00
Francois Deppierraz
1aeff62161 test_stringutils.py: Mock the open() call in test_open_unicode
This test ensure that open(a_unicode_string) is used on Unicode platforms
(Windows or MacOS X) and that open(a_correctly_encoded_bytestring) on other
platforms such as Unix.
2010-05-21 06:58:17 -07:00
Francois Deppierraz
eecaaca6b7 test_stringutils.py: Fix a trivial Python 2.4 syntax incompatibility 2010-05-21 02:33:45 -07:00
Francois Deppierraz
6c70fb7b3c test_cli.py: Fix tests when sys.stdout.encoding=None and refactor this code into functions 2010-05-20 01:44:47 -07:00
Francois Deppierraz
496802420e Fix handling of correctly encoded unicode filenames (#534)
Tahoe CLI commands working on local files, for instance 'tahoe cp' or 'tahoe
backup', have been improved to correctly handle filenames containing non-ASCII
characters.
  
In the case where Tahoe encounters a filename which cannot be decoded using the
system encoding, an error will be returned and the operation will fail.  Under
Linux, this typically happens when the filesystem contains filenames encoded
with another encoding, for instance latin1, than the system locale, for
instance UTF-8.  In such case, you'll need to fix your system with tools such
as 'convmv' before using Tahoe CLI.
  
All CLI commands have been improved to support non-ASCII parameters such as
filenames and aliases on all supported Operating Systems except Windows as of
now.
2010-05-19 17:43:56 -07:00
Francois Deppierraz
d0ed14e1bb stringutils.py: Unicode helper functions + associated tests
This file contains a bunch of helper functions which converts
unicode string from and to argv, filenames and stdout.
2010-05-19 17:41:05 -07:00
david-sarah
c9984c40e9 SFTP: further improvements to test coverage. 2010-06-02 16:44:22 -07:00
david-sarah
2a791b0d05 SFTP: improve test coverage. Also make creating a directory fail when permissions are read-only (rather than ignoring the permissions). 2010-06-01 21:19:34 -07:00
david-sarah
29a06457d2 dirnode.py: fix a bug in the no-write change for Adder, and improve test coverage. Add a 'metadata' argument to create_subdirectory, with documentation. Also update some comments in test_dirnode.py made stale by the ctime/mtime change. 2010-06-01 20:26:41 -07:00
david-sarah
de95140b7b SFTP: changes for #1063 ('no-write' field) including comment:1 (clearing owner write permission diminishes to a read cap). Includes documentation changes, but not tests for the new behaviour. 2010-05-31 22:11:39 -07:00
david-sarah
e867985539 SFTP: cater to clients that assume a file is created as soon as they have made an open request; also, fix some race conditions associated with closing a file at about the same time as renaming or removing it. 2010-05-28 21:52:53 -07:00
david-sarah
a143b1297b SFTP: handle removing a file while it is open. Also some simplifications of the logout handling. 2010-05-25 11:42:10 -07:00
david-sarah
e81ce2bbd2 SFTP: a posix-rename response should actually return an FXP_STATUS reply, not an FXP_EXTENDED_REPLY as Twisted Conch assumes. Work around this by raising an SFTPError with code FX_OK. 2010-05-24 20:33:23 -07:00
david-sarah
8f86490d48 SFTP: fix problem with posix-rename code returning a Deferred for the renamed filenode, not for the result of the request (an empty string). 2010-05-24 19:02:09 -07:00
david-sarah
549dccd608 SFTP: name of the POSIX rename extension should be 'posix-rename@openssh.com', not 'extposix-rename@openssh.com'. 2010-05-23 19:11:56 -07:00
david-sarah
75357fd2d5 SFTP: fix pyflakes warnings; drop 'noisy' versions of eventually_callback and eventually_errback; robustify conversion of exception messages to UTF-8. 2010-05-23 07:09:05 -07:00
david-sarah
38964fb35a SFTP: fixes and test cases for renaming of open files. 2010-05-22 20:25:49 -07:00
david-sarah
42ecd77f4e SFTP: Increase test_sftp timeout to cater for francois' ARM buildslave. 2010-05-22 12:16:39 -07:00
david-sarah
5974773969 SFTP: Fix error in support for getAttrs on an open file, to index open files by directory entry rather than path. Extend that support to renaming open files. Also, implement the extposix-rename@openssh.org extension, and some other minor refactoring. 2010-05-21 20:58:36 -07:00
david-sarah
5545a678bf SFTP tests: fix test_openDirectory_and_attrs that was failing in timezones west of UTC. 2010-05-20 11:10:27 -07:00
david-sarah
ce3872d10e SFTP: allow getAttrs to succeed on a file that has been opened for creation but not yet uploaded or linked (part of #1050). 2010-05-19 20:56:13 -07:00
david-sarah
7f9e351bce SFTP: improve logging so that results of requests are (usually) logged. 2010-05-19 17:36:52 -07:00
david-sarah
a62733ed08 SFTP: add tests for more combinations of open flags. 2010-05-18 22:39:33 -07:00
david-sarah
f0945526ce SFTP: allow FXF_WRITE | FXF_TRUNC (#1050). 2010-05-18 21:32:40 -07:00
david-sarah
819eaa74c1 SFTP: fixes related to reporting of permissions (needed for sshfs). 2010-05-17 22:45:21 -07:00
david-sarah
8d118e28f1 SFTP: change error code returned for ExistingChildError to FX_FAILURE (fixes gvfs with some picky programs such as gedit). 2010-05-17 17:42:05 -07:00
david-sarah
57699fd1eb SFTP: fixed bugs that caused hangs during write (#1037). 2010-05-16 21:42:28 -07:00
david-sarah
5f9c10901b SFTP: work around a probable bug in twisted.conch.ssh.session:loseConnection(). Also some minor error handling cleanups. 2010-05-16 18:26:06 -07:00
david-sarah
6a2b0229f0 SFTP: Support statvfs extensions, avoid logging actual data, and decline shell sessions politely. 2010-05-16 08:43:47 -07:00
david-sarah
31994aaf74 SFTP: fix error in SFTPUserHandler arguments introduced by execCommand patch. 2010-05-15 18:40:45 -07:00
david-sarah
8028bf01bc SFTP: implement execCommand to interoperate with clients that issue a 'df -P -k /' command. Also eliminate use of Zope adaptation. 2010-05-15 18:27:54 -07:00
david-sarah
14d0721246 Update SFTP implementation and tests: fix #1038 and switch to foolscap logging; also some code reorganization. 2010-05-13 21:31:13 -07:00
david-sarah
5fdd63d0ad Tests for new SFTP implementation 2010-05-11 23:05:52 -07:00
david-sarah
4556702044 Fix test failures in test_web caused by changes to web page titles in #1062. Also, change a 'target' field to '_blank' instead of 'blank' in welcome.xhtml. 2010-06-03 16:21:05 -07:00
david-sarah
4f58355a0e dirnode.py: Fix bug that caused 'tahoe' fields, 'ctime' and 'mtime' not to be updated when new metadata is present. 2010-06-01 18:46:44 -07:00
david-sarah
53f7d2c7fe dirnode.py: Fix #1034 (MetadataSetter does not enforce restriction on setting 'tahoe' subkeys), and expose the metadata updater for use by SFTP. Also, support diminishing a child cap to read-only if 'no-write' is set in the metadata. 2010-05-31 21:54:28 -07:00
david-sarah
6d91b1e00e Trivial whitespace changes. 2010-05-27 12:41:14 -07:00
david-sarah
e706f043f0 Change shouldFail to avoid Unicode errors when converting Failure to str 2010-05-11 23:07:54 -07:00
david-sarah
90e9fe28af Resolve conflict in patch to change imports to absolute. 2010-06-02 22:46:08 -07:00
Zooko O'Whielacronx
9e2da05837 tests: refactor test_web.py to have less duplication of literal caps-from-the-future
This is a prelude to a patch which will add tests of caps from the future which have non-ascii chars in them.
2010-05-18 22:51:46 -07:00
david-sarah
a80f19a084 Correct harmless indentation errors found by pylint 2010-02-25 23:21:51 -07:00
david-sarah
e76092e16c Change relative imports to absolute 2010-02-26 01:14:33 -07:00
Zooko O'Whielacronx
d4f6d4092c test: show stdout and stderr in case of non-zero exit code from "tahoe" command 2010-05-24 00:33:48 -07:00
Zooko O'Whielacronx
294c0a6279 tests: fix pyflakes warnings in bench_dirnode.py 2010-05-21 13:25:11 -07:00
Zooko O'Whielacronx
35efdf04ca tests: don't let bench_dirnode.py do stuff and have side-effects at import time (unless __name__ == '__main__') 2010-05-21 05:20:52 -07:00
Zooko O'Whielacronx
a48979516e tests: increase timeout to give François's ARM buildslave a chance to complete the tests 2010-05-20 06:45:26 -07:00
Kevan Carstensen
d0706d27ea Hush pyflakes warnings 2010-05-15 11:43:44 -07:00
Kevan Carstensen
9179dd099a Improve code coverage of the Tahoe2PeerSelector tests. 2010-05-14 20:29:13 -07:00
Kevan Carstensen
4e29060847 Change stray "shares_of_happiness" to "servers_of_happiness" 2009-11-16 15:24:59 -07:00
Kevan Carstensen
5683112a02 Revisions of the #778 tests, per reviewers' comments
- Fix comments and confusing naming.
- Add tests for the new error messages suggested by David-Sarah
  and Zooko.
- Alter existing tests for new error messages.
- Make sure that the tests continue to work with the trunk.
- Add a test for a mutual disjointedness assertion that I added to
  upload.servers_of_happiness.
- Fix the comments to correctly reflect read-onlyness
- Add a test for an edge case in should_add_server
- Add an assertion to make sure that share redistribution works as it 
  should
- Alter tests to work with revised servers_of_happiness semantics
- Remove tests for should_add_server, since that function no longer exists.
- Alter tests to know about merge_peers, and to use it before calling 
  servers_of_happiness.
- Add tests for merge_peers.
- Add Zooko's puzzles to the tests.
- Edit encoding tests to expect the new kind of failure message.
- Edit tests to expect error messages with the word "only" moved as far
  to the right as possible.
- Extended and cleaned up some helper functions.
- Changed some tests to call more appropriate helper functions.
- Added a test for the failing redistribution algorithm
- Added a test for the progress message
- Added a test for the upper bound on readonly peer share discovery.
2010-05-13 18:25:42 -07:00
Kevan Carstensen
d360ee0d2e Alter various unit tests to work with the new happy behavior 2010-01-07 12:13:25 -07:00
Kevan Carstensen
953c9555ca Replace "UploadHappinessError" with "UploadUnhappinessError" in tests. 2009-12-04 22:34:53 -07:00
Kevan Carstensen
c0f3dbb919 Add tests for the behavior described in #834. 2009-11-22 19:20:08 -07:00
Kevan Carstensen
c3b11dedea Re-work 'test_upload.py' to be more readable; add more tests for #778 2009-11-16 13:23:34 -07:00
Kevan Carstensen
9590690450 Test Tahoe2PeerSelector to make sure that it recognizeses existing shares on readonly servers 2009-11-08 18:37:35 -07:00
Kevan Carstensen
324a72e4a3 Add more tests for comment:53 in ticket #778 2009-11-04 05:28:49 -07:00
Kevan Carstensen
697f1cfae8 Add a test for upload.shares_by_server 2009-11-04 05:13:24 -07:00
Kevan Carstensen
0d8e70bfe5 Minor tweak to an existing test -- make the first server read-write, instead of read-only 2009-11-03 21:42:32 -07:00
Kevan Carstensen
5ad29d4e8b Alter tests to use the new form of set_shareholders 2009-11-03 21:36:02 -07:00
"Kevan Carstensen"
ee9690b357 Refactor some behavior into a mixin, and add tests for the behavior described in #778 2009-10-30 02:19:08 -07:00
Kevan Carstensen
362f204075 Alter NoNetworkGrid to allow the creation of readonly servers for testing purposes. 2009-10-17 18:30:13 -07:00
Zooko O'Whielacronx
6037df55b3 web: add a simple test that you can load directory.xhtml 2010-05-09 23:37:29 -07:00
Francois Deppierraz
dd95e8e71b CLI: Support for https url in option --node-url
This patch modifies the regular expression used for verifying of '--node-url'
parameter.  Support for accessing a Tahoe gateway over HTTPS was already
present, thanks to Python's urllib.
2010-04-30 11:56:09 -07:00
Brian Warner
f6f8256b35 "tahoe webopen": add --info flag, to get ?t=info
Also fix some trailing whitespace.
2010-04-24 16:30:03 -07:00
david-sarah
a2ed17f2a0 Additional test for DIR2-LIT directories in test_web.py, fixed version (#948) 2010-02-24 20:18:24 -08:00
david-sarah
e6aee33bb7 directories: add DIR2-LIT directories to test_deepcheck.py (#948) 2010-02-23 23:54:33 -08:00
david-sarah
187d837c1d dirnode: add tests of literal dirnodes (current and fix for #948) 2010-02-23 20:33:45 -08:00
david-sarah
fec9185f2a Additional fix for abbrev_si, with test 2010-02-21 19:36:52 -08:00
david-sarah
d29ec184a6 Additions to test_web.py for #948 2010-02-21 18:53:52 -08:00
david-sarah
973f0afdd3 Change direct accesses to an_uri.storage_index to calls to .get_storage_index() (fixes #948) 2010-02-21 18:45:04 -08:00
david-sarah
9741b9655f Test behaviour of 'tahoe ls' for unknown objects (#837) 2010-02-23 18:59:13 -08:00
Brian Warner
1de4d2c594 test_util.py: improve coverage of util.time_format 2010-02-23 23:46:37 -05:00
Kevan Carstensen
81ad52d6eb Change OphandleTable to use a deterministic clock, so we can test it
To test the changes for #577, we need a deterministic way to simulate
the passage of long periods of time. twisted.internet.task.Clock seems,
from my Googling, to be the way to go for this functionality. I changed
a few things so that OphandleTable would use twisted.internet.task.Clock
when testing:

  * WebishServer.__init___ now takes an optional 'clock' parameter,
  * which it passes to the root.Root instance it creates.
  * root.Root.__init__ now takes an optional 'clock' parameter, which it
    passes to the OphandleTable.__init__ method.
  * OphandleTable.__init__ now takes an optional 'clock' parameter. If
    it is provided, and it isn't None, its callLater method will be used
    to schedule ophandle expirations (as opposed to using
    reactor.callLater, which is what OphandleTable does normally).
  * The WebMixin object in test_web.py now sets a self.clock parameter,
    which is a twisted.internet.task.Clock that it feeds to the 
    WebishServer it creates. 

Tests using the WebMixin can control the passage of time in
OphandleTable by accessing self.clock.
2010-02-20 13:07:13 -08:00
Kevan Carstensen
bcdc78bd44 Add tests for the ophandle expiration behavior in #577 2010-02-20 17:04:55 -08:00
david-sarah
85a50feeaa More cleanups to test_cli using new utilities for reading and writing files. 2010-02-05 17:38:55 -08:00
david-sarah
c984a09fe7 Fix race conditions and missing callback in allmydata.test.test_cli.Cp.test_copy_using_filecap, add utilities for one-liner reading and writing of files, and fix cases in test_cli where files were not being closed after writing. 2010-02-05 17:37:27 -08:00
Kevan Carstensen
c00a62a2e7 Add tests for #939 2010-02-11 22:21:37 -08:00
Brian Warner
4040b1000b web/storage.py: display total-seen on the last-complete-cycle line. For #940. 2010-02-07 16:20:10 -08:00
Brian Warner
880f824103 code coverage: replace figleaf with coverage.py, should work on py2.6 now.
It still lacks the right HTML report (the builtin report is very pretty, but
lacks the "lines uncovered" numbers that I want), and the half-finished
delta-from-last-run measurements.
2010-02-03 08:54:21 -08:00
Zooko O'Whielacronx
3e4342ecb3 immutable: downloader accepts notifications of buckets even if those notifications arrive after he has begun downloading shares.
This can be useful if one of the ones that he has already begun downloading fails. See #287 for discussion. This fixes part of #287 which part was a regression caused by #928, namely this fixes fail-over in case a share is corrupted (or the server returns an error or disconnects). This does not fix the related issue mentioned in #287 if a server hangs and doesn't reply to requests for blocks.
2010-01-31 22:16:10 -08:00
Zooko O'Whielacronx
e4e2599017 tests: don't require tahoe to run with no noise if we are using an old twisted that emits DeprecationWarnings 2010-01-31 21:23:23 -08:00
Zooko O'Whielacronx
7d32fafc44 tests: stop being surprised that Nevow no longer prints out warnings when it tries to find its static files
Unless we are using a sufficiently new version of Nevow, in which case if it prints out warnings then this is a hard test failure. :-)
2010-01-31 20:11:44 -08:00
david-sarah
6215ebd95e cli: suppress DeprecationWarnings emitted from importing nevow and twisted. Fixes #859 2010-01-31 16:44:29 -08:00
david-sarah
ea3954372a Improvements to test_unknownnode to cover invalid cap URIs with known prefixes 2010-01-29 22:39:08 -08:00
david-sarah
37a242e01a Improvements to test_hung_server, and fix for status updates in download.py 2010-01-29 22:43:03 -08:00
Zooko O'Whielacronx
d62428c1e6 immutable: fix bug in tests, change line-endings to unix style, add comment 2010-01-29 10:42:37 -08:00
david-sarah
baa11a0ad4 New tests for #928 2010-01-29 04:38:45 -08:00
david-sarah
14280b009c Eliminate 'foo if test else bar' syntax that isn't supported by Python 2.4 2010-01-28 19:52:10 -08:00
david-sarah
4560e021a9 Show -IMM and -RO suffixes for types of immutable and read-only unknown nodes in directory listings 2010-01-28 14:08:00 -08:00
david-sarah
1bb627405f Fix inaccurate comment in test_mutant_dirnodes_are_omitted 2010-01-28 12:24:56 -08:00
david-sarah
b9eda4de6a Address comments by Kevan on 833 and add test for stripping spaces 2010-01-27 15:06:42 -08:00
david-sarah
56c00cb381 Miscellaneous documentation, test, and code formatting tweaks. 2010-01-26 23:03:09 -08:00
david-sarah
6057bc02cc Prevent mutable objects from being retrieved from an immutable directory, and associated forward-compatibility improvements. 2010-01-26 22:44:30 -08:00
Brian Warner
3880486f91 test_runner: cleanup, refactor common code into a non-executable method
Having both test_node() and test_client() (one of which calls the other) felt
confusing to me, so I changed it to have test_node(), test_client(), and a
common do_create() helper method.
2010-01-27 14:40:40 -08:00
Brian Warner
e769bbb6dd tahoe backup: skip all symlinks, with warning. Fixes #850, addresses #641. 2010-01-27 14:35:17 -08:00
david-sarah
5c5a6fe610 Patch to accept t=set-children as well as t=set_children 2010-01-23 19:00:20 -08:00
francois
b03406af9d tahoe_backup.py: display warnings on errors instead of stopping the whole backup. Fix #729.
This patch displays a warning to the user in two cases:
  
  1. When special files like symlinks, fifos, devices, etc. are found in the
     local source.
  
  2. If files or directories are not readables by the user running the 'tahoe
     backup' command.

In verbose mode, the number of skipped files and directories is printed at the
end of the backup.

Exit status returned by 'tahoe backup':

  - 0 everything went fine
  - 1 the backup failed
  - 2 files were skipped during the backup
2010-01-20 01:42:49 -08:00
david-sarah
174d356692 Message saying that we couldn't find bin/tahoe should say where we looked 2010-01-16 12:45:56 -08:00
david-sarah
87f1bae7fe Add create-node CLI command, and make create-client equivalent to create-node --no-storage (fixes #760) 2010-01-15 21:20:55 -08:00
Brian Warner
de14791caf Fix webapi t=mkdir with multpart/form-data, as on the Welcome page. Closes #919. 2010-01-20 22:50:52 -08:00
Brian Warner
9ab7524f0d test_dirnode.py: reduce scope of a Client instance, suggested by Kevan. 2010-01-14 22:27:13 -08:00
Brian Warner
01a7630334 test_provisioning: STAN is not always a list. Fix by David-Sarah Hopwood. 2010-01-14 17:46:32 -08:00
Brian Warner
e9223fcd0e hush pyflakes-0.4.0 warnings: slightly less-trivial fixes. Closes #900.
This includes one fix (in test_web) which was testing the wrong thing.
2010-01-14 14:17:19 -08:00
Brian Warner
731d15e56f hush pyflakes-0.4.0 warnings: remove trivial unused variables. For #900. 2010-01-14 14:15:29 -08:00
Brian Warner
874a979a8e tahoe add-alias/create-alias: don't corrupt non-newline-terminated alias
file. Closes #741.
2010-01-14 13:02:46 -08:00
Brian Warner
d3d1293d2f change docs and --help to use "grid" instead of "virtual drive": closes #892.
Thanks to David-Sarah Hopwood for the patch.
2010-01-14 12:11:19 -08:00
Brian Warner
d888bf3377 Clean up log.err calls, for one of the issues in #889.
allmydata.util.log.err() either takes a Failure as the first positional
argument, or takes no positional arguments and must be invoked in an
exception handler. Fixed its signature to match both foolscap.logging.log.err
and twisted.python.log.err . Included a brief unit test.
2010-01-11 17:33:43 -08:00
Brian Warner
bacb6fe5aa tidy up DeadReferenceError handling, ignore them in add_lease calls
Stop checking separately for ConnectionDone/ConnectionLost, since those have
been folded into DeadReferenceError since foolscap-0.3.1 . Write
rrefutil.trap_deadref() in terms of rrefutil.trap_and_discard() to improve
code coverage.
2010-01-11 16:07:23 -08:00
Brian Warner
1ed5bbdcb6 test_repairer: add (commented-out) test_each_byte, to see exactly what the
Verifier misses

The results (described in #819) match our expectations: it misses corruption
in unused share fields and in most container fields (which are only visible
to the storage server, not the client). 1265 bytes of a 2753 byte
share (hosting a 56-byte file with an artifically small segment size) are
unused, mostly in the unused tail of the overallocated UEB space (765 bytes),
and the allocated-but-unwritten plaintext_hash_tree (480 bytes).
2010-01-10 12:35:52 -08:00
Zooko O'Whielacronx
e7a9c000bf repairer: fix some wrong offsets in the randomized verifier tests, debugged by Brian
fixes #819
2010-01-10 12:37:21 -08:00
Brian Warner
1ff49a15bb test_repairer: fix colliding basedir names, which caused test inconsistencies 2010-01-10 00:46:19 -08:00
Zooko O'Whielacronx
876c4a153b repairer: add deterministic test for #819, mark as TODO 2010-01-09 17:36:19 -08:00
Brian Warner
ba0690c9d7 mutable repair: return successful=False when numshares<k (thus repair fails),
instead of weird errors. Closes #874 and #786.

Previously, if the file had 0 shares, this would raise TypeError as it tried
to call download_version(None). If the file had some shares but fewer than
'k', it would incorrectly raise MustForceRepairError.

Added get_successful() to the IRepairResults API, to give repair() a place to
report non-code-bug problems like this.
2009-12-29 15:37:46 -08:00
Brian Warner
794e32738f checker: don't let failures in add-lease affect checker results. Closes #875.
Mutable servermap updates and the immutable checker, when run with
add_lease=True, send both the do-you-have-block and add-lease commands in
parallel, to avoid an extra round trip time. Many older servers have problems
with add-lease and raise various exceptions, which don't generally matter.
The client-side code was catching+ignoring some of them, but unrecognized
exceptions were passed through to the DYHB code, concealing the DYHB results
from the checker, making it think the server had no shares.

The fix is to separate the code paths. Both commands are sent at the same
time, but the errback path from add-lease is handled separately. Known
exceptions are ignored, the others (both unknown-remote and all-local) are
logged (log.WEIRD, which will trigger an Incident), but neither will affect
the DYHB results.

The add-lease message is sent first, and we know that the server handles them
synchronously. So when the checker is done, we can be sure that all the
add-lease messages have been retired. This makes life easier for unit tests.
2009-12-29 15:01:08 -08:00
Brian Warner
a50fdca6ab test_cli: verify fix for "tahoe get" not creating empty file on error (#121) 2009-12-27 18:54:44 -05:00
Brian Warner
00d0ca3902 addendum to "Fix 'tahoe ls' on files (#771)"
tahoe_ls.py: tolerate missing metadata
web/filenode.py: minor cleanups
test_cli.py: test 'tahoe ls FILECAP'
2009-12-27 18:21:49 -05:00
Brian Warner
a8a768ef9d Fix 'tahoe ls' on files (#771). Patch adapted from Kevan Carstensen.
web/filenode.py: also serve edge metadata when using t=json on a
                 DIRCAP/childname object.
tahoe_ls.py: list file objects as if we were listing one-entry directories.
             Show edge metadata if we have it, which will be true when doing
             'tahoe ls DIRCAP/filename' and false when doing 'tahoe ls
             FILECAP'
2009-12-27 17:54:43 -05:00
Brian Warner
499add09e6 webapi: don't accept zero-length childnames during traversal. Closes #358, #676.
This forbids operations that would implicitly create a directory with a
zero-length (empty string) name, like what you'd get if you did "tahoe put
local /oops/blah" (#358) or "POST /uri/CAP//?t=mkdir" (#676). The error
message is fairly friendly too.

Also added code to "tahoe put" to catch this error beforehand and suggest the
correct syntax (i.e. without the leading slash).
2009-12-27 15:10:43 -05:00
Brian Warner
2e0a61a953 CLI: send 'Accept:' header to ask for text/plain tracebacks. Closes #646.
The webapi has been looking for an Accept header since 1.4.0, but it treats a
missing header as equal to */* (to honor RFC2616). This change finally
modifies our CLI tools to ask for "text/plain, application/octet-stream",
which seems roughly correct (we either want a plain-text traceback or error
message, or an uninterpreted chunk of binary data to save to disk). Some day
we'll figure out how JSON fits into this scheme.
2009-12-27 14:58:28 -05:00
Brian Warner
a4a6c02ef8 more #859: avoid deprecation warning for unit tests too, hush pyflakes
* factor maybe-import-sha logic into util.hashutil
2009-12-14 16:01:47 -08:00
Brian Warner
aa3caf812d mutable/servermap.py: stop reaching into private MutableFileNode attributes 2009-12-08 09:26:08 -08:00
Brian Warner
a980ea337b remove MutableFileNode.download(), prefer download_best_version() instead 2009-12-01 17:54:38 -05:00
Brian Warner
96834da0a2 Simplify immutable download API: use just filenode.read(consumer, offset, size)
* remove Downloader.download_to_data/download_to_filename/download_to_filehandle
* remove download.Data/FileName/FileHandle targets
* remove filenode.download/download_to_data/download_to_filename methods
* leave Downloader.download (the whole Downloader will go away eventually)
* add util.consumer.MemoryConsumer/download_to_data, for convenience
  (this is mostly used by unit tests, but it gets used by enough non-test
   code to warrant putting it in allmydata.util)
* update tests
* removes about 180 lines of code. Yay negative code days!

Overall plan is to rewrite immutable/download.py and leave filenode.read() as
the sole read-side API.
2009-12-01 17:53:30 -05:00
Brian Warner
74974b27fe Implement more coherent behavior when copying with dircaps/filecaps (closes #761). Patch by Kevan Carstensen. 2009-11-30 13:10:09 -08:00
david-sarah
ef002c935a storage server: detect disk space usage on Windows too (fixes #637) 2009-11-20 21:56:44 -08:00
Brian Warner
9901a08fd3 Add dirnodes to backupdb and "tahoe backup", closes #606.
* backups now share dirnodes with any previous backup, in any location,
  so renames and moves are handled very efficiently
* "tahoe backup" no longer bothers reading the previous snapshot
* if you switch grids, you should delete ~/.tahoe/private/backupdb.sqlite,
  to force new uploads of all files and directories
2009-11-26 15:42:57 -08:00
Brian Warner
366a309795 webapi: fix t=check for DIR2-LIT (i.e. empty immutable directories) 2009-11-26 15:27:31 -08:00
Brian Warner
41bcc9f39e PipelineError: fix str() on python2.4 . Closes #842. 2009-11-24 13:25:12 -08:00
Brian Warner
d918ab407e test_uri.py: s/NewDirnode/Dirnode/ , now that they aren't "new" anymore 2009-11-19 23:55:53 -08:00
Brian Warner
0cf320c2ab interface name cleanups: IFileNode, IImmutableFileNode, IMutableFileNode
The proper hierarchy is:
 IFilesystemNode
 +IFileNode
 ++IMutableFileNode
 ++IImmutableFileNode
 +IDirectoryNode

Also expand test_client.py (NodeMaker) to hit all IFilesystemNode types.
2009-11-19 23:52:55 -08:00
Brian Warner
d2badbea78 class name cleanups: s/FileNode/ImmutableFileNode/
also fix test/bench_dirnode.py for recent dirnode changes
2009-11-19 23:22:39 -08:00
Brian Warner
6e7fb1006d Use DIR-IMM and t=mkdir-immutable for "tahoe backup", for #828 2009-11-18 11:28:13 -08:00
Brian Warner
e046744f40 make get_size/get_current_size consistent for all IFilesystemNode classes
* stop caching most_recent_size in dirnode, rely upon backing filenode for it
* start caching most_recent_size in MutableFileNode
* return None when you don't know, not "?"
* only render None as "?" in the web "more info" page
* add get_size/get_current_size to UnknownNode
2009-11-18 11:16:24 -08:00
Brian Warner
512fe3ad62 ImmutableDirectoryURIVerifier: fix verifycap handling 2009-11-18 08:42:38 -08:00
Brian Warner
f85690697a Add t=mkdir-immutable to the webapi. Closes #607.
* change t=mkdir-with-children to not use multipart/form encoding. Instead,
  the request body is all JSON. t=mkdir-immutable uses this format too.
* make nodemaker.create_immutable_dirnode() get convergence from SecretHolder,
  but let callers override it
* raise NotDeepImmutableError instead of using assert()
* add mutable= argument to DirectoryNode.create_subdirectory(), default True
2009-11-17 23:09:00 -08:00
Brian Warner
d2426ea3bd move convergence secret into SecretHolder, next to lease secret 2009-11-17 17:54:44 -08:00
Brian Warner
5fe713fc52 nodemaker: implement immutable directories (internal interface), for #607
* nodemaker.create_from_cap() now handles DIR2-CHK and DIR2-LIT
* client.create_immutable_dirnode() is used to create them
* no webapi yet
2009-11-11 16:22:33 -08:00
Brian Warner
cc422f8dc0 stop using IURI()/etc as an adapter 2009-11-11 14:45:42 -08:00
Brian Warner
131e05b155 clean up uri-vs-cap terminology, emphasize cap instances instead of URI strings
* "cap" means a python instance which encapsulates a filecap/dircap (uri.py)
 * "uri" means a string with a "URI:" prefix
 * FileNode instances are created with (and retain) a cap instance, and
   generate uri strings on demand
 * .get_cap/get_readcap/get_verifycap/get_repaircap return cap instances
 * .get_uri/get_readonly_uri return uri strings

* add filenode.download_to_filename() for control.py, should find a better way
* use MutableFileNode.init_from_cap, not .init_from_uri
* directory URI instances: use get_filenode_cap, not get_filenode_uri
* update/cleanup bench_dirnode.py to match, add Makefile target to run it
2009-11-11 14:26:19 -08:00
Brian Warner
f47672d12a add parser for immutable directory caps: DIR2-CHK, DIR2-LIT, DIR2-CHK-Verifier 2009-11-04 10:13:51 -08:00
Zooko O'Whielacronx
9733201c0b wui: s/TahoeLAFS/Tahoe-LAFS/ 2009-10-28 18:50:50 -08:00
Zooko O'Whielacronx
1be4d45779 tests: bump up the timeout on test_repairer to see if 120 seconds was too short for François's ARM box to do the test even when it was doing it right. 2009-10-27 13:48:00 -08:00
Brian Warner
2695af91a7 dirnode.pack_children(): add deep_immutable= argument
This will be used by DIR2:CHK to enforce the deep-immutability requirement.
2009-10-26 09:28:09 -07:00
Brian Warner
768c76aa5f webapi: use t=mkdir-with-children instead of a children= arg to t=mkdir .
This is safer: in the earlier API, an old webapi server would silently ignore
the initial children, and clients trying to set them would have to fetch the
newly-created directory to discover the incompatibility. In the new API,
clients using t=mkdir-with-children against an old webapi server will get a
clear error.
2009-10-25 18:13:21 -07:00
Brian Warner
1273b5c233 nodemaker.create_new_mutable_directory: pack_children() in initial_contents=
instead of creating an empty file and then adding the children later.

This should speed up mkdir(initial_children) considerably, removing two
roundtrips and an entire read-modify-write cycle, probably bringing it down
to a single roundtrip. A quick test (against the volunteergrid) suggests a
30% speedup.

test_dirnode: add new tests to enforce the restrictions that interfaces.py
claims for create_new_mutable_directory(): no UnknownNodes, metadata dicts
2009-10-19 17:51:18 -07:00
Brian Warner
59ee02e82f test_dirnode.py: add tests of initial_children= args to client.create_dirnode
and nodemaker.create_new_mutable_directory
2009-10-17 12:41:59 -07:00
Brian Warner
b4ec86c95a update many dirnode interfaces to accept dict-of-nodes instead of dict-of-caps
interfaces.py: define INodeMaker, document argument values, change
               create_new_mutable_directory() to take dict-of-nodes. Change
               dirnode.set_nodes() and dirnode.create_subdirectory() too.
nodemaker.py: use INodeMaker, update create_new_mutable_directory()
client.py: have create_dirnode() delegate initial_children= to nodemaker
dirnode.py (Adder): take dict-of-nodes instead of list-of-nodes, which
                    updates set_nodes() and create_subdirectory()
web/common.py (convert_initial_children_json): create dict-of-nodes
web/directory.py: same
web/unlinked.py: same
test_dirnode.py: update tests to match
2009-10-17 12:28:29 -07:00
Brian Warner
ea373de042 move dirnode.CachingDict to dictutil.AuxValueDict, generalize method names,
improve tests. Let dirnode _pack_children accept either dict or AuxValueDict.
2009-10-17 11:00:05 -07:00
Brian Warner
e63f59f50c test/common.py: update FakeMutableFileNode to new contents= callable scheme 2009-10-12 22:21:54 -07:00
Brian Warner
c2520e4ec7 client.create_mutable_file(contents=) now accepts a callable, which is
invoked with the new MutableFileNode and is supposed to return the initial
contents. This can be used by e.g. a new dirnode which needs the filenode's
writekey to encrypt its initial children.

create_mutable_file() still accepts a bytestring too, or None for an empty
file.
2009-10-12 20:12:32 -07:00
Brian Warner
b30041c5ec webapi: t=mkdir now accepts initial children, using the same JSON that t=json
emits.

client.create_dirnode(initial_children=) now works.
2009-10-12 19:34:44 -07:00
Brian Warner
cf65cc2ae3 replace dirnode.create_empty_directory() with create_subdirectory(), which
takes an initial_children= argument
2009-10-12 19:15:20 -07:00
Brian Warner
d079eb45f6 dirnode.set_children: change return value: fire with self instead of None 2009-10-12 18:50:26 -07:00
Brian Warner
f871c3bb3d dirnode.set_nodes: change return value: fire with self instead of None 2009-10-12 18:45:46 -07:00
Brian Warner
304aadd4f7 dirnode.set_children: take a dict, not a list 2009-10-12 17:24:40 -07:00
Brian Warner
e2ffc3dc03 dirnode.set_uri/set_children: change signature to take writecap+readcap
instead of a single cap. The webapi t=set_children call benefits too.
2009-10-12 16:51:26 -07:00