Commit Graph

2620 Commits

Author SHA1 Message Date
Zooko O'Whielacronx
35efdf04ca tests: don't let bench_dirnode.py do stuff and have side-effects at import time (unless __name__ == '__main__') 2010-05-21 05:20:52 -07:00
Zooko O'Whielacronx
a48979516e tests: increase timeout to give François's ARM buildslave a chance to complete the tests 2010-05-20 06:45:26 -07:00
Kevan Carstensen
d0706d27ea Hush pyflakes warnings 2010-05-15 11:43:44 -07:00
Kevan Carstensen
9179dd099a Improve code coverage of the Tahoe2PeerSelector tests. 2010-05-14 20:29:13 -07:00
Kevan Carstensen
55a2849f38 Remove a comment that no longer makes sense. 2010-05-14 13:35:16 -07:00
Kevan Carstensen
e225f573b9 Fix up the behavior of #778, per reviewers' comments
- Make some important utility functions clearer and more thoroughly 
    documented.
  - Assert in upload.servers_of_happiness that the buckets attributes
    of PeerTrackers passed to it are mutually disjoint.
  - Get rid of some silly non-Pythonisms that I didn't see when I first
    wrote these patches.
  - Make sure that should_add_server returns true when queried about a 
    shnum that it doesn't know about yet.
  - Change Tahoe2PeerSelector.preexisting_shares to map a shareid to a set
    of peerids, alter dependencies to deal with that.
  - Remove upload.should_add_servers, because it is no longer necessary
  - Move upload.shares_of_happiness and upload.shares_by_server to a utility
    file.
  - Change some points in Tahoe2PeerSelector.
  - Compute servers_of_happiness using a bipartite matching algorithm that 
    we know is optimal instead of an ad-hoc greedy algorithm that isn't.
  - Change servers_of_happiness to just take a sharemap as an argument,
    change its callers to merge existing_shares and used_peers before 
    calling it.
  - Change an error message in the encoder to be more appropriate for 
    servers of happiness.
  - Clarify the wording of an error message in immutable/upload.py
  - Refactor a happiness failure message to happinessutil.py, and make
    immutable/upload.py and immutable/encode.py use it.
  - Move the word "only" as far to the right as possible in failure 
    messages.
  - Use a better definition of progress during peer selection.
  - Do read-only peer share detection queries in parallel, not sequentially.
  - Clean up logging semantics; print the query statistics whenever an
    upload is unsuccessful, not just in one case.
2010-05-13 17:49:17 -07:00
Kevan Carstensen
9bc71d3da0 Alter the error message when an upload fails, per some comments in #778.
When I first implemented #778, I just altered the error messages to refer to
servers where they referred to shares. The resulting error messages weren't
very good. These are a bit better.
2009-12-30 15:03:44 -07:00
Kevan Carstensen
8bcc771e26 Change "UploadHappinessError" to "UploadUnhappinessError" 2009-12-04 22:30:37 -07:00
Kevan Carstensen
68fb556e93 Alter the error message returned when peer selection fails
The Tahoe2PeerSelector returned either NoSharesError or NotEnoughSharesError
for a variety of error conditions that weren't informatively described by them.
This patch creates a new error, UploadHappinessError, replaces uses of 
NoSharesError and NotEnoughSharesError with it, and alters the error message
raised with the errors to be more in line with the new servers_of_happiness
behavior. See ticket #834 for more information.
2009-11-22 18:24:05 -07:00
Kevan Carstensen
320582be5a Eliminate overcounting iof servers_of_happiness in Tahoe2PeerSelector; also reorganize some things. 2009-11-17 19:45:42 -07:00
Kevan Carstensen
4e29060847 Change stray "shares_of_happiness" to "servers_of_happiness" 2009-11-16 15:24:59 -07:00
Kevan Carstensen
a816de3f23 Alter Tahoe2PeerSelector to make sure that it recognizes existing shares on readonly servers, fixing an issue in #778 2009-11-16 13:28:05 -07:00
Kevan Carstensen
8c71df53f9 Alter 'immutable/encode.py' and 'immutable/upload.py' to use servers_of_happiness instead of shares_of_happiness. 2009-11-04 05:12:22 -07:00
Kevan Carstensen
b2d8a7cec2 Alter the signature of set_shareholders in IEncoder to add a 'servermap' parameter, which gives IEncoders enough information to perform a sane check for servers_of_happiness. 2009-11-03 21:32:41 -07:00
Kevan Carstensen
89a8da9dd2 Alter CiphertextDownloader to work with servers_of_happiness 2009-09-23 21:19:32 -07:00
Kevan Carstensen
5683112a02 Revisions of the #778 tests, per reviewers' comments
- Fix comments and confusing naming.
- Add tests for the new error messages suggested by David-Sarah
  and Zooko.
- Alter existing tests for new error messages.
- Make sure that the tests continue to work with the trunk.
- Add a test for a mutual disjointedness assertion that I added to
  upload.servers_of_happiness.
- Fix the comments to correctly reflect read-onlyness
- Add a test for an edge case in should_add_server
- Add an assertion to make sure that share redistribution works as it 
  should
- Alter tests to work with revised servers_of_happiness semantics
- Remove tests for should_add_server, since that function no longer exists.
- Alter tests to know about merge_peers, and to use it before calling 
  servers_of_happiness.
- Add tests for merge_peers.
- Add Zooko's puzzles to the tests.
- Edit encoding tests to expect the new kind of failure message.
- Edit tests to expect error messages with the word "only" moved as far
  to the right as possible.
- Extended and cleaned up some helper functions.
- Changed some tests to call more appropriate helper functions.
- Added a test for the failing redistribution algorithm
- Added a test for the progress message
- Added a test for the upper bound on readonly peer share discovery.
2010-05-13 18:25:42 -07:00
Kevan Carstensen
d360ee0d2e Alter various unit tests to work with the new happy behavior 2010-01-07 12:13:25 -07:00
Kevan Carstensen
953c9555ca Replace "UploadHappinessError" with "UploadUnhappinessError" in tests. 2009-12-04 22:34:53 -07:00
Kevan Carstensen
c0f3dbb919 Add tests for the behavior described in #834. 2009-11-22 19:20:08 -07:00
Kevan Carstensen
c3b11dedea Re-work 'test_upload.py' to be more readable; add more tests for #778 2009-11-16 13:23:34 -07:00
Kevan Carstensen
9590690450 Test Tahoe2PeerSelector to make sure that it recognizeses existing shares on readonly servers 2009-11-08 18:37:35 -07:00
Kevan Carstensen
324a72e4a3 Add more tests for comment:53 in ticket #778 2009-11-04 05:28:49 -07:00
Kevan Carstensen
697f1cfae8 Add a test for upload.shares_by_server 2009-11-04 05:13:24 -07:00
Kevan Carstensen
0d8e70bfe5 Minor tweak to an existing test -- make the first server read-write, instead of read-only 2009-11-03 21:42:32 -07:00
Kevan Carstensen
5ad29d4e8b Alter tests to use the new form of set_shareholders 2009-11-03 21:36:02 -07:00
"Kevan Carstensen"
ee9690b357 Refactor some behavior into a mixin, and add tests for the behavior described in #778 2009-10-30 02:19:08 -07:00
Kevan Carstensen
362f204075 Alter NoNetworkGrid to allow the creation of readonly servers for testing purposes. 2009-10-17 18:30:13 -07:00
"Kevan Carstensen"
5fe125ed74 Alter wording in 'interfaces.py' to be correct wrt #778 2009-12-04 21:40:05 -07:00
Zooko O'Whielacronx
6037df55b3 web: add a simple test that you can load directory.xhtml 2010-05-09 23:37:29 -07:00
Francois Deppierraz
dd95e8e71b CLI: Support for https url in option --node-url
This patch modifies the regular expression used for verifying of '--node-url'
parameter.  Support for accessing a Tahoe gateway over HTTPS was already
present, thanks to Python's urllib.
2010-04-30 11:56:09 -07:00
Brian Warner
e2031aff0f backupdb.did_create_directory: use REPLACE INTO, not INSERT INTO + ignore error
This handles the case where we upload a new tahoe directory for a
previously-processed local directory, possibly creating a new dircap (if the
metadata had changed). Now we replace the old dirhash->dircap record. The
previous behavior left the old record in place (with the old dircap and
timestamps), so we'd never stop creating new directories and never converge
on a null backup.
2010-04-27 22:08:03 -07:00
Brian Warner
f6f8256b35 "tahoe webopen": add --info flag, to get ?t=info
Also fix some trailing whitespace.
2010-04-24 16:30:03 -07:00
Zooko O'Whielacronx
80ae6f2706 setup: fix error in licensing declaration from Shawn Willden, as noted by the FSF compliance division 2010-03-09 08:37:36 -08:00
Zooko O'Whielacronx
34a50b857a docs: a few small edits to try to guide newcomers through the docs
These edits were suggested by my watching over Jake Appelbaum's shoulder as he completely ignored/skipped/missed install.html and also as he decided that debian.txt wouldn't help him with basic installation. Then I threw in a few docs edits that have been sitting around in my sandbox asking to be committed for months.
2010-03-03 15:19:02 -08:00
david-sarah
40edf8f419 Change code that gives a base32 SI or an empty string to be more straightforward. (#948) 2010-02-26 22:55:51 -08:00
david-sarah
a2ed17f2a0 Additional test for DIR2-LIT directories in test_web.py, fixed version (#948) 2010-02-24 20:18:24 -08:00
david-sarah
d7b50a3b86 Additional fixes for DIR2-LIT More Info page and deep-check/manifest operations (#948) 2010-02-24 00:02:20 -08:00
david-sarah
e6aee33bb7 directories: add DIR2-LIT directories to test_deepcheck.py (#948) 2010-02-23 23:54:33 -08:00
david-sarah
187d837c1d dirnode: add tests of literal dirnodes (current and fix for #948) 2010-02-23 20:33:45 -08:00
david-sarah
fec9185f2a Additional fix for abbrev_si, with test 2010-02-21 19:36:52 -08:00
david-sarah
d29ec184a6 Additions to test_web.py for #948 2010-02-21 18:53:52 -08:00
david-sarah
973f0afdd3 Change direct accesses to an_uri.storage_index to calls to .get_storage_index() (fixes #948) 2010-02-21 18:45:04 -08:00
david-sarah
6a7feea455 Tweak to 'tahoe ls --help' output (#837) 2010-02-23 19:02:31 -08:00
david-sarah
9741b9655f Test behaviour of 'tahoe ls' for unknown objects (#837) 2010-02-23 18:59:13 -08:00
david-sarah
03134eedb5 Improve behaviour of 'tahoe ls' for unknown objects, addressing kevan's comments 2010-02-19 22:13:13 -08:00
Brian Warner
1de4d2c594 test_util.py: improve coverage of util.time_format 2010-02-23 23:46:37 -05:00
Kevan Carstensen
81ad52d6eb Change OphandleTable to use a deterministic clock, so we can test it
To test the changes for #577, we need a deterministic way to simulate
the passage of long periods of time. twisted.internet.task.Clock seems,
from my Googling, to be the way to go for this functionality. I changed
a few things so that OphandleTable would use twisted.internet.task.Clock
when testing:

  * WebishServer.__init___ now takes an optional 'clock' parameter,
  * which it passes to the root.Root instance it creates.
  * root.Root.__init__ now takes an optional 'clock' parameter, which it
    passes to the OphandleTable.__init__ method.
  * OphandleTable.__init__ now takes an optional 'clock' parameter. If
    it is provided, and it isn't None, its callLater method will be used
    to schedule ophandle expirations (as opposed to using
    reactor.callLater, which is what OphandleTable does normally).
  * The WebMixin object in test_web.py now sets a self.clock parameter,
    which is a twisted.internet.task.Clock that it feeds to the 
    WebishServer it creates. 

Tests using the WebMixin can control the passage of time in
OphandleTable by accessing self.clock.
2010-02-20 13:07:13 -08:00
Kevan Carstensen
bcdc78bd44 Add tests for the ophandle expiration behavior in #577 2010-02-20 17:04:55 -08:00
Kevan Carstensen
e9b59a4949 Increase ophandle expiration times, per #577 2010-02-20 17:05:12 -08:00
david-sarah
85a50feeaa More cleanups to test_cli using new utilities for reading and writing files. 2010-02-05 17:38:55 -08:00
david-sarah
c984a09fe7 Fix race conditions and missing callback in allmydata.test.test_cli.Cp.test_copy_using_filecap, add utilities for one-liner reading and writing of files, and fix cases in test_cli where files were not being closed after writing. 2010-02-05 17:37:27 -08:00
Kevan Carstensen
c00a62a2e7 Add tests for #939 2010-02-11 22:21:37 -08:00
Kevan Carstensen
63c8c65747 Alter CLI utilities to handle nonexistent aliases better 2010-02-10 18:43:18 -08:00
Brian Warner
4040b1000b web/storage.py: display total-seen on the last-complete-cycle line. For #940. 2010-02-07 16:20:10 -08:00
Brian Warner
880f824103 code coverage: replace figleaf with coverage.py, should work on py2.6 now.
It still lacks the right HTML report (the builtin report is very pretty, but
lacks the "lines uncovered" numbers that I want), and the half-finished
delta-from-last-run measurements.
2010-02-03 08:54:21 -08:00
Zooko O'Whielacronx
3e4342ecb3 immutable: downloader accepts notifications of buckets even if those notifications arrive after he has begun downloading shares.
This can be useful if one of the ones that he has already begun downloading fails. See #287 for discussion. This fixes part of #287 which part was a regression caused by #928, namely this fixes fail-over in case a share is corrupted (or the server returns an error or disconnects). This does not fix the related issue mentioned in #287 if a server hangs and doesn't reply to requests for blocks.
2010-01-31 22:16:10 -08:00
Zooko O'Whielacronx
e4e2599017 tests: don't require tahoe to run with no noise if we are using an old twisted that emits DeprecationWarnings 2010-01-31 21:23:23 -08:00
david-sarah
100548257a Use if instead of assert to check for twisted ftp patch 2010-01-26 17:55:29 -08:00
Zooko O'Whielacronx
7d32fafc44 tests: stop being surprised that Nevow no longer prints out warnings when it tries to find its static files
Unless we are using a sufficiently new version of Nevow, in which case if it prints out warnings then this is a hard test failure. :-)
2010-01-31 20:11:44 -08:00
david-sarah
6215ebd95e cli: suppress DeprecationWarnings emitted from importing nevow and twisted. Fixes #859 2010-01-31 16:44:29 -08:00
david-sarah
ea3954372a Improvements to test_unknownnode to cover invalid cap URIs with known prefixes 2010-01-29 22:39:08 -08:00
david-sarah
37a242e01a Improvements to test_hung_server, and fix for status updates in download.py 2010-01-29 22:43:03 -08:00
Zooko O'Whielacronx
d62428c1e6 immutable: fix bug in tests, change line-endings to unix style, add comment 2010-01-29 10:42:37 -08:00
david-sarah
baa11a0ad4 New tests for #928 2010-01-29 04:38:45 -08:00
Zooko O'Whielacronx
2bd9dfa5bd immutable: download from the first servers which provide at least K buckets instead of waiting for all servers to reply
This should put an end to the phenomenon I've been seeing that a single hung server can cause all downloads on a grid to hang.  Also it should speed up all downloads by (a) not-waiting for responses to queries that it doesn't need, and (b) downloading shares from the servers which answered the initial query the fastest.
Also, do not count how many buckets you've gotten when deciding whether the download has enough shares or not -- instead count how many buckets to *unique* shares that you've gotten.  This appears to improve a slightly weird behavior in the current download code in which receiving >= K different buckets all to the same sharenumber would make it think it had enough to download the file when in fact it hadn't.
This patch needs tests before it is actually ready for trunk.
2010-01-27 15:34:17 -08:00
david-sarah
14280b009c Eliminate 'foo if test else bar' syntax that isn't supported by Python 2.4 2010-01-28 19:52:10 -08:00
david-sarah
3e35959e9b Add mutable field to t=json output for unknown nodes, when mutability is known 2010-01-28 19:14:24 -08:00
david-sarah
4560e021a9 Show -IMM and -RO suffixes for types of immutable and read-only unknown nodes in directory listings 2010-01-28 14:08:00 -08:00
david-sarah
1bb627405f Fix inaccurate comment in test_mutant_dirnodes_are_omitted 2010-01-28 12:24:56 -08:00
david-sarah
b9eda4de6a Address comments by Kevan on 833 and add test for stripping spaces 2010-01-27 15:06:42 -08:00
david-sarah
56c00cb381 Miscellaneous documentation, test, and code formatting tweaks. 2010-01-26 23:03:09 -08:00
david-sarah
6057bc02cc Prevent mutable objects from being retrieved from an immutable directory, and associated forward-compatibility improvements. 2010-01-26 22:44:30 -08:00
Brian Warner
3880486f91 test_runner: cleanup, refactor common code into a non-executable method
Having both test_node() and test_client() (one of which calls the other) felt
confusing to me, so I changed it to have test_node(), test_client(), and a
common do_create() helper method.
2010-01-27 14:40:40 -08:00
Brian Warner
5045a8721a scripts/runner.py: simplify David-Sarah's clever grouped-commands usage trick 2010-01-27 14:37:58 -08:00
Brian Warner
e769bbb6dd tahoe backup: skip all symlinks, with warning. Fixes #850, addresses #641. 2010-01-27 14:35:17 -08:00
Brian Warner
52185053c3 "tahoe backup": fix --exclude-vcs docs to include Git 2010-01-27 12:10:44 -08:00
david-sarah
5c5a6fe610 Patch to accept t=set-children as well as t=set_children 2010-01-23 19:00:20 -08:00
Zooko O'Whielacronx
4a4305e7e6 ftpd: clearer error message if Twisted needs a patch (by Nils Durner) 2010-01-26 06:34:11 -08:00
francois
b03406af9d tahoe_backup.py: display warnings on errors instead of stopping the whole backup. Fix #729.
This patch displays a warning to the user in two cases:
  
  1. When special files like symlinks, fifos, devices, etc. are found in the
     local source.
  
  2. If files or directories are not readables by the user running the 'tahoe
     backup' command.

In verbose mode, the number of skipped files and directories is printed at the
end of the backup.

Exit status returned by 'tahoe backup':

  - 0 everything went fine
  - 1 the backup failed
  - 2 files were skipped during the backup
2010-01-20 01:42:49 -08:00
david-sarah
174d356692 Message saying that we couldn't find bin/tahoe should say where we looked 2010-01-16 12:45:56 -08:00
Zooko O'Whielacronx
a1444d9367 cli: merge the better version of David-Sarah's split-usage-and-help patch with the earlier version that I mistakenly committed 2010-01-25 20:45:59 -08:00
david-sarah
b079f32da2 Split tahoe --help options into groups. 2010-01-11 20:39:35 -08:00
Zooko O'Whielacronx
b94b9af189 cli: split usage strings into groups (patch by David-Sarah Hopwood) 2010-01-25 20:39:21 -08:00
david-sarah
87f1bae7fe Add create-node CLI command, and make create-client equivalent to create-node --no-storage (fixes #760) 2010-01-15 21:20:55 -08:00
david-sarah
26ab58e006 Remove replace= parameter to mkdir-immutable and mkdir-with-children 2010-01-24 14:43:25 -08:00
Brian Warner
de14791caf Fix webapi t=mkdir with multpart/form-data, as on the Welcome page. Closes #919. 2010-01-20 22:50:52 -08:00
Brian Warner
bb4883bbd4 tahoe_add_alias.py: minor refactoring 2010-01-14 22:42:20 -08:00
Brian Warner
9ab7524f0d test_dirnode.py: reduce scope of a Client instance, suggested by Kevan. 2010-01-14 22:27:13 -08:00
Brian Warner
01a7630334 test_provisioning: STAN is not always a list. Fix by David-Sarah Hopwood. 2010-01-14 17:46:32 -08:00
Brian Warner
2e098e2a4d web/directory.py mkdir-immutable: hush pyflakes, add TODO for #903 behavior 2010-01-14 14:28:04 -08:00
Brian Warner
e9223fcd0e hush pyflakes-0.4.0 warnings: slightly less-trivial fixes. Closes #900.
This includes one fix (in test_web) which was testing the wrong thing.
2010-01-14 14:17:19 -08:00
Brian Warner
731d15e56f hush pyflakes-0.4.0 warnings: remove trivial unused variables. For #900. 2010-01-14 14:15:29 -08:00
Brian Warner
874a979a8e tahoe add-alias/create-alias: don't corrupt non-newline-terminated alias
file. Closes #741.
2010-01-14 13:02:46 -08:00
Brian Warner
d3d1293d2f change docs and --help to use "grid" instead of "virtual drive": closes #892.
Thanks to David-Sarah Hopwood for the patch.
2010-01-14 12:11:19 -08:00
Brian Warner
fb879ddea4 client.py: fix/update comments on KeyGenerator 2010-01-12 16:42:26 -08:00
Brian Warner
d888bf3377 Clean up log.err calls, for one of the issues in #889.
allmydata.util.log.err() either takes a Failure as the first positional
argument, or takes no positional arguments and must be invoked in an
exception handler. Fixed its signature to match both foolscap.logging.log.err
and twisted.python.log.err . Included a brief unit test.
2010-01-11 17:33:43 -08:00
Brian Warner
bacb6fe5aa tidy up DeadReferenceError handling, ignore them in add_lease calls
Stop checking separately for ConnectionDone/ConnectionLost, since those have
been folded into DeadReferenceError since foolscap-0.3.1 . Write
rrefutil.trap_deadref() in terms of rrefutil.trap_and_discard() to improve
code coverage.
2010-01-11 16:07:23 -08:00
Brian Warner
1ed5bbdcb6 test_repairer: add (commented-out) test_each_byte, to see exactly what the
Verifier misses

The results (described in #819) match our expectations: it misses corruption
in unused share fields and in most container fields (which are only visible
to the storage server, not the client). 1265 bytes of a 2753 byte
share (hosting a 56-byte file with an artifically small segment size) are
unused, mostly in the unused tail of the overallocated UEB space (765 bytes),
and the allocated-but-unwritten plaintext_hash_tree (480 bytes).
2010-01-10 12:35:52 -08:00
Zooko O'Whielacronx
e7a9c000bf repairer: fix some wrong offsets in the randomized verifier tests, debugged by Brian
fixes #819
2010-01-10 12:37:21 -08:00
Brian Warner
1ff49a15bb test_repairer: fix colliding basedir names, which caused test inconsistencies 2010-01-10 00:46:19 -08:00
Zooko O'Whielacronx
876c4a153b repairer: add deterministic test for #819, mark as TODO 2010-01-09 17:36:19 -08:00
Brian Warner
419a076f59 mutable/publish: don't loop() right away upon DeadReferenceError. Closes #877
The bug was that a disconnected server could cause us to re-enter the initial
loop() call, sending multiple queries to a single server, provoking an
incorrect UCWE. To fix it, stall the loop() with an eventual.fireEventually()
2010-01-02 14:08:41 -08:00
Brian Warner
db19b62702 immutable/checker.py: oops, forgot some imports. Also hush pyflakes. 2009-12-29 15:39:09 -08:00
Brian Warner
ba0690c9d7 mutable repair: return successful=False when numshares<k (thus repair fails),
instead of weird errors. Closes #874 and #786.

Previously, if the file had 0 shares, this would raise TypeError as it tried
to call download_version(None). If the file had some shares but fewer than
'k', it would incorrectly raise MustForceRepairError.

Added get_successful() to the IRepairResults API, to give repair() a place to
report non-code-bug problems like this.
2009-12-29 15:37:46 -08:00
Brian Warner
acd211765c node.py/interfaces.py: minor docs fixes 2009-12-29 15:04:09 -08:00
Brian Warner
794e32738f checker: don't let failures in add-lease affect checker results. Closes #875.
Mutable servermap updates and the immutable checker, when run with
add_lease=True, send both the do-you-have-block and add-lease commands in
parallel, to avoid an extra round trip time. Many older servers have problems
with add-lease and raise various exceptions, which don't generally matter.
The client-side code was catching+ignoring some of them, but unrecognized
exceptions were passed through to the DYHB code, concealing the DYHB results
from the checker, making it think the server had no shares.

The fix is to separate the code paths. Both commands are sent at the same
time, but the errback path from add-lease is handled separately. Known
exceptions are ignored, the others (both unknown-remote and all-local) are
logged (log.WEIRD, which will trigger an Incident), but neither will affect
the DYHB results.

The add-lease message is sent first, and we know that the server handles them
synchronously. So when the checker is done, we can be sure that all the
add-lease messages have been retired. This makes life easier for unit tests.
2009-12-29 15:01:08 -08:00
Brian Warner
a50fdca6ab test_cli: verify fix for "tahoe get" not creating empty file on error (#121) 2009-12-27 18:54:44 -05:00
Brian Warner
00d0ca3902 addendum to "Fix 'tahoe ls' on files (#771)"
tahoe_ls.py: tolerate missing metadata
web/filenode.py: minor cleanups
test_cli.py: test 'tahoe ls FILECAP'
2009-12-27 18:21:49 -05:00
Brian Warner
a8a768ef9d Fix 'tahoe ls' on files (#771). Patch adapted from Kevan Carstensen.
web/filenode.py: also serve edge metadata when using t=json on a
                 DIRCAP/childname object.
tahoe_ls.py: list file objects as if we were listing one-entry directories.
             Show edge metadata if we have it, which will be true when doing
             'tahoe ls DIRCAP/filename' and false when doing 'tahoe ls
             FILECAP'
2009-12-27 17:54:43 -05:00
Brian Warner
931ab76588 tahoe_get: don't create the output file on error. Closes #121. 2009-12-27 17:04:04 -05:00
Brian Warner
499add09e6 webapi: don't accept zero-length childnames during traversal. Closes #358, #676.
This forbids operations that would implicitly create a directory with a
zero-length (empty string) name, like what you'd get if you did "tahoe put
local /oops/blah" (#358) or "POST /uri/CAP//?t=mkdir" (#676). The error
message is fairly friendly too.

Also added code to "tahoe put" to catch this error beforehand and suggest the
correct syntax (i.e. without the leading slash).
2009-12-27 15:10:43 -05:00
Brian Warner
2e0a61a953 CLI: send 'Accept:' header to ask for text/plain tracebacks. Closes #646.
The webapi has been looking for an Accept header since 1.4.0, but it treats a
missing header as equal to */* (to honor RFC2616). This change finally
modifies our CLI tools to ask for "text/plain, application/octet-stream",
which seems roughly correct (we either want a plain-text traceback or error
message, or an uninterpreted chunk of binary data to save to disk). Some day
we'll figure out how JSON fits into this scheme.
2009-12-27 14:58:28 -05:00
Brian Warner
a4a6c02ef8 more #859: avoid deprecation warning for unit tests too, hush pyflakes
* factor maybe-import-sha logic into util.hashutil
2009-12-14 16:01:47 -08:00
Zooko O'Whielacronx
b69e1c600d use hashlib module if available, thus avoiding a DeprecationWarning for importing the old sha module; fixes #859 2009-12-14 13:27:03 -08:00
Brian Warner
47cba4b51a mutable/retrieve.py: stop reaching into private MutableFileNode attributes 2009-12-08 09:29:21 -08:00
Brian Warner
aa3caf812d mutable/servermap.py: stop reaching into private MutableFileNode attributes 2009-12-08 09:26:08 -08:00
Brian Warner
eb18686284 mutable/servermap.py: oops, query N+e servers in MODE_WRITE, not k+e
under normal conditions, this wouldn't cause any problems, but if the shares
are really sparse (perhaps because new servers were added), then
file-modifies might stop looking too early and leave old shares in place
2009-12-08 09:11:56 -08:00
Brian Warner
696cc000d0 control.py: fix speedtest: use download_best_version (not read) on mutable nodes 2009-12-06 22:05:12 -08:00
Zooko O'Whielacronx
4706859967 setup: ignore _darcs in the "test-clean" test and make the "clean" step remove all .egg's in the root dir 2009-12-06 10:48:35 -08:00
Brian Warner
a980ea337b remove MutableFileNode.download(), prefer download_best_version() instead 2009-12-01 17:54:38 -05:00
Brian Warner
96834da0a2 Simplify immutable download API: use just filenode.read(consumer, offset, size)
* remove Downloader.download_to_data/download_to_filename/download_to_filehandle
* remove download.Data/FileName/FileHandle targets
* remove filenode.download/download_to_data/download_to_filename methods
* leave Downloader.download (the whole Downloader will go away eventually)
* add util.consumer.MemoryConsumer/download_to_data, for convenience
  (this is mostly used by unit tests, but it gets used by enough non-test
   code to warrant putting it in allmydata.util)
* update tests
* removes about 180 lines of code. Yay negative code days!

Overall plan is to rewrite immutable/download.py and leave filenode.read() as
the sole read-side API.
2009-12-01 17:53:30 -05:00
Brian Warner
1059db51f2 server.py: undo my bogus 'correction' of David-Sarah's comment fix
and move it to a better line
2009-11-30 21:46:07 -05:00
Brian Warner
74974b27fe Implement more coherent behavior when copying with dircaps/filecaps (closes #761). Patch by Kevan Carstensen. 2009-11-30 13:10:09 -08:00
Brian Warner
91e7cb484b storage.py: update comment 2009-11-30 11:59:13 -08:00
david-sarah
ef002c935a storage server: detect disk space usage on Windows too (fixes #637) 2009-11-20 21:56:44 -08:00
david-sarah
c4d38ad4c5 make status of finished operations consistently "Finished" 2009-11-20 22:15:43 -08:00
Brian Warner
9901a08fd3 Add dirnodes to backupdb and "tahoe backup", closes #606.
* backups now share dirnodes with any previous backup, in any location,
  so renames and moves are handled very efficiently
* "tahoe backup" no longer bothers reading the previous snapshot
* if you switch grids, you should delete ~/.tahoe/private/backupdb.sqlite,
  to force new uploads of all files and directories
2009-11-26 15:42:57 -08:00
Brian Warner
366a309795 webapi: fix t=check for DIR2-LIT (i.e. empty immutable directories) 2009-11-26 15:27:31 -08:00
Brian Warner
41bcc9f39e PipelineError: fix str() on python2.4 . Closes #842. 2009-11-24 13:25:12 -08:00
Brian Warner
d918ab407e test_uri.py: s/NewDirnode/Dirnode/ , now that they aren't "new" anymore 2009-11-19 23:55:53 -08:00
Brian Warner
0cf320c2ab interface name cleanups: IFileNode, IImmutableFileNode, IMutableFileNode
The proper hierarchy is:
 IFilesystemNode
 +IFileNode
 ++IMutableFileNode
 ++IImmutableFileNode
 +IDirectoryNode

Also expand test_client.py (NodeMaker) to hit all IFilesystemNode types.
2009-11-19 23:52:55 -08:00
Brian Warner
d2badbea78 class name cleanups: s/FileNode/ImmutableFileNode/
also fix test/bench_dirnode.py for recent dirnode changes
2009-11-19 23:22:39 -08:00
Brian Warner
6e7fb1006d Use DIR-IMM and t=mkdir-immutable for "tahoe backup", for #828 2009-11-18 11:28:13 -08:00
Brian Warner
834b20210a web/directory.py: use "DIR-IMM" to describe immutable directories, not DIR-RO 2009-11-18 11:18:32 -08:00
Brian Warner
e5452290f7 web/info.py: hush pyflakes 2009-11-18 11:17:36 -08:00
Brian Warner
e046744f40 make get_size/get_current_size consistent for all IFilesystemNode classes
* stop caching most_recent_size in dirnode, rely upon backing filenode for it
* start caching most_recent_size in MutableFileNode
* return None when you don't know, not "?"
* only render None as "?" in the web "more info" page
* add get_size/get_current_size to UnknownNode
2009-11-18 11:16:24 -08:00
Brian Warner
512fe3ad62 ImmutableDirectoryURIVerifier: fix verifycap handling 2009-11-18 08:42:38 -08:00
Brian Warner
f85690697a Add t=mkdir-immutable to the webapi. Closes #607.
* change t=mkdir-with-children to not use multipart/form encoding. Instead,
  the request body is all JSON. t=mkdir-immutable uses this format too.
* make nodemaker.create_immutable_dirnode() get convergence from SecretHolder,
  but let callers override it
* raise NotDeepImmutableError instead of using assert()
* add mutable= argument to DirectoryNode.create_subdirectory(), default True
2009-11-17 23:09:00 -08:00
Brian Warner
d2426ea3bd move convergence secret into SecretHolder, next to lease secret 2009-11-17 17:54:44 -08:00
Brian Warner
5fe713fc52 nodemaker: implement immutable directories (internal interface), for #607
* nodemaker.create_from_cap() now handles DIR2-CHK and DIR2-LIT
* client.create_immutable_dirnode() is used to create them
* no webapi yet
2009-11-11 16:22:33 -08:00
Brian Warner
cc422f8dc0 stop using IURI()/etc as an adapter 2009-11-11 14:45:42 -08:00
Brian Warner
131e05b155 clean up uri-vs-cap terminology, emphasize cap instances instead of URI strings
* "cap" means a python instance which encapsulates a filecap/dircap (uri.py)
 * "uri" means a string with a "URI:" prefix
 * FileNode instances are created with (and retain) a cap instance, and
   generate uri strings on demand
 * .get_cap/get_readcap/get_verifycap/get_repaircap return cap instances
 * .get_uri/get_readonly_uri return uri strings

* add filenode.download_to_filename() for control.py, should find a better way
* use MutableFileNode.init_from_cap, not .init_from_uri
* directory URI instances: use get_filenode_cap, not get_filenode_uri
* update/cleanup bench_dirnode.py to match, add Makefile target to run it
2009-11-11 14:26:19 -08:00
Brian Warner
f47672d12a add parser for immutable directory caps: DIR2-CHK, DIR2-LIT, DIR2-CHK-Verifier 2009-11-04 10:13:51 -08:00
Zooko O'Whielacronx
9733201c0b wui: s/TahoeLAFS/Tahoe-LAFS/ 2009-10-28 18:50:50 -08:00
Zooko O'Whielacronx
1be4d45779 tests: bump up the timeout on test_repairer to see if 120 seconds was too short for François's ARM box to do the test even when it was doing it right. 2009-10-27 13:48:00 -08:00
Brian Warner
2695af91a7 dirnode.pack_children(): add deep_immutable= argument
This will be used by DIR2:CHK to enforce the deep-immutability requirement.
2009-10-26 09:28:09 -07:00
Brian Warner
768c76aa5f webapi: use t=mkdir-with-children instead of a children= arg to t=mkdir .
This is safer: in the earlier API, an old webapi server would silently ignore
the initial children, and clients trying to set them would have to fetch the
newly-created directory to discover the incompatibility. In the new API,
clients using t=mkdir-with-children against an old webapi server will get a
clear error.
2009-10-25 18:13:21 -07:00
Brian Warner
1273b5c233 nodemaker.create_new_mutable_directory: pack_children() in initial_contents=
instead of creating an empty file and then adding the children later.

This should speed up mkdir(initial_children) considerably, removing two
roundtrips and an entire read-modify-write cycle, probably bringing it down
to a single roundtrip. A quick test (against the volunteergrid) suggests a
30% speedup.

test_dirnode: add new tests to enforce the restrictions that interfaces.py
claims for create_new_mutable_directory(): no UnknownNodes, metadata dicts
2009-10-19 17:51:18 -07:00
Brian Warner
59ee02e82f test_dirnode.py: add tests of initial_children= args to client.create_dirnode
and nodemaker.create_new_mutable_directory
2009-10-17 12:41:59 -07:00
Brian Warner
b4ec86c95a update many dirnode interfaces to accept dict-of-nodes instead of dict-of-caps
interfaces.py: define INodeMaker, document argument values, change
               create_new_mutable_directory() to take dict-of-nodes. Change
               dirnode.set_nodes() and dirnode.create_subdirectory() too.
nodemaker.py: use INodeMaker, update create_new_mutable_directory()
client.py: have create_dirnode() delegate initial_children= to nodemaker
dirnode.py (Adder): take dict-of-nodes instead of list-of-nodes, which
                    updates set_nodes() and create_subdirectory()
web/common.py (convert_initial_children_json): create dict-of-nodes
web/directory.py: same
web/unlinked.py: same
test_dirnode.py: update tests to match
2009-10-17 12:28:29 -07:00
Brian Warner
8a7c980e37 dirnode.py: move pack_children() out to a function, for eventual use by others 2009-10-17 11:07:07 -07:00
Brian Warner
ea373de042 move dirnode.CachingDict to dictutil.AuxValueDict, generalize method names,
improve tests. Let dirnode _pack_children accept either dict or AuxValueDict.
2009-10-17 11:00:05 -07:00
Brian Warner
e63f59f50c test/common.py: update FakeMutableFileNode to new contents= callable scheme 2009-10-12 22:21:54 -07:00
Brian Warner
480e1d318d The initial_children= argument to nodemaker.create_new_mutable_directory is
now enabled.
2009-10-12 20:19:22 -07:00
Brian Warner
c2520e4ec7 client.create_mutable_file(contents=) now accepts a callable, which is
invoked with the new MutableFileNode and is supposed to return the initial
contents. This can be used by e.g. a new dirnode which needs the filenode's
writekey to encrypt its initial children.

create_mutable_file() still accepts a bytestring too, or None for an empty
file.
2009-10-12 20:12:32 -07:00
Brian Warner
b30041c5ec webapi: t=mkdir now accepts initial children, using the same JSON that t=json
emits.

client.create_dirnode(initial_children=) now works.
2009-10-12 19:34:44 -07:00
Brian Warner
cf65cc2ae3 replace dirnode.create_empty_directory() with create_subdirectory(), which
takes an initial_children= argument
2009-10-12 19:15:20 -07:00
Brian Warner
d079eb45f6 dirnode.set_children: change return value: fire with self instead of None 2009-10-12 18:50:26 -07:00
Brian Warner
f871c3bb3d dirnode.set_nodes: change return value: fire with self instead of None 2009-10-12 18:45:46 -07:00
Brian Warner
304aadd4f7 dirnode.set_children: take a dict, not a list 2009-10-12 17:24:40 -07:00
Brian Warner
e2ffc3dc03 dirnode.set_uri/set_children: change signature to take writecap+readcap
instead of a single cap. The webapi t=set_children call benefits too.
2009-10-12 16:51:26 -07:00
Brian Warner
3ee740628a replace Client.create_empty_dirnode() with create_dirnode(), in anticipation
of adding initial_children= argument.

Includes stubbed-out initial_children= support.
2009-10-12 15:45:06 -07:00
Brian Warner
576c47d61e test_web.py: use a less-fake client, making test harness smaller 2009-10-12 15:28:08 -07:00
Brian Warner
f4aa418086 Verifier: check the full cryptext-hash tree on each share. Removed .todos
from the last few test_repairer tests that were waiting on this.
2009-10-05 15:18:49 -07:00
Brian Warner
504c767d03 Verifier: check the full block-hash-tree on each share
Removed the .todo from two test_repairer tests that check this. The only
remaining .todos are on the three crypttext-hash-tree tests.
2009-10-05 14:48:44 -07:00
Brian Warner
e8f56af5a7 Verifier: check the full share-hash chain on each share
Removed the .todo from two test_repairer tests that check this.
2009-10-05 14:34:43 -07:00
Brian Warner
bbd00de495 test_repairer: rename Verifier test cases to be more precise and less verbose 2009-10-05 13:11:15 -07:00
Brian Warner
be95129833 immutable/checker.py: rearrange code a little bit, make it easier to follow 2009-10-05 13:02:52 -07:00
Brian Warner
9dba4ec641 test/common.py: wrap docstrings to 80cols so I can read them more easily 2009-10-05 13:01:43 -07:00
Brian Warner
19d336513c immutable/download.py: wrap to 80cols, no functional changes 2009-10-05 12:25:42 -07:00
Zooko O'Whielacronx
00a44c057a docs: a few licensing clarifications requested by Ubuntu 2009-09-26 20:32:26 -07:00
Zooko O'Whielacronx
aaaa633f18 setup: remove a convenience copy of figleaf, to ease inclusion into Ubuntu Karmic Koala
We need to carefully document the licence of figleaf in order to get Tahoe-LAFS into Ubuntu Karmic Koala.  However, figleaf isn't really a part of Tahoe-LAFS per se -- this is just a "convenience copy" of a development tool.  The quickest way to make Tahoe-LAFS acceptable for Karmic then, is to remove figleaf from the Tahoe-LAFS tarball itself.  People who want to run figleaf on Tahoe-LAFS (as everyone should want) can install figleaf themselves.  I haven't tested this -- there may be incompatibilities between upstream figleaf and the copy that we had here...
2009-09-23 22:32:15 -07:00
Zooko O'Whielacronx
9f0da4012e docs: Shawn Willden grants permission for his contributions under GPL2+|TGPPL1+ 2009-09-21 09:46:51 -07:00
Zooko O'Whielacronx
603e08c4a9 doc: licensing cleanups
Use nice utf-8 © instead of "(c)". Remove licensing statements on utility modules that have been assigned to allmydata.com by their original authors. (Nattraverso was not assigned to allmydata.com -- it was LGPL'ed -- but I checked and src/allmydata/util/iputil.py was completely rewritten and doesn't contain any line of code from nattraverso.)  Add notes to misc/debian/copyright about licensing on files that aren't just allmydata.com-licensed.
2009-09-20 10:16:31 -07:00
Brian Warner
14cb5ca436 test/check_speed.py: fix shbang line 2009-08-17 17:59:48 -07:00
Brian Warner
5283d4c19e de-Service-ify Helper, pass in storage_broker and secret_holder directly.
This makes it more obvious that the Helper currently generates leases with
the Helper's own secrets, rather than getting values from the client, which
is arguably a bug that will likely be resolved with the Accounting project.
2009-08-15 13:17:37 -07:00
Brian Warner
4a4a4f9520 immutable.Downloader: pass StorageBroker to constructor, stop being a Service
child of the client, access with client.downloader instead of
client.getServiceNamed("downloader"). The single "Downloader" instance is
scheduled for demolition anyways, to be replaced by individual
filenode.download calls.
2009-08-15 12:25:43 -07:00
Zooko O'Whielacronx
22c962bbc2 tests: double the timeout on test_runner.RunNode.test_introducer since feisty hit a timeout
I'm not sure if this is an actual timing issue (feisty is running on an overloaded VM if I recall correctly), or it there is a deeper bug.
2009-08-15 09:05:12 -07:00
Brian Warner
d214f7eebe stop making History be a Service, it wasn't necessary 2009-08-15 04:44:15 -07:00
Brian Warner
0d5dc51617 Overhaul IFilesystemNode handling, to simplify tests and use POLA internally.
* stop using IURI as an adapter
* pass cap strings around instead of URI instances
* move filenode/dirnode creation duties from Client to new NodeMaker class
* move other Client duties to KeyGenerator, SecretHolder, History classes
* stop passing Client reference to dirnode/filenode constructors
  - pass less-powerful references instead, like StorageBroker or Uploader
* always create DirectoryNodes by wrapping a filenode (mutable for now)
* remove some specialized mock classes from unit tests

Detailed list of changes (done one at a time, then merged together)

always pass a string to create_node_from_uri(), not an IURI instance
always pass a string to IFilesystemNode constructors, not an IURI instance
stop using IURI() as an adapter, switch on cap prefix in create_node_from_uri()
client.py: move SecretHolder code out to a separate class
test_web.py: hush pyflakes
client.py: move NodeMaker functionality out into a separate object
LiteralFileNode: stop storing a Client reference
immutable Checker: remove Client reference, it only needs a SecretHolder
immutable Upload: remove Client reference, leave SecretHolder and StorageBroker
immutable Repairer: replace Client reference with StorageBroker and SecretHolder
immutable FileNode: remove Client reference
mutable.Publish: stop passing Client
mutable.ServermapUpdater: get StorageBroker in constructor, not by peeking into Client reference
MutableChecker: reference StorageBroker and History directly, not through Client
mutable.FileNode: removed unused indirection to checker classes
mutable.FileNode: remove Client reference
client.py: move RSA key generation into a separate class, so it can be passed to the nodemaker
move create_mutable_file() into NodeMaker
test_dirnode.py: stop using FakeClient mockups, use NoNetworkGrid instead. This simplifies the code, but takes longer to run (17s instead of 6s). This should come down later when other cleanups make it possible to use simpler (non-RSA) fake mutable files for dirnode tests.
test_mutable.py: clean up basedir names
client.py: move create_empty_dirnode() into NodeMaker
dirnode.py: get rid of DirectoryNode.create
remove DirectoryNode.init_from_uri, refactor NodeMaker for customization, simplify test_web's mock Client to match
stop passing Client to DirectoryNode, make DirectoryNode.create_with_mutablefile the normal DirectoryNode constructor, start removing client from NodeMaker
remove Client from NodeMaker
move helper status into History, pass History to web.Status instead of Client
test_mutable.py: fix minor typo
2009-08-15 04:28:46 -07:00
Zooko O'Whielacronx
2a63fc9159 docs: update relnotes.txt, relnotes-short.txt, and others documentation bits for v1.5.0 release! 2009-08-01 19:57:10 -07:00
Brian Warner
ad45511156 ftpd: update docs, point to Twisted ticket for the proposed patch 2009-07-31 11:32:26 -07:00
Brian Warner
0d0e0ba2fb test_runner.py: tolerate symlinks in the source tree directory's ancestor path 2009-07-30 10:17:24 -07:00
Brian Warner
bf1e61c8f3 Touch up #705 changes:
webapi.txt: clarify replace=only-files argument, mention replace= on POST t=uri
 test_cli.py: insert whitespace between logical operations
 web.common.parse_replace_arg: make it case-insensitive, to match the docs
2009-07-20 11:38:03 -04:00
Zooko O'Whielacronx
b76ca4549a iputil: added "openbsd3" => "bsd" and "openbsd4" => "bsd" per tdelaet's report; should fix ticket #768 2009-07-20 06:13:14 -07:00
kevan
0d8b1e29fa Add tests for tahoe mv behavior 2009-07-19 20:46:09 -07:00
kevan
4331326b21 Add tests for tahoe mv help text 2009-07-19 20:45:53 -07:00
kevan
52aceb1a8e Implement more clearly defined moving semantics in tahoe_mv.py 2009-07-19 20:45:23 -07:00
kevan
36f2e01275 Update tahoe mv help text. 2009-07-19 20:45:03 -07:00
kevan
8eb7ddab6b Add tests for parse_replace_arg() 2009-07-19 20:47:05 -07:00
kevan
40360a7a9c Add tests for new PUT behavior 2009-07-19 20:46:32 -07:00
kevan
18a80d99b1 Alter Adder + Adder tests to look for 'only-files' instead of 'only_files' 2009-07-19 20:43:18 -07:00
kevan
74207d8334 Alter filenode.py to use parse_replace_arg 2009-07-19 20:48:31 -07:00
kevan
90677745b3 Alter directory.py to use parse_replace_arg() 2009-07-19 20:47:46 -07:00
kevan
7ab92c7511 Add a function to parse arguments for the replace parameter 2009-07-19 20:47:23 -07:00
Zooko O'Whielacronx
7607268a58 trivial: remove obsolete comment text 2009-07-20 06:08:43 -07:00
Zooko O'Whielacronx
c6fce3a1c6 directories: minor change: fix warning message from the benchmarking script 2009-07-19 17:21:14 -07:00
kevan
ca4de9ee97 Add unit tests for the Adder in dirnode.py 2009-07-18 12:50:49 -07:00
kevan
c476c66b0e Add 'only_files' option to the overwrite field in Adder 2009-07-17 20:00:10 -07:00
Brian Warner
531cc7899f rename NewDirectoryNode to DirectoryNode, NewDirectoryURI to DirectoryURI 2009-07-17 17:15:49 -05:00