Commit Graph

264 Commits

Author SHA1 Message Date
Brian Warner
4bb88fd2ee dirnode: refactor recursive-traversal methods, add stats to deep_check() method results and t=deep-check webapi 2008-09-10 01:45:04 -07:00
Brian Warner
28a9f8f076 test_system: check t=deep-stats too 2008-09-09 23:54:57 -07:00
Brian Warner
12ff8a6eae test_system: add deep-check-JSON tests, fix a bug 2008-09-09 23:14:16 -07:00
Brian Warner
ce7fcbde36 test_system: oops, re-enable some tests that got bypassed 2008-09-09 23:02:45 -07:00
Brian Warner
ba336aed3e test_system: add deep-stats test 2008-09-09 22:56:34 -07:00
Brian Warner
80d8f3e862 hush pyflakes 2008-09-09 19:50:17 -07:00
Brian Warner
1d2d6a35a6 checker results: add output=JSON to webapi, add tests, clean up APIs
to make the internal ones use binary strings (nodeid, storage index) and
the web/JSON ones use base32-encoded strings. The immutable verifier is
still incomplete (it returns imaginary healty results).
2008-09-09 19:45:17 -07:00
Brian Warner
04513e3ac5 immutable verifier: provide some dummy results so deep-check works, make the tests ignore these results until we finish it off 2008-09-09 18:08:27 -07:00
Brian Warner
7fb3308498 mutable checker: even more tests. Everything in ICheckerResults should be covered now, except for immutable-verify which is incomplete 2008-09-09 17:57:06 -07:00
Brian Warner
84a5778507 checker results: more tests, update interface docs 2008-09-09 17:30:10 -07:00
Brian Warner
f895e39d48 checker results: more tests, more results. immutable verifier tests are disabled until they emit more complete results 2008-09-09 17:15:46 -07:00
Brian Warner
90b934eb71 checker: add tests, add stub for immutable check_and_repair 2008-09-09 16:34:49 -07:00
Brian Warner
918b0543b9 test_system: make log() tolerate the format= form 2008-09-07 20:03:36 -07:00
Brian Warner
3408d552cd checker: overhaul checker results, split check/check_and_repair into separate methods, improve web displays 2008-09-07 12:44:56 -07:00
Brian Warner
1668401c16 mutable: make mutable-repair work for non-verifier runs, add tests 2008-08-26 16:34:54 -07:00
Brian Warner
014c9b5969 CLI: add 'tahoe debug corrupt-share', and use it for deep-verify tests, and fix non-deep web checker API to pass verify=true into node 2008-08-12 17:05:01 -07:00
Brian Warner
5c0bfd7681 test_system: rename Checker to ImmutableChecker, to make room for a mutable one 2008-08-12 15:59:32 -07:00
Brian Warner
67e7333955 'tahoe debug dump-share': add --offsets, to show section offsets 2008-08-12 14:46:56 -07:00
Brian Warner
bb33e3e4c2 CLI: move all debug commands (dump-share, dump-cap, find-shares, catalog-shares) into a 'debug' subcommand, and improve --help output 2008-08-12 13:37:32 -07:00
Brian Warner
3ad0168fbf test_system: factor out find_shares/replace_shares to a common class, so they can be used by other tests 2008-08-05 18:49:58 -07:00
Brian Warner
f7d2fcc233 CLI: change one-arg forms of 'tahoe put' to make an unlinked file, fix replace-mutable #441 2008-08-01 19:27:29 -07:00
Brian Warner
071cbec865 tests: simplify CLI tests that use stdin, now that runner supports it 2008-08-01 15:05:14 -07:00
Brian Warner
f181a0458a CLI: simplify argument-passing, use options= for everthing, including stdout 2008-08-01 11:46:24 -07:00
Zooko O'Whielacronx
6d3ec390b0 tests: add test that verifier notices any (randomly chosen) bit flipped in the verifiable part of any (randomly chosen) share
The currently verifier doesn't (usually) pass this randomized test, hence the TODO.
2008-07-30 17:20:15 -07:00
Zooko O'Whielacronx
86160c3fc9 tests: test that checker doesn't cause reads on the storage servers
It would still pass the test if it noticed a corrupted share.  (It won't
notice, of course.)  But it is required to do its work without causing storage
servers to read blocks from the filesystem.
2008-07-30 16:54:20 -07:00
Zooko O'Whielacronx
dbd1c2397e tests: use the handy dandy TestCase.mktemp() function from trial to give unique and nicely named directories for each testcase 2008-07-30 15:49:20 -07:00
Zooko O'Whielacronx
56282613c5 tests: add test_system.Checker which tests basic checking (without verification) functionality 2008-07-28 16:43:17 -07:00
Brian Warner
48c64b7a29 test_system.py: factor SystemTestMixin out of SystemTest 2008-07-25 15:33:49 -07:00
Brian Warner
10bbc740e9 test_system.py: modify system-test setup code in preparation for merge with common.SystemTestMixin 2008-07-25 15:29:31 -07:00
Brian Warner
766cdd9c52 test_system.py: move SystemTestMixin out into common.py, where further improvements will occur 2008-07-25 15:17:58 -07:00
Brian Warner
e317fcc08c test_system.py: create SystemTestMixin, with less cruft, for faster system-like tests 2008-07-25 15:13:00 -07:00
Brian Warner
7394607141 move encode/upload/download/checker.py into a new immutable/ directory. No behavior changes expected. 2008-07-16 13:14:39 -07:00
Brian Warner
94e619c1f6 overhaul checker invocation
Removed the Checker service, removed checker results storage (both in-memory
and the tiny stub of sqlite-based storage). Added ICheckable, all
check/verify is now done by calling the check() method on filenodes and
dirnodes (immutable files, literal files, mutable files, and directory
instances).

Checker results are returned in a Results instance, with an html() method for
display. Checker results have been temporarily removed from the wui directory
listing until we make some other fixes.

Also fixed client.create_node_from_uri() to create LiteralFileNodes properly,
since they have different checking behavior. Previously we were creating full
FileNodes with LIT uris inside, which were downloadable but not checkable.
2008-07-15 17:23:25 -07:00
Brian Warner
9b6f23ac05 'tahoe dump-share': show verify-cap too 2008-07-07 14:11:02 -07:00
Brian Warner
28f4652b96 break introducer up into separate modules in the new allmydata.introducer package 2008-06-18 12:24:16 -07:00
Brian Warner
b6c2f80170 test_system: even more 'cp -r' coverage 2008-05-21 18:40:49 -07:00
Brian Warner
4824d058cc test_system.py: improve 'cp -r' coverage: exercise copy from tahoe to local disk 2008-05-21 18:36:25 -07:00
Brian Warner
0f8f213baf CLI: remove the '-r' shortcut for --dir-cap, to make it available for cp -r 2008-05-21 17:55:32 -07:00
Brian Warner
457248f023 tahoe_cp: rewrite, make --recursive work 2008-05-21 17:35:21 -07:00
Brian Warner
15fbe05092 tahoe_ls: improve error message when the target is missing 2008-05-21 17:34:52 -07:00
Brian Warner
7afd869854 cli: initial implementation of 'cp -r', probably doesn't work yet 2008-05-21 11:49:22 -07:00
Brian Warner
0df663b7ae CLI: implement the easy part of cp (no -r, only two arguments) 2008-05-20 16:56:03 -07:00
Brian Warner
1d8e452184 CLI: add 'list-aliases', factor out get_aliases 2008-05-20 14:36:04 -07:00
Brian Warner
1236bc3408 CLI: add 'ln', just like move but without the delete 2008-05-20 13:30:31 -07:00
Brian Warner
f09f92c371 test_system.py: remove dead code 2008-05-20 13:20:48 -07:00
Brian Warner
a8ed3208ba CLI ls: add --readonly-uri to display readonly URIs for all children 2008-05-20 12:49:11 -07:00
Brian Warner
e889463f0c CLI: add put --mutable, enhance ls to show mutable vs immutable as rw/r- 2008-05-20 12:36:55 -07:00
Brian Warner
88457fbc32 test_web/test_system: improve test coverage 2008-05-19 23:28:52 -07:00
Brian Warner
506ce2dfd0 test_system: fix windows test failure, open files with binary mode 2008-05-19 20:24:12 -07:00
Brian Warner
8e92dfcb50 overhaul CLI: not quite complete but it works a lot better than it used to. The new scheme uses 'tahoe add-alias' and rsync/scp-style 'alias:foo/bar.txt' arguments 2008-05-19 19:28:50 -07:00
Brian Warner
b1d2922ea3 test_system: match webapi change, making new files now returns 201 Created, not 200 OK 2008-05-19 14:09:31 -07:00
Brian Warner
26187bfc81 use a weakref cache in the client to manage singleton filenodes/dirnodes, to avoid autocollision. Should close #391. 2008-05-08 18:02:55 -07:00
Brian Warner
fabdc28c06 deep-stats: add file-size histogram 2008-05-08 16:19:42 -07:00
Brian Warner
acdd98432d hush pyflakes warning about code that got moved in the recent StallMixin refactoring 2008-04-22 17:14:26 -07:00
Brian Warner
81ab1ec785 testutil: factor stall() out into a common location 2008-04-22 16:47:15 -07:00
Brian Warner
a379690b04 mutable: replace MutableFileNode API, update tests. Changed all callers to use overwrite(), but that will change soon 2008-04-17 17:51:38 -07:00
Brian Warner
e1838ba217 mutable WIP: clean up status handling, shrink the code a lot, improve test coverage 2008-04-17 13:02:22 -07:00
Brian Warner
a1670497a8 mutable WIP: add servermap update status pages 2008-04-16 19:05:41 -07:00
Brian Warner
1b4b4cbd4a mutable WIP: rename NotEnoughPeersError to NotEnoughSharesError 2008-04-15 16:08:32 -07:00
Brian Warner
cf84bae850 mutable WIP: merge conflicts in test_system.py 2008-04-15 15:57:28 -07:00
Brian Warner
d4230d1781 mutable WIP: split mutable.py into separate files. All tests pass. 2008-04-11 14:31:16 -07:00
Brian Warner
0515b1099c mutable WIP: all tests pass, but publish/retrieve status is still stubbed out 2008-04-10 19:26:35 -07:00
Brian Warner
6de2a0794f web: don't break status page when there is no helper running 2008-04-14 23:42:20 -07:00
Brian Warner
86578bc3cd webish: fix 'not running helper' status indicator on welcome page, add tests 2008-04-14 16:28:11 -07:00
Brian Warner
5b8320442a stats: add /statistics web page to show them, add tests 2008-04-14 14:17:08 -07:00
Brian Warner
500934b72f helper stats: fix the /helper_status page, the recent conflict merging missed some uses. Added tests, updated the munin plugins to match 2008-04-14 13:18:53 -07:00
robk-tahoe
1ae2c39862 key_generator: service related cleanups, incorporation into system test
this cleans up KeyGenerator to be a service (a subservice of the
KeyGeneratorService as instantiated by the key-generator.tac app)
this means that the timer which replenishes the keypool will be
shutdown cleanly when the service is stopped.

adds checks on the key_generator service and client into the system
test 'test_mutable' such that one of the nodes (clients[3]) uses
the key_generator service, and checks that mutable file creation
in that node, via a variety of means, are all consuming keys from
the key_generator.
2008-04-03 15:57:07 -07:00
Brian Warner
4531d1e953 introweb.py: add ?t=json, to provide machine-readable subscriber counts 2008-03-25 12:56:12 -07:00
Brian Warner
1e097766c9 disable plaintext hashes in shares, but leave a switch to turn it back on 2008-03-24 13:39:51 -07:00
Brian Warner
7b21054c33 UNDO: upload: stop putting plaintext and ciphertext hashes in shares.
This removes the guess-partial-information attack vector, and reduces
the amount of overhead that we consume with each file. It also introduces
a forwards-compability break: older versions of the code (before the
previous download-time "make hashes optional" patch) will be unable
to read files uploaded by this version, as they will complain about the
missing hashes. This patch is experimental, and is being pushed into
trunk to obtain test coverage. We may undo it before releasing 1.0.
2008-03-23 15:35:54 -07:00
Zooko O'Whielacronx
fc3bd0c987 use added secret to protect convergent encryption
Now upload or encode methods take a required argument named "convergence" which can be either None, indicating no convergent encryption at all, or a string, which is the "added secret" to be mixed in to the content hash key.  If you want traditional convergent encryption behavior, set the added secret to be the empty string.

This patch also renames "content hash key" to "convergent encryption" in a argument names and variable names.  (A different and larger renaming is needed in order to clarify that Tahoe supports immutable files which are not encrypted content-hash-key a.k.a. convergent encryption.)

This patch also changes a few unit tests to use non-convergent encryption, because it doesn't matter for what they are testing and non-convergent encryption is slightly faster.
2008-03-24 09:46:06 -07:00
Brian Warner
7996131a0a upload: stop putting plaintext and ciphertext hashes in shares.
This removes the guess-partial-information attack vector, and reduces
the amount of overhead that we consume with each file. It also introduces
a forwards-compability break: older versions of the code (before the
previous download-time "make hashes optional" patch) will be unable
to read files uploaded by this version, as they will complain about the
missing hashes. This patch is experimental, and is being pushed into
trunk to obtain test coverage. We may undo it before releasing 1.0.
2008-03-23 15:35:54 -07:00
Brian Warner
2ef70ab814 mutable.py: split replace() into update() and overwrite(). Addresses #328. 2008-03-12 18:00:43 -07:00
Brian Warner
89be2e1bea introweb: combine announcement and subscriber information to show version+nickname for each client 2008-03-11 19:21:29 -07:00
Brian Warner
810ba68343 add a webserver for the Introducer, showing service announcements and subscriber lists 2008-03-11 17:36:25 -07:00
Brian Warner
f6ca62df31 test_system: improve test coverage of publish/retrieve status 2008-03-04 01:24:35 -07:00
Brian Warner
7e159feb27 stats: make StatsGatherer happy about sharing a process with other services, add one during system test to get some test coverage 2008-03-03 23:55:58 -07:00
Brian Warner
3ecb483e35 test_system: add test coverage for download-status and upload-status 2008-03-03 21:37:17 -07:00
Brian Warner
1a7651ce82 retain 10 most recent upload/download status objects, show them in /status . Prep for showing individual status objects 2008-02-29 22:19:03 -07:00
Brian Warner
c3a1491cf4 test_system.py: improve coverage of webish.py 2008-02-15 04:02:50 -07:00
Brian Warner
7927495cbe unicode handling: declare dirnodes to contain unicode child names, update webish to match 2008-02-14 15:45:56 -07:00
Brian Warner
a050204833 add test coverage for the /stats web page 2008-02-13 13:57:39 -07:00
Zooko O'Whielacronx
3f8df27063 use base62 encoding for storage indexes, on disk and in verifier caps, and in logging and diagnostic tools
base62 encoding fits more information into alphanumeric chars while avoiding the troublesome non-alphanumeric chars of base64 encoding.  In particular, this allows us to work around the ext3 "32,000 entries in a directory" limit while retaining the convenient property that the intermediate directory names are leading prefixes of the storage index file names.
2008-02-12 20:48:37 -07:00
Brian Warner
ca27b8e5ad add 'tahoe catalog-shares' tool, to make a one-line summary of each share file. This can help do cross-server correlation of sharefiles, looking for anomalies 2008-02-11 18:17:01 -07:00
Brian Warner
7123c9d875 test_system.py: refactor bounce_client, probably make it stop failing on cygwin 2008-02-11 15:26:58 -07:00
Brian Warner
873dee53e9 test_system: remove the hackish debug_interrupt= attribute magic used to exercise interrupted-upload resumption, instead just make the Uploadable bounce the helper halfway through the upload 2008-02-07 20:15:37 -07:00
Brian Warner
3bd79917b2 test_system.py: remove that ugly debug_stash_RemoteencryptedUploadable hack, now that UploadResults give us a better approach 2008-02-07 17:27:30 -07:00
Brian Warner
4d4073fb16 test_system: deferred-handling logic in test_upload_and_download was broken 2008-02-06 14:05:11 -07:00
Brian Warner
2dcac796e9 add 'tahoe find-shares' command, to locate share files on a local node's disk 2008-02-06 13:19:51 -07:00
Brian Warner
e92a2b5ab2 dump-cap: include UEB_hash in output 2008-02-06 12:48:19 -07:00
Brian Warner
66f33ee504 upload: return an UploadResults instance (with .uri) instead of just a URI 2008-02-05 21:01:38 -07:00
Brian Warner
5103bf8148 storage: change service name from 'storageserver' to 'storage' 2008-02-05 20:28:59 -07:00
Brian Warner
d146ef7e09 webish: add extra introducer data (version, timestamps) to Welcome page 2008-02-05 17:32:27 -07:00
Brian Warner
daecca6589 big introducer refactoring: separate publish+subscribe. Addresses #271. 2008-02-05 13:05:13 -07:00
Zooko O'Whielacronx
79c439d026 storage: make two levels of share directories so as not to exceed certain filesystems's limitations on directory size
The filesystem which gets my vote for most undeservedly popular is ext3, and it has a hard limit of 32,000 entries in a directory.  Many other filesystems (even ones that I like more than I like ext3) have either hard limits or bad performance consequences or weird edge cases when you get too many entries in a single directory.

This patch makes it so that there is a layer of intermediate directories between the "shares" directory and the actual storage-index directory (the one whose name contains the entire storage index (z-base-32 encoded) and which contains one or more share files named by their share number).

The intermediate directories are named by the first 14 bits of the storage index, which means there are at most 16384 of them.  (This also means that the intermediate directory names are not a leading prefix of the storage-index directory names -- to do that would have required us to have intermediate directories limited to either 1024 (2-char), which is too few, or 32768 (3-chars of a full 5 bits each), which would overrun ext3's funny hard limit of 32,000.))

This closes #150, and please see the "convertshares.py" script attached to #150 to convert your old tahoe-0.7.0 storage/shares directory into a new tahoe-0.8.0 storage/shares directory.
2008-01-31 16:26:28 -07:00
Brian Warner
81eeafc574 upload-helper: avoid duplicate uploads: check the grid to see if the file already exists 2008-01-30 18:49:02 -07:00
Zooko O'Whielacronx
ca971559e6 make content-hash-key encryption a parameter of uploading
fixes #293
2008-01-30 12:24:50 -07:00
robk-tahoe
7c34658438 offloaded: fix failure in unit test on windows
in trying to test my fix for the failure of the offloaded unit test on windows
(by closing the reader before unlinking the encoding file - which, perhaps
disturbingly doesn't actually make a difference in my windows environment)
I was unable too because the unit test failed every time with a connection lost
error.

after much more time than I'd like to admit it took, I eventually managed to
track that down to a part of the unit test which is supposed to be be dropping
a connection.   it looks like the exceptions that get thrown on unix, or at
least all the specific environments brian tested in, for that dropped 
connection are different from what is thrown on my box (which is running py2.4
and twisted 2.4.0, for reference)  adding ConnectionLost to the list of
expected exceptions makes the test pass.

though curiously still my test logs a NotEnoughWritersError error, and I'm not
currently able to fathom why that exception isn't leading to any overall 
failure of the unit test itself.

for general interest, a large part of the time spent trying to track this down
was lost to the state of logging.  I added a whole bunch of logging to try
and track down where the tests were failing, but then spent a bunch of time
searching in vain for that log output.  as far as I can tell at this point
the unit tests are themselves logging to foolscap's log module, but that isn't
being directed anywhere, so all the test's logging is being black holed.
2008-01-17 20:57:29 -07:00
Brian Warner
fd0dc3013c offloaded: update unit tests: assert that interrupt/resume works, and that the helper deletes tempfiles 2008-01-17 01:18:10 -07:00
Brian Warner
51321944f0 megapatch: overhaul encoding_parameters handling: now it comes from the Uploadable, or the Client. Removed options= too. Also move helper towards resumability. 2008-01-16 03:03:35 -07:00