ExternalVendorCode/tahoe-lafs
1
0
Fork 0
mirror of https://github.com/tahoe-lafs/tahoe-lafs.git synced 2025-01-02 19:26:44 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
4,504 Commits 185 Branches 90 Tags 89 MiB
4712875193
Commit Graph

2590 Commits

Author SHA1 Message Date
Brian Warner
24ab5ec26f expirer: add mode to expire only-mutable or only-immutable shares 2009-03-16 23:51:18 -07:00
Brian Warner
475e7b8804 tahoe_ls.py: add comment about error cases to improve 2009-03-16 22:12:06 -07:00
Brian Warner
c7254c5f1d GC: add date-cutoff -based expiration, add proposed docs 2009-03-16 22:10:41 -07:00
Brian Warner
2b525a42d3 test_deepcheck: remove the 10s timeout: our dapper buildslave requires 30s, and the reduced timeout was only there because this tests fails by timeout rather than explicitly 2009-03-15 17:51:34 -07:00
Brian Warner
f0b0ad1c8f tahoe cp -r: add --caps-only flag, to write filecaps into local files instead of actual file contents. Used only for debugging and as a quick tree-comparison tool. 2009-03-15 16:19:58 -07:00
Brian Warner
6e57576f2e dirnode deep_traverse: insert a turn break (fireEventually) at least once every 100 files, otherwise a CHK followed by more than 158 LITs can overflow the stack, sort of like #237. 2009-03-13 16:31:35 -07:00
Brian Warner
e59164e0bb consolidate: remove pointless 'else' after for loop 2009-03-13 01:27:51 -07:00
Brian Warner
ae64ddf8e6 consolidate: add eta, flush stdout 2009-03-13 01:24:51 -07:00
Brian Warner
06bf443e91 consolidate: tolerate unicode dirnames 2009-03-12 23:54:02 -07:00
Brian Warner
1a741fdb03 dirnode.py: when doing deep-traverse, walk each directory in alphabetical order, to make things like 'manifest' more predictable 2009-03-12 23:50:46 -07:00
Brian Warner
a18f8d4cc7 consolidator: add progress to scan-old-directory passes 2009-03-12 22:47:28 -07:00
Brian Warner
ddb2f1a6e3 consolidator: fix cycle detection to not trigger on merely shared directories, add snapshot counter to progress 2009-03-12 21:22:29 -07:00
Brian Warner
ddc9a7ae74 consolidator: re-use more directories, add total directories seen-vs-used counts 2009-03-12 20:48:01 -07:00
Brian Warner
809ec25ffa tahoe_backup.py: tolerate more time formats 2009-03-12 18:16:00 -07:00
Brian Warner
1c24707f19 consolidator: add more verbose traversal of directories 2009-03-12 16:29:00 -07:00
Brian Warner
760688a224 consolidate: create multiple numbered backups of the original Archives directory, not just the first time 2009-03-12 16:04:27 -07:00
Brian Warner
c4c6a62954 add 'tahoe debug consolidate' command, to merge directories created by repeated 'tahoe cp -r' or the allmydata win32 backup tool, into the form that would have been created by 'tahoe backup'. 2009-03-12 13:56:06 -07:00
Brian Warner
969b758bf5 tahoe_cp.py: return 0 for success, instead of None 2009-03-12 13:53:45 -07:00
Brian Warner
1ccd426a34 expirer: fix prediction math, thanks to Zandr for the catch 2009-03-09 13:42:17 -07:00
Brian Warner
8708045a98 storage.expirer: oops, fix upgrade-handler code 2009-03-08 20:55:16 -07:00
Brian Warner
df3f7f93e0 storage.expirer: handle upgrades better 2009-03-08 20:42:20 -07:00
Brian Warner
a68ad06254 storage.expirer: exercise the last missing line of webstatus code 2009-03-08 20:38:28 -07:00
Brian Warner
6d6049430c expirer: tolerate corrupt shares, add them to the state and history for future examination 2009-03-08 20:08:40 -07:00
Brian Warner
1a98521c3d storage/immutable: raise a specific error upon seeing a bad version number, instead of using assert. Also wrap to 80cols. 2009-03-08 20:07:32 -07:00
Brian Warner
02b40ec499 storage: add test coverage for BucketReader.__repr__ 2009-03-08 20:05:27 -07:00
Brian Warner
6d7319c588 storage/mutable: raise a specific error upon seeing bad magic, instead of using assert 2009-03-08 19:02:01 -07:00
Zooko O'Whielacronx
7e7167f2e5 nodeadmin: node stops itself if a hotline file hasn't been touched in 120 seconds now, instead of in 60 seconds 
A test failed on draco (MacPPC) because it took 67.1 seconds to get around to running the test, and the node had already stopped itself when the hotline file was 60 seconds old.
 2009-03-07 20:50:39 -07:00
Brian Warner
5675b4e7e0 expirer: make web display a bit more consistent 2009-03-07 16:14:42 -07:00
Brian Warner
df045650e0 web/storage.py: tolerate unknown-future displays, I'm not sure why LeaseCrawler.test_unpredictable_future didn't catch this 2009-03-07 16:02:43 -07:00
Brian Warner
192c161be7 tahoe_cp.py: improve error reporting slightly: don't json-interpret HTTP errors, pass through tahoe webapi error messages 2009-03-07 05:40:51 -07:00
Brian Warner
2c2c3c2295 test_cli.py: assert that 'ls' on an unrecoverable file now gives a better error message 2009-03-07 05:08:15 -07:00
Brian Warner
67d7600781 test_web: test behavior of broken-dirnode GET, both html and json 2009-03-07 04:57:07 -07:00
Brian Warner
950200fece web: when a dirnode can't be read, emit a regular HTML page but with the child-table and upload-forms replaced with an apologetic message. Make sure to include the 'get info' links so the user can do a filecheck 2009-03-07 04:56:01 -07:00
Brian Warner
badd79671c web/common: split out exception-to-explanation+code mapping to a separate humanize_failure() function, so it can be used by other code. Add explanation for mutable UnrecoverableFileError. 2009-03-07 04:54:08 -07:00
Brian Warner
9c7fafd5af test_cli: use explicit (and stable) testdirs, instead of using self.mktemp 2009-03-07 03:04:28 -07:00
Brian Warner
091cfd04ca test_storage: solaris doesn't appear to give the same block count as other platforms, so don't assert as much about 'diskbytes' recovered 2009-03-07 02:45:18 -07:00
Brian Warner
24ebdb3954 tahoe_cp.py: don't be verbose unless --verbose is passed: default is one line of success 2009-03-07 02:44:21 -07:00
Brian Warner
0dee2a6036 storage: add a lease-checker-and-expirer crawler, plus web status page. 
This walks slowly through all shares, examining their leases, deciding which
are still valid and which have expired. Once enabled, it will then remove the
expired leases, and delete shares which no longer have any valid leases. Note
that there is not yet a tahoe.cfg option to enable lease-deletion: the
current code is read-only. A subsequent patch will add a tahoe.cfg knob to
control this, as well as docs. Some other minor items included in this patch:

 tahoe debug dump-share has a new --leases-only flag
 storage sharefile/leaseinfo code is cleaned up
 storage web status page (/storage) has more info, more tests coverage
 space-left measurement on OS-X should be more accurate (it was off by 2048x)
  (use stat .f_frsize instead of f_bsize)
 2009-03-06 22:45:17 -07:00
Zooko O'Whielacronx
e9199a89ab trivial: remove unused import detected by pyflakes, and remove trailing whitespace 2009-03-05 16:32:04 -07:00
Zooko O'Whielacronx
5e90d82a02 trivial: use more specific function for ascii-encoding storage index 2009-02-22 11:57:51 -07:00
Brian Warner
c5ecb7b3ba test_cli: validate non-HTML error response of 'tahoe get' on an unrecoverable file 2009-03-03 22:11:46 -07:00
Brian Warner
f42e3bb107 web: full patch for HTML-vs-plaintext traceback renderings, improve test coverage of exception rendering 2009-03-03 21:56:30 -07:00
Brian Warner
90226f335f web/common.py: use 'Accept:' header to control HTML-vs-text/plain traceback renderings 2009-03-03 21:54:57 -07:00
Brian Warner
53f50fcc78 test/common: rearrange shouldHTTPError 2009-03-03 19:40:59 -07:00
Brian Warner
c4bda3daa3 web: move plural() to common.py 2009-03-03 19:40:19 -07:00
Brian Warner
67571eb033 add more information to NotEnoughSharesError, split out new exceptions for no-servers and no-source-of-ueb-hash 2009-03-03 19:37:15 -07:00
Brian Warner
f15c0be5af test_web: move shouldHTTPError into common.WebErrorMixin 2009-03-03 16:56:20 -07:00
Brian Warner
559c369574 test_web: clean up shouldHTTPError methods 2009-03-03 16:46:04 -07:00
Brian Warner
400c04c19a immutable checker add-lease: catch remote IndexError here too 2009-02-27 01:17:24 -07:00
Brian Warner
99ebf6eaf5 servermap add-lease: fix the code that's supposed to catch remote IndexErrors, I forgot that they present as ServerFailures instead. This should stop the deluge of Incidents that occur when you do add-lease against 1.3.0 servers 2009-02-27 01:04:26 -07:00
Brian Warner
8c3013c4f7 rrefutil: add check_remote utility function 2009-02-27 00:59:57 -07:00
Brian Warner
1b3e635936 rrefutil: add trap_remote utility and friends 2009-02-27 00:55:24 -07:00
Brian Warner
8251572e01 web: improve layout of storage status with a table 2009-02-26 19:58:38 -07:00
Brian Warner
112dc35563 crawler: add ETA to get_progress() 2009-02-26 19:42:48 -07:00
Alberto Berti
3035dfb8ed Fix for bug #645, correct path handling logic so that it works from sshfs 2009-02-26 09:02:37 -07:00
Zooko O'Whielacronx
9ab4aa9016 wui: s/Provisioning/Reliability/ ; suggested by Terrell 2009-02-25 21:09:04 -07:00
Brian Warner
7c35f01073 startstop_node.py: when restarting, tolerate a stale pid too 2009-02-25 19:52:45 -07:00
Brian Warner
63b19e567c web: fix the ERROR: line to work the same in python2.4 and 2.5 2009-02-25 01:46:21 -07:00
Brian Warner
7d4901197a test_cli/test_web: fix spurious test failure on solaris (maybe python2.4?) due to variations in the way that exceptions are stringified 2009-02-25 00:01:28 -07:00
Brian Warner
a3c1fe35d9 CLI: modify 'tahoe manifest' and 'tahoe deep-check' to report ERROR: properly. For #590. 2009-02-24 23:44:15 -07:00
Brian Warner
fd4ceb6a87 webapi: modify streaming deep-manifest/deep-checker to emit an ERROR: line if they encounter an unrecoverable+untraversable directory. For #590. 2009-02-24 23:13:35 -07:00
Brian Warner
7f8708cb7c scripts/common: fix alias handling on windows again, emit slightly nicer error message in response to an unknown alias 2009-02-24 22:21:36 -07:00
Brian Warner
fdb0200eca #165: make 'tahoe restart --force' the default behavior: warn but do not stop if restart is used on something that wasn't a running node, and always try to start it afterwards. This is particularly important for #315 (restart -m), because otherwise a single not-already-running node will prevent all nodes from being restarted, resulting in longer downtime than necessary 2009-02-24 21:42:13 -07:00
Brian Warner
51e3baeefc test_deepcheck: switch deep-check tests to use no-network too. This cuts the runtime down by about 50% 2009-02-24 21:04:57 -07:00
Brian Warner
6cf4096156 test_deepcheck: convert MutableChecker to no-network GridTest 2009-02-24 20:00:10 -07:00
Brian Warner
2e7f64d392 tests/no_network: move GET into the GridTestMixin class 2009-02-24 18:33:00 -07:00
Brian Warner
6f981e4eab common_web.py: oops, add .fields to the other FakeRequest 2009-02-24 18:04:59 -07:00
Brian Warner
376eab0c04 test_cli: exercise the recent tolerate-'c:\dir\file.txt' fix in scripts/common, recorded in a separate match to make it easier to merge the fix to prod 2009-02-24 17:56:20 -07:00
Brian Warner
a0aea06a9e scripts/common: on windows, tolerate paths like 'c:\dir\file.txt', by treating single-letter aliases on windows/cygwin as non-aliases 2009-02-24 17:55:22 -07:00
Brian Warner
8d8250e44a test/common_web.py: add a .fields attribute to our FakeRequest, since we support versions of Nevow that are old enough to not do it themselves 2009-02-24 17:20:50 -07:00
Alberto Berti
e1c925b373 Two small fixes on documentation for cli backup command. 2009-02-24 16:36:34 -07:00
Brian Warner
2810de32b1 test_web: add (disabled) test to see what happens when deep-check encounters an unrecoverable directory. We still need code changes to improve this behavior. 2009-02-24 15:40:17 -07:00
Alberto Berti
7e8958671b Add elapsed timestamp to cli backup command final summary. 2009-02-24 11:14:25 -07:00
Brian Warner
7cfbb9c832 test_repairer: change to use faster no_network.GridTestMixin, split Verifier tests into separate cases, refactor judgement funcs into shared methods 2009-02-23 22:15:06 -07:00
Brian Warner
f95e9b5964 immutable/checker.py: trap ShareVersionIncompatible too. Also, use f.check 
instead of examining the value returned by f.trap, because the latter appears
to squash exception types down into their base classes (i.e. since
ShareVersionIncompatible is a subclass of LayoutInvalid,
f.trap(Failure(ShareVersionIncompatible)) == LayoutInvalid).

All this resulted in 'incompatible' shares being misclassified as 'corrupt'.
 2009-02-23 22:14:05 -07:00
Brian Warner
9af9d8ae35 immutable/layout.py: wrap to 80 cols, no functional changes 2009-02-23 18:58:37 -07:00
Brian Warner
2be729b1e4 test_repairer: change Repairer to use much-faster no_network.GridTestMixin. As a side-effect, fix what I think was a bug: some of the assert-minimal-effort-expended checks were mixing write counts and allocate counts 2009-02-23 17:42:27 -07:00
Brian Warner
cc3c1ae8df test/no_network.py: add a basic stats provider 2009-02-23 17:39:37 -07:00
Brian Warner
f35c659b2f tests: stop using setUpClass/tearDownClass, since they've been deprecated in Twisted-8.2.0 2009-02-23 14:43:12 -07:00
Brian Warner
bc91689f8e test_checker: improve test coverage for checker results 2009-02-23 14:19:43 -07:00
Alberto Berti
d9649052d4 Fixed tests again so they will pass on windows. 2009-02-22 18:35:02 -07:00
Brian Warner
15ee186499 scripts: stop using RuntimeError, for #639 2009-02-22 17:31:06 -07:00
Brian Warner
c23d051d3f mutable/publish: stop using RuntimeError, for #639 2009-02-22 17:30:56 -07:00
Brian Warner
c4f8005867 remove more RuntimeError from unit tests, for #639 2009-02-22 17:28:55 -07:00
Brian Warner
5c3d7d8e73 stop using RuntimeError in unit tests, for #639 2009-02-22 17:27:22 -07:00
Brian Warner
d95b01a214 ftpd/sftpd: stop using RuntimeError, for #639 2009-02-22 17:24:26 -07:00
Alberto Berti
b945c275c2 Added tests for the cse when listdir is an iterator 2009-02-22 16:43:56 -07:00
Alberto Berti
cd0bd97954 Fixed tests so that they pass also on buildbots. 2009-02-22 16:43:11 -07:00
Alberto Berti
b2f8fe04c6 Use failUnlessEqual instead of failUnless(a == b) 2009-02-22 16:42:14 -07:00
Alberto Berti
89a20fe4dd Better implementation of filtering algorithm. 2009-02-22 16:40:49 -07:00
Alberto Berti
2fbca1596b Removed '.hgrags' from vcs excludes 2009-02-22 16:39:46 -07:00
Alberto Berti
5d629d0050 Added tests for the --exclude* options of backup command. 2009-02-22 10:51:06 -07:00
Alberto Berti
5d57da93fd Added --exclude, --exclude-from and --exclude-vcs options to backup command. 
It is still impossible to specify absolute exclusion path, only
relative. I must check with tar or rsync how they allow them to be
specified.
 2009-02-22 11:08:29 -07:00
Alberto Berti
c54fdd61b0 Raise a more explanatory exception for errors encountered during backup processing. 2009-02-22 11:02:52 -07:00
Alberto Berti
ff6a77cc6b Added tests for the fixed alias related command's synopsis 2009-02-22 10:37:32 -07:00
Alberto Berti
ffab4002aa Add missing synopsis and descriptions for alias commands. 2009-02-20 18:31:06 -07:00
Brian Warner
c85eccbf64 test_web.py: increase test coverage of web.status.plural() 2009-02-21 18:01:16 -07:00
Brian Warner
77f3b83d68 crawler: fix performance problems: only save state once per timeslice (not after every bucket), don't start the crawler until 5 minutes after node startup 2009-02-21 14:56:49 -07:00
Brian Warner
a04d3b8fe8 test_system: oops, don't assume that all files in storage/ are in a deep storage/shares/prefix/si/shnum path, since now the crawler pickle has a short path 2009-02-21 00:17:10 -07:00
Brian Warner
b9c4f4bdf6 crawler: tolerate low-resolution system clocks (i.e. windows) 2009-02-21 00:15:33 -07:00
Brian Warner
106d31b112 BucketCountingCrawler: store just the count, not cycle+count, since it's too easy to make usage mistakes otherwise 2009-02-20 21:58:31 -07:00
Brian Warner
3c6471c717 test_storage: startService the server, as is now the standard practice 2009-02-20 21:57:55 -07:00
Brian Warner
f934289d2d crawler: load state from the pickle in init, rather than waiting until startService, so get_state() can be called early 2009-02-20 21:57:20 -07:00
Brian Warner
1077826357 BucketCountingCrawler: rename status and state keys to use 'bucket' instead of 'share', because the former is more accurate 2009-02-20 21:46:06 -07:00
Brian Warner
d2d297f12f storage: also report space-free-for-root and space-free-for-nonroot, since that helps users understand the space-left-for-tahoe number better 2009-02-20 21:28:56 -07:00
Brian Warner
b3cd4952bd storage: add bucket-counting share crawler, add its output (number of files+directories maintained by a storage server) and status to the webapi /storage page 2009-02-20 21:04:08 -07:00
Brian Warner
d14f00c537 storage: move si_b2a/si_a2b/storage_index_to_dir out of server.py and into common.py 2009-02-20 21:03:09 -07:00
Brian Warner
73e05bf967 crawler: add get_progress, clean up get_state 2009-02-20 18:27:43 -07:00
Brian Warner
2e45619844 web/storage: make sure we can handle platforms without os.statvfs too 2009-02-20 16:03:53 -07:00
Brian Warner
c6a061e600 crawler: provide for one-shot crawlers, which stop after their first full cycle, for share-upgraders and database-populaters 2009-02-20 15:19:11 -07:00
Brian Warner
00677ff9a5 web: add Storage status page, improve tests 2009-02-20 14:29:26 -07:00
Brian Warner
ff6907a557 storage: include reserved_space in stats 2009-02-20 14:29:20 -07:00
Brian Warner
4b8cc42f18 web/check_results: sort share identifiers in the sharemap display 2009-02-20 12:29:22 -07:00
Brian Warner
c6c30ac5d4 webapi: pass client through constructor arguments, remove IClient, should make it easier to test web renderers in isolation 2009-02-20 12:15:54 -07:00
Brian Warner
e357b8a679 test/no_network: do startService on the storage servers, make it easier to customize the storage servers 2009-02-19 20:22:54 -07:00
Brian Warner
ef4ff21ae7 crawler: modify API to support upcoming bucket-counting crawler 2009-02-19 19:31:42 -07:00
Brian Warner
9bc08158c6 test_backupdb: make the not-a-database file larger, since the older sqlite-2.3.2 on OS-X is easily fooled 2009-02-19 18:04:09 -07:00
Brian Warner
26fbe9df95 web/reliability: add parameter descriptions, adapted from a patch from Terrell Russell. 2009-02-19 16:29:18 -07:00
Brian Warner
2f751332e8 test_crawler: hush pyflakes 2009-02-19 14:23:40 -07:00
Brian Warner
ff9811211d test_crawler: disable the percentage-of-cpu-used test, since it is too unreliable on our slow buildslaves. But leave the code in place for developers to run by hand. 2009-02-19 14:16:54 -07:00
Brian Warner
790a10d1b2 reliability.py: fix the numpy conversion, it was completely broken. Thanks to Terrell Russell for the help. 2009-02-19 13:55:15 -07:00
Brian Warner
268cd538e2 reliability: switch to NumPy, since Numeric is deprecated 2009-02-19 01:44:35 -07:00
Brian Warner
f10aa59e5b test_crawler: don't require >=1 cycle on cygwin 2009-02-19 00:58:18 -07:00
Brian Warner
dfd72c6483 crawler: use fileutil.move_info_place in preference to our own version 2009-02-18 23:13:42 -07:00
Brian Warner
89f041ac83 fileutil: add move_into_place(), to perform the standard unix trick of atomically replacing a file, with a fallback for windows 2009-02-18 23:13:10 -07:00
Brian Warner
b949ea4f32 crawler: fix problems on windows and our slow cygwin slave 2009-02-18 22:24:31 -07:00
Brian Warner
193889f793 #633: first version of a rate-limited interruptable share-crawler 2009-02-18 21:46:33 -07:00
Brian Warner
4646451de6 change StorageServer to take nodeid in the constructor, instead of assigning it later, since it's cleaner and because the original problem (Tubs not being ready until later) went away 2009-02-18 16:23:01 -07:00
Brian Warner
b398a69f14 test_system: split off checker tests to test_deepcheck.py, this file is too big 2009-02-18 15:42:34 -07:00
Brian Warner
ef53da2b12 break storage.py into smaller pieces in storage/*.py . No behavioral changes. 2009-02-18 14:46:55 -07:00
Brian Warner
a0c5f92cbd immutable/layout: minor change to repr name 2009-02-18 14:46:48 -07:00
Brian Warner
2346d8621d interfaces.py: allow add/renew/cancel-lease to return Any, so that 1.3.1 clients (the first to use these calls) can tolerate future storage servers which might return something other than None 2009-02-18 13:29:03 -07:00
Brian Warner
bce4a5385b add --add-lease to 'tahoe check', 'tahoe deep-check', and webapi. 2009-02-17 19:32:43 -07:00
Brian Warner
e9563ebc02 change RIStorageServer.remote_add_lease to exit silently in case of no-such-bucket, instead of raising IndexError, because that makes the upcoming --add-lease feature faster and less noisy 2009-02-17 19:30:53 -07:00
Brian Warner
fde2289e7b CLI #590: convert 'tahoe deep-check' to streaming form, improve display, add tests 2009-02-17 17:15:11 -07:00
Brian Warner
0e78b2587c interfaces.py: document behavior of add_lease/renew_lease/cancel_lease, before I change it 2009-02-17 13:48:09 -07:00
Brian Warner
d52eadfc58 test_backupdb: improve error messages if the test fails 2009-02-17 11:08:38 -07:00
Brian Warner
476a5c8fac webapi #590: add streaming deep-check. Still need a CLI tool to use it. 2009-02-16 23:35:53 -07:00
Brian Warner
8579e25059 test_web.Grid: change the CHECK() function to make it easier to test t= values with hyphens in them 2009-02-16 23:00:34 -07:00
Brian Warner
522641bb32 test_web: improve checker-results coverage with a no-network -based test, enhance no-network harness to assist, fix some bugs in web/check_results.py that were exposed 2009-02-16 22:12:42 -07:00
Brian Warner
059f24adca web: fix handling of reliability page when Numeric is not available 2009-02-16 19:56:58 -07:00
Brian Warner
f2f8ede11f test/no_network: update comments with setup timing: no_network takes 50ms, SystemTestMixin takes 2s (on my laptop) 2009-02-16 18:06:43 -07:00
Brian Warner
040cb39613 test_upload: rewrite in terms of no-network GridTestMixin, improve no_network.py as necessary 2009-02-16 17:44:57 -07:00
Brian Warner
911abcc34b test_download: rewrite in terms of no-network GridTestMixin, improve no_network.py as necessary 2009-02-16 17:36:58 -07:00
Brian Warner
e9427ea777 test_dirnode.py: convert Deleter to new no-network gridtest 2009-02-16 17:23:48 -07:00
Brian Warner
267d91f797 test_cli.py: modify to use the new 'no-network' gridtest instead of SystemTestMixin, which speeds it up from 73s to 43s on my system 2009-02-16 17:20:05 -07:00
Brian Warner
191fd8556a tests: fix no_network framework to work with upload/download and checker 2009-02-16 17:19:47 -07:00
Brian Warner
0409bc5628 client.py: improve docstring 2009-02-16 17:15:32 -07:00
Brian Warner
770f976a85 test_cli: add test coverage for help strings 2009-02-16 15:08:33 -07:00
Brian Warner
ebe0f2ac08 test/no_network: new test harness, like system-test but doesn't use the network so it's faster 2009-02-16 14:58:44 -07:00
Brian Warner
8ff76c6269 interfaces.py: minor docstring edit 2009-02-16 14:58:16 -07:00
Brian Warner
df90dd8e73 test_cli: improve test coverage slightly 2009-02-15 21:04:51 -07:00
Brian Warner
3235b9630b test_util: get almost full test coverage of dictutil, starting with the original pyutil tests as a base. The remaining three uncovered lines involve funny cases of ValueOrderedDict that I can't figure out how to get at 2009-02-15 20:32:10 -07:00
Brian Warner
cc27b218f5 provisioning/reliability: add tests, hush pyflakes, remove dead code, fix web links 2009-02-15 16:24:51 -07:00
Brian Warner
e1380b132b util/statistics: add tests, fix mean_repair_cost 2009-02-15 16:23:26 -07:00
Brian Warner
812f23585d test_repairer: hush pyflakes 2009-02-15 16:23:10 -07:00
Brian Warner
e2efd911a3 build a 'reliability' web page, with a simulation of file decay and repair over time 2009-02-13 17:42:34 -07:00
Shawn Willden
51ab76875b More lossmodel work, on repair. 2009-01-15 20:56:48 -07:00
Shawn Willden
21832280da Statistics module 
Added a statistics module for calculating various facets of
share survival statistics.
 2009-01-13 20:12:35 -07:00
Brian Warner
60b06b8d32 test_dirnode: #625 run deep-check on a readonly dirnode too 2009-02-13 14:53:37 -07:00
Brian Warner
4aee51ff6f figleaf_htmlizer: fix order of summary counters 2009-02-13 09:57:53 -07:00
Brian Warner
aee319e236 figleaf_htmlizer: oops, re-ignore files that aren't under root, like code in auto-built eggs 2009-02-13 00:00:22 -07:00
Zooko O'Whielacronx
14d23d99a5 setup: make the "full version string" be "allmydata-tahoe/1.3.0" instead of "allmydata-tahoe-1.3.0" and the UserAgent string of the cli be "allmydata-tahoe/1.3.0 (tahoe-client)" 
This is webbish.  Thanks to kpreid for suggesting it.
 2009-02-12 22:37:38 -07:00
Brian Warner
30724593fc figleaf_htmlizer: emit stats to stdout, so buildbot can see it 2009-02-12 15:10:20 -07:00
Brian Warner
781c271b82 figleaf_htmlizer: render changes in coverage relative to a previous test run using --old-coverage 2009-02-12 15:04:12 -07:00
Zooko O'Whielacronx
d7dbd6675e immutable repairer: fix DownUpConnector so that it satisfies short reads the were requested after the last write and before the close 
This is probably the cause of the very rare "loss of progress" bug.  This is tested by unit tests.  A recent patch changed this to errback instead of losing progress, and now this patch is changing it again to return a short read instead of errbacking.  Returning a short read is what the uploader (in encode.py) is expecting, when it is reading the last block of the ciphertext, which might be shorter than the other blocks.
 2009-02-12 17:04:47 -07:00
Zooko O'Whielacronx
bdb992467c immutable repairer: add an assertion that a certain value in this tricky function is always what I think it is 2009-02-12 16:31:32 -07:00
Zooko O'Whielacronx
2d45d192d1 immutable repairer: add tests of how it handles short reads 2009-02-12 16:48:53 -07:00
Zooko O'Whielacronx
8303f90da0 nodeadmin: node stops itself if a hotline file hasn't been touched in 60 seconds now, instead of in 40 seconds 
A test failed on draco (MacPPC) because it took 49 seconds to get around to running the test, and the node had already stopped itself when the hotline file was 40 seconds old.
 2009-02-12 15:21:31 -07:00
Zooko O'Whielacronx
76d7cc4404 immutable repairer: errback any pending readers of DownUpConnectorwhen it runs out of bytes, and test that fact 2009-02-11 20:11:29 -07:00
Brian Warner
125bf09528 figleaf_htmlizer: more rearranging, behavior should still be unchanged 2009-02-11 20:05:15 -07:00
Brian Warner
e02ecec1ad figleaf_htmlizer: break it up into subfunctions, behavior should still be unchanged 2009-02-11 19:56:07 -07:00
Brian Warner
6bde1b0345 figleaf_htmlizer: rewrite in class form, behavior should be the same as before 2009-02-11 19:40:50 -07:00
Brian Warner
6db747b1f0 figleaf_htmlizer: rewrite with twisted.python.usage, remove logging: should behave the same as before 2009-02-11 19:16:43 -07:00
Brian Warner
f3ed579e74 figleaf_htmlizer: expand tabs, fix to 4-space indents. No functional changes. 2009-02-11 19:05:42 -07:00
Zooko O'Whielacronx
7eb260a9cf versioning: include an "appname" in the application version string in the versioning protocol, and make that appname be controlled by setup.py 
It is currently hardcoded in setup.py to be 'allmydata-tahoe'.  Ticket #556 is to make it configurable by a runtime command-line argument to setup.py: "--appname=foo", but I suddenly wondered if we really wanted that and at the same time realized that we don't need that for tahoe-1.3.0 release, so this patch just hardcodes it in setup.py.
setup.py inspects a file named 'src/allmydata/_appname.py' and assert that it contains the string "__appname__ = 'allmydata-tahoe'", and creates it if it isn't already present.  src/allmydata/__init__.py import _appname and reads __appname__ from it.  The rest of the Python code imports allmydata and inspects "allmydata.__appname__", although actually every use it uses "allmydata.__full_version__" instead, where "allmydata.__full_version__" is created in src/allmydata/__init__.py to be:

__full_version__ = __appname + '-' + str(__version__).

All the code that emits an "application version string" when describing what version of a protocol it supports (introducer server, storage server, upload helper), or when describing itself in general (introducer client), usese allmydata.__full_version__.

This fixes ticket #556 at least well enough for tahoe-1.3.0 release.
 2009-02-11 17:18:16 -07:00
Brian Warner
23fdd4f48e backupdb: cosmetic: capitalize the no-pysqlite instructions properly. Thanks to Terrell Russell for the catch. 2009-02-11 15:28:30 -07:00
Brian Warner
5f288842d5 test_repairer: disable repair-from-corruption tests until other things are improved well enough to make it useful 2009-02-11 15:01:59 -07:00
Zooko O'Whielacronx
079c1851a9 trivial: whitespace 2009-02-11 11:19:35 -07:00
Brian Warner
7d907f35aa storage: make add-lease work, change default ownernum=1 since 0 is reserved to mean 'no lease here' 2009-02-10 23:39:38 -07:00
Brian Warner
13a3ef5ec1 #620: storage: allow mutable shares to be deleted, with a writev where new_length=0 2009-02-10 23:37:56 -07:00
Brian Warner
6a5f28f47d test_cli.Backup: insert some stalls to make sure two successive backups get distinct timestamps, avoiding intermittent failures 2009-02-10 20:37:09 -07:00
Brian Warner
f7a263eb0b #619: make 'tahoe backup' complain and refuse to run if sqlite is unavailable and --no-backupdb is not passed 2009-02-10 18:49:10 -07:00
Brian Warner
3b1c8ceb86 tests: increase the default timeout for SystemTestMixin -based tests to 300 seconds, since our slower buildslaves sometimes take longer than the default 120s 2009-02-10 14:44:12 -07:00
Zooko O'Whielacronx
ef1bfdd2bf immutable: repairer: add a simple test to exercise the "leftover" code path, fix the bug (and rename the variable "leftover" to "extra") 2009-02-10 12:12:45 -07:00
Zooko O'Whielacronx
75e4e67ed7 immutable: tighten preconditions -- you can write empty strings or read zero bytes, and add the first simple unit test of DownUpConnector 2009-02-10 00:56:47 -07:00
Zooko O'Whielacronx
d436c6235d immutable: tests: the real WRITE_LEEWAY is 35 (it was a mistake to move it from 10 to 35 earlier -- I had seen a failure in which it took 35 times as many writes as I thought were optimal, but I misread and thought it took only 20 times as many) 2009-02-09 23:53:48 -07:00
Zooko O'Whielacronx
c59940852b immutable: defensive programming: assert that the encrypted readable gave you no more than the number of bytes you asked for 
(There is a bug in the current DownUpConnector which can cause it to give more bytes than you asked for on one request, and then less on the next, effectively shifting some of the bytes to an earlier request, but I think this bug never gets triggered in practice.)
 2009-02-09 23:46:05 -07:00
Zooko O'Whielacronx
65c12b24b8 immutable: tests: sigh, raise, again the limit of how many extra writes you can do and still pass this test 
Obviously requiring the code under test to perform within some limit isn't very meaningful if we raise the limit whenever the test goes outside of it.
But I still don't want to remove the test code which measures how many writes (and, elsewhere, how many reads) a client does in order to fulfill these duties.
Let this number -- now 20 -- stand as an approximation of the inefficiency of our code divided by my mental model of how many operations are actually optimal for these duties.
 2009-02-09 20:09:31 -07:00
Zooko O'Whielacronx
5738d94ccd immutable: tests: assert that verifier gives a clean bill of health after corruption and repair (the previous patch mistakenly did this only after deletion and repair), and also test whether deleting seven other shares and then downloading works. Also count the number of shares stored in the local filesystem. 2009-02-09 20:08:41 -07:00
Zooko O'Whielacronx
bbf48f4a88 immutable: test: add a test after attempting to repair from corruption: does a full verify run give the file a clean bill of health? If not, the you haven't successfully repaired it. 
This will make the repairer tests more consistent -- less accidentally passing due to getting lucky.
 2009-02-09 19:01:49 -07:00
Zooko O'Whielacronx
4091f73f34 immutable: tests: put shares back to their pristine condition in between each test of corrupting-and-repairing them 
This is important, because if the repairer doesn't completely repair all kinds of corruption (as the current one doesn't), then the successive tests get messed up by assuming that the shares were uncorrupted when the test first set about to corrupt them.
 2009-02-09 18:29:56 -07:00
Brian Warner
a9a3b509df upload: add a think-of-the-compatibility note to UploadResults 2009-02-09 14:50:04 -07:00
Brian Warner
a5ab6c060d helper #609: uploading client should ignore old helper's UploadResults, which were in a different format 2009-02-09 14:45:43 -07:00
Brian Warner
b197c5bb8c test_runner: skip all spawnProcess-using tests on cygwin, since spawnProcess just hangs forever 2009-02-09 02:34:00 -07:00
Brian Warner
19902a0f16 test_runner.py: revert the CreateNode section to using runner() inline, rather than spawning a process, to get more precise coverage 2009-02-09 02:26:17 -07:00
Brian Warner
0b29aae7cf storage #596: announce 'tolerates-immutable-read-overrun' to the version announcement, to indicate that a read() on an immutable share where offset+length is beyond the end of the file will return a truncated string instead of raising an exception 2009-02-08 19:56:02 -07:00
Brian Warner
89683a3906 test_upload: add test of maximum-immutable-share-size, to complete the last item of #538 2009-02-08 19:41:27 -07:00
Brian Warner
d8e00c37b5 test_cli.Backup: capture stderr when sqlite is unavailable 2009-02-07 15:14:40 -07:00
Brian Warner
213ef3d64b test_repairer: wrap comments to 80cols, my laptop does not have a wide screen. No functional changes. 2009-02-07 14:06:26 -07:00
Brian Warner
38ee95fec4 immutable/checker: wrap comments to 80cols, my laptop does not have a wide screen. No functional changes. 2009-02-07 14:04:39 -07:00
Brian Warner
50886ea494 test/common.py: in share-layout-reading code, use '>L' consistently, since '>l' doesn't specify the signedness and the windows tests appear to be failing with an endianness-like problem (version==0x01000000). Also use binary mode when editing sharefiles 2009-02-07 13:38:17 -07:00
Brian Warner
1fd8f023ab test_cli: increase timeout on test_backup, since our dapper buildslave is really slow 2009-02-06 02:17:53 -07:00
Brian Warner
5bc0ef7ffe backupdb.py: catch OperationalError on duplicate-insert too, since pysqlite2 on dapper raises it instead of IntegrityError 2009-02-06 01:34:01 -07:00
Brian Warner
9933aa0d49 test_backupdb.py: reset the check-timers after one step, otherwise a slow host can false-fail 2009-02-06 01:33:05 -07:00
Brian Warner
814fec5495 test_cli.backup: oops, fix test to work even when sqlite is unavailable 2009-02-05 22:10:42 -07:00
Brian Warner
ca32db5b39 #598: add cli+backupdb tests, improve user display, update docs, move docs out of proposed/ 2009-02-05 22:07:01 -07:00
Brian Warner
add0e1f2f8 #598: add backupdb to 'tahoe backup' command, enable it by default 2009-02-05 19:56:40 -07:00
Brian Warner
4be2e11d11 add sqlite-based backupdb, for #598 and others (including 'tahoe cp'). Not enabled yet. 2009-02-05 18:17:56 -07:00
Zooko O'Whielacronx
9ba5f9347d setup: add a case to execute "python .../twistd.py" if "twistd" is not found 2009-02-04 18:06:20 -07:00
Brian Warner
4e572a5f24 webapi: add verifycap (spelled 'verify_url') to the t=json output on files and directories. Closes #559. 2009-02-03 19:22:48 -07:00
Brian Warner
0716fbcd16 CLI: tahoe_check: stop escaping the JSON output when using --raw 2009-02-02 21:12:32 -07:00
Brian Warner
cfce8b5eab #598: first cut of 'tahoe backup' command: no backupdb, but yes shared-unchanged-directories and Archives/TIMESTAMP and Latest/ 2009-02-02 21:09:02 -07:00
Brian Warner
91dacc3dbc CLI: move node-url -should-end-in-slash code into a common place, so other tools can rely upon it later 2009-02-02 21:08:56 -07:00
Brian Warner
a7de479767 tahoe_ls: CLI command should return rc=0, not None 2009-02-02 21:07:20 -07:00
Zooko O'Whielacronx
e08e8d3f41 storage: disable test_large_share again: my linux laptop has less than 4 GiB free 2009-01-30 22:16:49 -07:00
Brian Warner
529a8e9fdb web/directory: add a link from readwrite directories to a read-only version, and fix the 'SI=xxx' header to actually use the storage index, not the writekey 2009-01-30 19:32:05 -07:00
Brian Warner
b77a017872 uri: add abbrev_si() method, which returns the abbreviated storage index 2009-01-30 19:31:10 -07:00
Zooko O'Whielacronx
f0682cf43e setup: more verbose assertion failure in test_runner 2009-01-29 10:49:06 -07:00
Zooko O'Whielacronx
204629be43 storage: enable the test of a share larger than 2 GiB; this will probably be too expensive on Mac OS X, but I think it won't be on all the other platforms ; I will probably set it to SkipTest if the sys.platform is Mac after seeing the results of this buildbot run 2009-01-28 16:33:12 -07:00
Brian Warner
1427db1c88 CLI: fix examples in tahoe put --help 2009-01-27 15:39:09 -07:00
Zooko O'Whielacronx
8d7fe2adb3 trivial: remove unused imports noticed by pyflakes 2009-01-27 15:11:48 -07:00
Zooko O'Whielacronx
64d12a9402 setup: fix test_runner to invoke bin/tahoe.exe instead of bin/tahoe if on Windows (this is what happens when a user invokes bin/tahoe on Windows) 2009-01-27 14:37:17 -07:00
Zooko O'Whielacronx
1ae08acc8f setup: fix test_runner to assert that lines which are output to stderr must end with a punctuation mark (apparently re.search("x$", "x\r\n") does not match. :-() 2009-01-27 14:35:05 -07:00
Zooko O'Whielacronx
3cd4722c9b setup: fix test_runner.RunNode.test_baddir -- it was left incomplete and broken by a recent patch 2009-01-27 14:32:45 -07:00
Zooko O'Whielacronx
688bb7a453 setup: go ahead and check for noise in test_client_no_noise 2009-01-26 17:46:16 -07:00
Zooko O'Whielacronx
8d2a3e78ca setup: add a test for a warning message from importing nevow, marked as TODO, comment-out the assertion of no-noise inside other test_runner tests 2009-01-26 17:30:46 -07:00
Brian Warner
9d6534d78b make streaming-manifest stop doing work after the HTTP connection is dropped 2009-01-23 19:39:08 -07:00
Zooko O'Whielacronx
3e4450cf3f setup: find a "bin/tahoe" executable to test based on allmydata.__file__ instead of based on the CWD 
This means that the tests still work if you are executing them from a CWD other than the src dir -- *if* the "bin/tahoe" is found at os.path.dirname(os.path.dirname(os.path.dirname(allmydata.__file__))).
If no file is found at that location, then just skip the tests of executing the "tahoe" executable, because we don't want to accidentally run those tests against an executable from a different version of tahoe.
 2009-01-23 18:34:37 -07:00
Brian Warner
1de4110b35 #509: remove non-streaming 'tahoe manifest' CLI form 2009-01-23 17:00:02 -07:00
Brian Warner
6a04103cc2 #509 CLI: add 'tahoe manifest --stream' 2009-01-23 16:33:21 -07:00
Brian Warner
5a0e3f3a40 #509: test_system.py: add test for streamed-manifest 2009-01-23 16:32:47 -07:00
Brian Warner
9ade197c3b test_system: rearrange DeepCheckWebGood to make it easier to add CLI tests 2009-01-23 16:13:06 -07:00
Brian Warner
6ba154851f CLI: remove the '-v' alias for --version-and-path, to free up '-v' for more traditional uses (like --verbose) 2009-01-23 15:21:50 -07:00
Brian Warner
26260374e9 #590: add webish t=stream-manifest 2009-01-22 22:01:36 -07:00
Brian Warner
72adeccf2d dirnode: add get_repair_cap() 2009-01-22 21:44:49 -07:00
Brian Warner
94ab90273d dirnode.deep_traverse: fix docstring to describe the correct return value 2009-01-22 21:39:50 -07:00
Brian Warner
d8b3505cf5 filenode: add get_repair_cap(), which uses the read-write filecap for immutable files, and the verifycap for immutable files 2009-01-22 21:38:36 -07:00
Zooko O'Whielacronx
b77c89ac60 setup: add test that the tests are testing the right source code 
This is a test of #145, and I think that now the code passes this test.
 2009-01-22 15:52:40 -07:00
Zooko O'Whielacronx
967102e1b9 trivial: removed unused imports noticed by pyflakes 2009-01-22 15:52:13 -07:00
Zooko O'Whielacronx
210da5cd9a setup: change test_runner to invoke "bin/tahoe" in a subprocess instead of executing runner.runner() 
This is necessary because loading allmydata code now depends on PYTHONPATH manipulation which is done in the "bin/tahoe" script.  Unfortunately it makes test_runner slower since it launches and waits for many subprocesses.
 2009-01-22 15:38:18 -07:00
Zooko O'Whielacronx
fafdd2ce59 setup: fix "tahoe start" to work on Windows even when a Tahoe base dir hasn't been configured in the Windows registry 2009-01-21 12:47:20 -07:00
Zooko O'Whielacronx
7d91dc5efe rollback the feature of making "ambient upload authority" configurable 
This reverses some, but not all, of the changes that were committed in the following set of patches.

rolling back:

Sun Jan 18 09:54:30 MST 2009  toby.murray
  * add 'web.ambient_upload_authority' as a paramater to tahoe.cfg

    M ./src/allmydata/client.py -1 +3
    M ./src/allmydata/test/common.py -7 +9
    A ./src/allmydata/test/test_ambient_upload_authority.py
    M ./src/allmydata/web/root.py +12
    M ./src/allmydata/webish.py -1 +4
Sun Jan 18 09:56:08 MST 2009  zooko@zooko.com
  * trivial: whitespace
  I ran emacs's "M-x whitespace-cleanup" on the files that Toby's recent patch had touched that had trailing whitespace on some lines.

    M ./src/allmydata/test/test_ambient_upload_authority.py -9 +8
    M ./src/allmydata/web/root.py -2 +1
    M ./src/allmydata/webish.py -2 +1
Mon Jan 19 14:16:19 MST 2009  zooko@zooko.com
  * trivial: remove unused import noticed by pyflakes

    M ./src/allmydata/test/test_ambient_upload_authority.py -1
Mon Jan 19 21:38:35 MST 2009  toby.murray
  * doc: describe web.ambient_upload_authority

    M ./docs/configuration.txt +14
    M ./docs/frontends/webapi.txt +11
Mon Jan 19 21:38:57 MST 2009  zooko@zooko.com
  * doc: add Toby Murray to the CREDITS

    M ./CREDITS +4
 2009-01-20 20:47:35 -07:00
Zooko O'Whielacronx
6bbbb375e1 setup: fix test_system to require tahoe under its package==distribution name "allmydata-tahoe" instead of its module name "allmydata" 2009-01-20 12:38:09 -07:00
Zooko O'Whielacronx
845b21a5ad trivial: remove unused import noticed by pyflakes 2009-01-19 15:16:19 -07:00
Zooko O'Whielacronx
f4f69f4a95 setup: refactor versions-and-paths and use pkg_resources to find them 
Using pkg_resources is probably better if it works -- zope.interface doesn't have a __version__ attribute that we can query, but pkg_resources knows zope.interface's version number, for one thing.
This code falls back to the old way -- looking at the __version__ attributes and __file__ attributes -- if the pkg_resources way doesn't answer.
Note that this patch also changes the capitalization of "Nevow", "Twisted", and "pyOpenSSL", and the spelling of "allmydata-tahoe".  These changes are not frivolous: they are reflecting the fact that we are naming Python packages (technically called Python "distributions") instead of Python modules (technically and confusingly called Python "packages") here.  The package ("distribution") is named "allmydata-tahoe".  The module ("package") is named "allmydata".
 2009-01-19 15:04:35 -07:00
Zooko O'Whielacronx
4771e1c9d5 trivial: whitespace 
Ran "M-x whitespace-cleanup" on files that Toby's recent patch touched, even though they didn't have trailing whitespace.
 2009-01-18 10:58:15 -07:00
Zooko O'Whielacronx
ce7c448bc0 trivial: whitespace 
I ran emacs's "M-x whitespace-cleanup" on the files that Toby's recent patch had touched that had trailing whitespace on some lines.
 2009-01-18 10:56:08 -07:00
Zooko O'Whielacronx
9247dc70e5 trivial: whitespace 2009-01-18 10:54:58 -07:00
toby.murray
66f83c7356 add 'web.ambient_upload_authority' as a paramater to tahoe.cfg 2009-01-18 10:54:30 -07:00
cgalvan
7bfaa74ef1 add option to show version and path to the tahoe executable 2009-01-16 12:47:51 -07:00
Brian Warner
26ca53fa3d node.py: use NODEDIR/tmp for the 'tempfile' module's temporary directory, so webapi upload tempfiles are put there instead of /tmp . You can set it to something else by setting [node]tempdir in tahoe.cfg 2009-01-14 20:00:15 -07:00
Brian Warner
e9d4334159 web/operations: undo the disable-ophandle-expiration change that inadvertently got included in zooko's recent 'rename wapi.txt to webapi.txt' patch, causing test failures 2009-01-14 19:14:59 -07:00
Zooko O'Whielacronx
f9ee60a88e cli: tests: skip symlink test if there is no os.symlink 2009-01-14 18:10:10 -07:00
Brian Warner
2fe099a0b3 mutable: move recent operation history management code (MutableWatcher) into history.py, have History provide stats 2009-01-14 17:36:20 -07:00
Brian Warner
aa50c30aa2 download: tiny cleanup of history code 2009-01-14 16:41:51 -07:00
Brian Warner
10268a4f7f upload: move upload history into History object 2009-01-14 16:41:06 -07:00
Brian Warner
3920e6d1e7 immutable/download.py move recent-downloads history out of Downloader and into a separate class. upload/etc will follow soon. 2009-01-14 16:14:24 -07:00
Zooko O'Whielacronx
0a2895e17a docs: rename wapi.txt to webapi.txt 
Because Brian argues that the file contains a description of the wui as well as of the wapi, and because the name "webapi.txt" might be more obvious to the untrained eye.
 2009-01-14 13:53:48 -07:00
Brian Warner
cc50e2f4aa upload: use WriteBucketProxy_v2 when uploading a large file (with shares larger than 4GiB). This finally closes #346. I think we can now handle immutable files up to 48EiB. 2009-01-12 20:14:42 -07:00
Brian Warner
bf56e2bb51 deep-check-and-repair: improve results and their HTML representation 2009-01-12 18:56:19 -07:00
Brian Warner
8f2ec16b2c test_repairer.py: hush pyflakes: remove duplicate/shadowed function name, by using the earlier definition (which is identical) 2009-01-12 15:45:09 -07:00
Brian Warner
fe362c0021 hush pyflakes by removing unused imports 2009-01-12 15:41:20 -07:00
Zooko O'Whielacronx
25063688b4 immutable repairer 
This implements an immutable repairer by marrying a CiphertextDownloader to a CHKUploader.  It extends the IDownloadTarget interface so that the downloader can provide some metadata that the uploader requires.
The processing is incremental -- it uploads the first segments before it finishes downloading the whole file.  This is necessary so that you can repair large files without running out of RAM or using a temporary file on the repairer.
It requires only a verifycap, not a readcap.  That is: it doesn't need or use the decryption key, only the integrity check codes.
There are several tests marked TODO and several instances of XXX in the source code.  I intend to open tickets to document further improvements to functionality and testing, but the current version is probably good enough for Tahoe-1.3.0.
 2009-01-12 11:00:22 -07:00
Zooko O'Whielacronx
e449052a17 util: dictutil: add DictOfSets.union(key, values) and DictOfSets.update(otherdictofsets) 2009-01-12 10:55:39 -07:00
Zooko O'Whielacronx
8dd3b3185f immutable: separate tests of immutable upload/download from tests of immutable checking/repair 2009-01-10 15:07:39 -07:00
Zooko O'Whielacronx
b496eba072 trivial: minor changes to in-line comments -- mark plaintext-hash-tree as obsolete 2009-01-10 14:56:01 -07:00
Zooko O'Whielacronx
38bb3e464b immutable: make the web display of upload results more human-friendly, like they were before my recent change to the meaning of the "sharemap" 2009-01-10 14:02:09 -07:00
Zooko O'Whielacronx
0f9c11cfde immutable: fix edit-o in interfaces.py documentation introduced in recent patch 2009-01-10 12:54:08 -07:00
Zooko O'Whielacronx
6e3396fb88 immutable: redefine the "sharemap" member of the upload results to be a map from shnum to set of serverids 
It used to be a map from shnum to a string saying "placed this share on XYZ server".  The new definition is more in keeping with the "sharemap" object that results from immutable file checking and repair, and it is more useful to the repairer, which is a consumer of immutable upload results.
 2009-01-10 11:46:23 -07:00
Zooko O'Whielacronx
ef60e85ec6 naming: finish renaming "CheckerResults" to "CheckResults" 2009-01-09 18:00:52 -07:00
Brian Warner
167742c2b3 storage.py : replace 4294967295 with 2**32-1: python does constant folding, I measured this statement as taking 50ns, versus the 400ns for the call to min(), or the 9us required for the 'assert not os.path.exists' syscall 2009-01-09 19:52:22 -07:00
Brian Warner
c7cd3f38e7 storage.py: announce a maximum-immutable-share-size based upon a 'df' of the disk. Fixes #569, and this should be the last requirement for #346 (remove 12GiB filesize limit) 2009-01-09 19:37:36 -07:00
Brian Warner
45fc3819a2 test_helper: hush pyflakes by avoiding use of 'uri' as a variable, since it shadows an import of the same name 2009-01-08 20:59:41 -07:00
Brian Warner
f8de336039 immutable/checker: include a summary (with 'Healthy' or 'Not Healthy' and a count of shares) in the checker results 2009-01-08 20:01:45 -07:00
Brian Warner
7ee336b274 webapi/deep-manifest t=JSON: don't return the (large) manifest/SI/verifycap lists unless the operation has completed, to avoid the considerable CPU+memory cost of creating the JSON (for 330k dirnodes, it could take two minutes to generate 275MB of JSON). They must be paid eventually, but not on every poll 2009-01-08 19:59:32 -07:00
Brian Warner
39a089dc7e dirnode deep-traversal: remove use of Limiter, stick with strict depth-first-traversal, to reduce memory usage during very large (300k+ dirnode) traversals 2009-01-08 19:41:16 -07:00
Zooko O'Whielacronx
ade6a4fa74 immutable: add a monitor API to CiphertextDownloader with which to tell it to stop its work 2009-01-08 14:42:15 -07:00
Zooko O'Whielacronx
157e365d2b naming: Rename a few things which I touched or changed in the recent patch to download-without-decrypting. 
Rename "downloadable" to "target".
Rename "u" to "v" in FileDownloader.__init__().
Rename "_uri" to "_verifycap" in FileDownloader.
Rename "_downloadable" to "_target" in FileDownloader.
Rename "FileDownloader" to "CiphertextDownloader".
 2009-01-08 12:13:07 -07:00
Zooko O'Whielacronx
600196f571 immutable: refactor download to do only download-and-decode, not decryption 
FileDownloader takes a verify cap and produces ciphertext, instead of taking a read cap and producing plaintext.
FileDownloader does all integrity checking including the mandatory ciphertext hash tree and the optional ciphertext flat hash, rather than expecting its target to do some of that checking.
Rename immutable.download.Output to immutable.download.DecryptingOutput. An instance of DecryptingOutput can be passed to FileDownloader to use as the latter's target.  Text pushed to the DecryptingOutput is decrypted and then pushed to *its* target.
DecryptingOutput satisfies the IConsumer interface, and if its target also satisfies IConsumer, then it forwards and pause/unpause signals to its producer (which is the FileDownloader).
This patch also changes some logging code to use the new logging mixin class.
Check integrity of a segment and decrypt the segment one block-sized buffer at a time instead of copying the buffers together into one segment-sized buffer (reduces peak memory usage, I think, and is probably a tad faster/less CPU, depending on your encoding parameters).
Refactor FileDownloader so that processing of segments and of tail-segment share as much code is possible.
FileDownloader and FileNode take caps as instances of URI (Python objects), not as strings.
 2009-01-08 11:53:49 -07:00
Zooko O'Whielacronx
9bba578776 trivial: tiny changes to test code 2009-01-08 11:20:48 -07:00
Zooko O'Whielacronx
ecabcc674c immutable: Make more parts of download use logging mixins and know what their "parent msg id" is. 2009-01-08 11:25:30 -07:00
Zooko O'Whielacronx
2a443cd049 trivial: M-x whitespace-cleanup on src/immutable/download.py 2009-01-08 10:49:01 -07:00
Zooko O'Whielacronx
7d15928faa immutable: ValidatedExtendedURIProxy computes and stores the tail data size as a convenience to its caller. 
The "tail data size" is how many of the bytes of the tail segment are data (as opposed to padding).
 2009-01-08 10:41:39 -07:00
Zooko O'Whielacronx
2e762f39f6 immutable: define a new interface IImmutableFileURI and declare that CHKFileURI and LiteralFileURI provide it 2009-01-07 12:24:51 -07:00
Zooko O'Whielacronx
c01cfc0035 util: log: allow empty msgs (because downloader is using the "format" alternative with no "msg" argument) 2009-01-07 11:54:11 -07:00
Larry Hosken
3267984fa9 'tahoe cp -r', upon encountering a dangling symlink, would assert out. 
This was somewhat sad; the assertion didn't say what path caused the
error, what went wrong.  So... silently skip over things that are
neither dirs nor files.
 2009-01-07 23:51:14 -07:00
Zooko O'Whielacronx
83b97ee79f immutable: fix error in validation of ciphertext hash tree and add test for that code 
pyflakes pointed out to me that I had committed some code that is untested, since it uses an undefined name.  This patch exercises that code -- the validation of the ciphertext hash tree -- by corrupting some of the share files in a very specific way, and also fixes the bug.
 2009-01-07 23:40:12 -07:00
Zooko O'Whielacronx
6011f4522f immutable: do not catch arbitrary exceptions/failures from the attempt to get a crypttext hash tree -- catch only ServerFailure, IntegrityCheckReject, LayoutInvalid, ShareVersionIncompatible, and DeadReferenceError 
Once again I inserted a bug into the code, and once again it was hidden by something catching arbitrary exception/failure and assuming that it means the server failed to provide valid data.
 2009-01-07 22:25:51 -07:00
Zooko O'Whielacronx
e598ca2f3f download: make sure you really get all the crypttext hashes 
We were not making sure that we really got all the crypttext hashes during download.  If a server were to return less than the complete set of crypttext hashes, then our subsequent attempt to verify the correctness of the ciphertext would fail.  (And it wouldn't be obvious without very careful debugging why it had failed.)
This patch makes it so that you keep trying to get ciphertext hashes until you have a full set or you run out of servers to ask.
 2009-01-07 20:26:38 -07:00
Zooko O'Whielacronx
2122ba78b3 util: deferredutil: undo my recent patch to use our own implementation of gatherResults 
It seems to cause lots of failures on some builders.
 2009-01-07 11:00:05 -07:00
Zooko O'Whielacronx
79535e4f5b util: deferredutil: implement our own gatherResults instead of using Twisted's 
Because we want to maintain backwards compatibility to Twisted 2.4.0.
 2009-01-07 10:32:07 -07:00
Zooko O'Whielacronx
78306cf7fa trivial: M-x whitespace-cleanup 2009-01-07 10:25:28 -07:00
Zooko O'Whielacronx
a54eba1d80 util: deferredutil: add basic test for deferredutil.gatherResults 
Also I checked and Twisted 2.4.0 supports .subFailure and the other parts of the API that we require.
 2009-01-07 08:13:42 -07:00
Zooko O'Whielacronx
d5a6eed407 trivial: fix redefinition of name "log" in imports (pyflakes) 2009-01-06 22:08:29 -07:00
Zooko O'Whielacronx
c85f75bb08 immutable: refactor uploader to do just encoding-and-uploading, not encryption 
This makes Uploader take an EncryptedUploadable object instead of an Uploadable object.  I also changed it to return a verify cap instead of a tuple of the bits of data that one finds in a verify cap.
This will facilitate hooking together an Uploader and a Downloader to make a Repairer.
Also move offloaded.py into src/allmydata/immutable/.
 2009-01-06 21:48:22 -07:00
Zooko O'Whielacronx
81add135dc trivial: whitespace and docstring tidyups 2009-01-06 21:41:04 -07:00
Brian Warner
ebe767bccf storage.py: explain what this large and hard-to-recognize 4294967295 number is 2009-01-06 13:57:21 -07:00
Zooko O'Whielacronx
5e6f90a015 rename "checker results" to "check results", because it is more parallel to "check-and-repair results" 2009-01-06 13:37:03 -07:00
Zooko O'Whielacronx
4ada923e2b immutable: tests: verifier doesn't always catch corrupted share hashes 
Maybe it already got one of the corrupted hashes from a different server and it doesn't double-check that the hash from every server is correct.  Or another problem.  But in any case I'm marking this as TODO because an even better (more picky) verifier is less urgent than repairer.
 2009-01-06 13:04:49 -07:00
Zooko O'Whielacronx
3bc4b015c8 immutable: fix the writing of share data size into share file in case the share file is used by a < v1.3.0 storage server 
Brian noticed that the constant was wrong, and in fixing that I noticed that we should be saturating instead of modding.
This code would never matter unless a server downgraded or a share migrated from Tahoe >= v1.3.0 to Tahoe < v1.3.0.  Even in that case, this bug would never matter unless the share size were exactly 4,294,967,296 bytes long.
Brian, for good reason, wanted this to be spelled "2**32" instead of "4294967296", but I couldn't stand to see a couple of more Python bytecodes interpreted in the middle of a core, frequent operation on the server like immutable share creation.
 2009-01-06 12:24:04 -07:00
Zooko O'Whielacronx
8e92dd1231 trivial: whitespace cleanup 2009-01-06 11:20:58 -07:00
Zooko O'Whielacronx
d4a5f9131b util: base32: require str-not-unicode inputs -- effectively rolls back [3306] and [3307] 2009-01-06 10:41:22 -07:00
Zooko O'Whielacronx
c35a6ee3a2 trivial: fix a bunch of pyflakes complaints 2009-01-06 08:00:54 -07:00
Zooko O'Whielacronx
821e64203c cli: make startstop_node wait 40 seconds instead of 20 for a process to go away after we signalled it to go away, before emitting a warning 
Because the unit tests on the VirtualZooko? buildslave failed when it took 31 seconds for a process to go away.
Perhaps getting warning message after only 5 seconds instead of 40 seconds is desirable, and we should change the unit tests and set this back to 5, but I don't know exactly how to change the unit tests. Perhaps match this particular warning message about the shutdown taking a while and allow the code under test to pass if the only stderr that it emits is this warning.
 2009-01-06 07:51:06 -07:00
Zooko O'Whielacronx
6a12f316a4 immutable: new checker and verifier 
New checker and verifier use the new download class.  They are robust against various sorts of failures or corruption.  They return detailed results explaining what they learned about your immutable files.  Some grotesque sorts of corruption are not properly handled yet, and those ones are marked as TODO or commented-out in the unit tests.
There is also a repairer module in this patch with the beginnings of a repairer in it.  That repairer is mostly just the interface to the outside world -- the core operation of actually reconstructing the missing data blocks and uploading them is not in there yet.
This patch also refactors the unit tests in test_immutable so that the handling of each kind of corruption is reported as passing or failing separately, can be separately TODO'ified, etc.  The unit tests are also improved in various ways to require more of the code under test or to stop requiring unreasonable things of it.  :-)
 2009-01-05 18:28:18 -07:00
Zooko O'Whielacronx
4921a9f243 trivial: fix inline comment in test code 2009-01-05 17:53:42 -07:00
Zooko O'Whielacronx
206ab2b44d immutable: handle another form of share corruption with LayoutInvalid exception instead of AssertionError 2009-01-05 17:46:45 -07:00
Zooko O'Whielacronx
c84bb795f3 trivial: remove unused import (pyflakes) 2009-01-05 17:31:20 -07:00
Zooko O'Whielacronx
67d72aadd3 immutable: skip the test of large files, because that is too hard on the host if it doesn't efficiently handle sparse files 2009-01-05 17:07:27 -07:00
Zooko O'Whielacronx
f4fab23bf6 immutable: raise a LayoutInvalid exception instead of an AssertionError if the share is corrupted so that the sharehashtree is the wrong size 2009-01-05 14:01:14 -07:00
Zooko O'Whielacronx
98b28c1d5e immutable: stop reading past the end of the sharefile in the process of optimizing download -- Tahoe storage servers < 1.3.0 return an error if you read past the end of the share file 2009-01-05 13:40:57 -07:00
Zooko O'Whielacronx
8a840469c3 immutable: tidy up the notification of waiters for ReadBucketProxy 2009-01-05 13:35:22 -07:00
Zooko O'Whielacronx
778167c2b1 immutable: refactor downloader to be more reusable for checker/verifier/repairer (and better) 
The code for validating the share hash tree and the block hash tree has been rewritten to make sure it handles all cases, to share metadata about the file (such as the share hash tree, block hash trees, and UEB) among different share downloads, and not to require hashes to be stored on the server unnecessarily, such as the roots of the block hash trees (not needed since they are also the leaves of the share hash tree), and the root of the share hash tree (not needed since it is also included in the UEB).  It also passes the latest tests including handling corrupted shares well.
  
ValidatedReadBucketProxy takes a share_hash_tree argument to its constructor, which is a reference to a share hash tree shared by all ValidatedReadBucketProxies for that immutable file download.
  
ValidatedReadBucketProxy requires the block_size and share_size to be provided in its constructor, and it then uses those to compute the offsets and lengths of blocks when it needs them, instead of reading those values out of the share.  The user of ValidatedReadBucketProxy therefore has to have first used a ValidatedExtendedURIProxy to compute those two values from the validated contents of the URI.  This is pleasingly simplifies safety analysis: the client knows which span of bytes corresponds to a given block from the validated URI data, rather than from the unvalidated data stored on the storage server.  It also simplifies unit testing of verifier/repairer, because now it doesn't care about the contents of the "share size" and "block size" fields in the share.  It does not relieve the need for share data v2 layout, because we still need to store and retrieve the offsets of the fields which come after the share data, therefore we still need to use share data v2 with its 8-byte fields if we want to store share data larger than about 2^32.
  
Specify which subset of the block hashes and share hashes you need while downloading a particular share.  In the future this will hopefully be used to fetch only a subset, for network efficiency, but currently all of them are fetched, regardless of which subset you specify.
  
ReadBucketProxy hides the question of whether it has "started" or not (sent a request to the server to get metadata) from its user.

Download is optimized to do as few roundtrips and as few requests as possible, hopefully speeding up download a bit.
 2009-01-05 09:51:45 -07:00
Zooko O'Whielacronx
5d5e89d96d util: add gatherResults which is a deferred-list-like thing that doesn't wrap failures in a FirstError 2009-01-04 10:52:02 -07:00
Zooko O'Whielacronx
38f293a913 immutable: fix think-o in previous patch which caused all reads to return "", and also optimize by not opening the file when the answer is going to be "" 2009-01-03 14:02:45 -07:00
Zooko O'Whielacronx
53b28c1650 immutable: when storage server reads from immutable share, don't try to read past the end of the file (Python allocates space according to the amount of data requested, so if there is corruption and that number is huge it will do a huge memory allocation) 2009-01-03 13:22:22 -07:00
Zooko O'Whielacronx
de4269a643 immutable: mark a failing download test as "todo", because I think it is revealing a limitation of the current downloader's handling of corrupted shares 2009-01-03 13:00:03 -07:00
Zooko O'Whielacronx
8f5cc24948 trivial: remove unused import (pyflakes) 2009-01-03 12:22:15 -07:00
Zooko O'Whielacronx
5954ab456d immutable: fix test for truncated reads of URI extension block size 2009-01-03 11:44:27 -07:00
Zooko O'Whielacronx
7adf905b9f immutable: further loosen the performance-regression test to allow up to 45 reads 
This does raise the question of if there is any point to this test, since I apparently don't know what the answer *should* be, and whenever one of the buildbots fails then I redefine success.

But, I'm about to commit a bunch of patches to implement checker, verifier, and repairer as well as to refactor downloader, and I would really like to know if these patches *increase* the number of reads required even higher than it currently is.
 2009-01-03 11:41:09 -07:00
Zooko O'Whielacronx
2788c80496 trivial: another place where I accidentally committed a note-to-self about the lease fields in the server-side share file 2009-01-03 11:29:41 -07:00
Zooko O'Whielacronx
54787771c3 immutable: fix detection of truncated shares to take into account the fieldsize -- either 4 or 8 2009-01-02 18:57:45 -07:00
Zooko O'Whielacronx
21e0ff97f2 immutable: raise LayoutInvalid instead of struct.error when a share is truncated 
To fix this error from the Windows buildslave:

[ERROR]: allmydata.test.test_immutable.Test.test_download_from_only_3_remaining_shares

Traceback (most recent call last):
  File "C:\Documents and Settings\buildslave\windows-native-tahoe\windows\build\src\allmydata\immutable\download.py", line 135, in _bad
    raise NotEnoughSharesError("ran out of peers, last error was %s" % (f,))
allmydata.interfaces.NotEnoughSharesError: ran out of peers, last error was [Failure instance: Traceback: <class 'struct.error'>: unpack requires a string argument of length 4
c:\documents and settings\buildslave\windows-native-tahoe\windows\build\support\lib\site-packages\foolscap-0.3.2-py2.5.egg\foolscap\call.py:667:_done
c:\documents and settings\buildslave\windows-native-tahoe\windows\build\support\lib\site-packages\foolscap-0.3.2-py2.5.egg\foolscap\call.py:53:complete
c:\Python25\lib\site-packages\twisted\internet\defer.py:239:callback
c:\Python25\lib\site-packages\twisted\internet\defer.py:304:_startRunCallbacks
--- <exception caught here> ---
c:\Python25\lib\site-packages\twisted\internet\defer.py:317:_runCallbacks
C:\Documents and Settings\buildslave\windows-native-tahoe\windows\build\src\allmydata\immutable\layout.py:374:_got_length
C:\Python25\lib\struct.py:87:unpack
]
===============================================================================
 2009-01-02 18:48:06 -07:00
Zooko O'Whielacronx
5a21638853 immutable: whoops, it actually takes up to 39 reads sometimes to download a corrupted file 2009-01-02 17:43:02 -07:00
Zooko O'Whielacronx
e26cec2502 immutable: add more detailed tests of download, including testing the count of how many reads different sorts of downloads take 2009-01-02 16:54:59 -07:00
Zooko O'Whielacronx
cc70c163ba trivial: a few improvements to in-line doc and code, and renaming of test/test_immutable_checker.py to test/test_immutable.py 
That file currently tests checker and verifier and repairer, and will soon also test downloader.
 2009-01-02 16:49:41 -07:00
Zooko O'Whielacronx
a52b5542e9 immutable: fix name change from BadOrMissingShareHash to BadOrMissingHash 
One of the instances of the name accidentally didn't get changed, and pyflakes noticed.  The new downloader/checker/verifier/repairer unit tests would also have noticed, but those tests haven't been rolled into a patch and applied to this repo yet...
 2009-01-02 13:27:09 -07:00
Zooko O'Whielacronx
c72be1c553 trivial: remove unused import -- thanks, pyflakes 2009-01-02 13:21:28 -07:00
Zooko O'Whielacronx
d8c9c3dc99 immutable: download.py: Raise the appropriate type of exception to indicate the cause of failure, e.g. BadOrMissingHash, ServerFailure, IntegrityCheckReject (which is a supertype of BadOrMissingHash). This helps users (such as verifier/repairer) catch certain classes of reasons for "why did this download not work". The tests of verifier/repairer test this code and rely on this code. 2009-01-02 12:58:58 -07:00
Zooko O'Whielacronx
fa5c1d8326 immutable: ReadBucketProxy defines classes of exception: LayoutInvalid and its two subtypes RidiculouslyLargeURIExtensionBlock and ShareVersionIncompatible. This helps users (such as verifier/repairer) catch certain classes of reasons for "why did this download not work". This code gets exercised by the verifier/repairer unit tests, which corrupt the shares on disk in order to trigger problems like these. 2009-01-02 12:15:54 -07:00
Zooko O'Whielacronx
0ee027c180 immutable: ValidatedExtendedURIProxy computes and stores block_size and share_size for the convenience of its users 2009-01-02 11:43:17 -07:00
Zooko O'Whielacronx
fe707630af doc: remove notes to self that I accidentally included in a recent patch 2009-01-01 22:14:57 -07:00
Zooko O'Whielacronx
63efa1665b immutable: make the test of large files more likely to work by requesting to allocate space for only one huge share, not three 2008-12-31 15:59:42 -07:00
Zooko O'Whielacronx
b0b9e8c1da trivial: "M-x whitespace-cleanup", and also remove an unused variable 2008-12-31 15:42:33 -07:00
Zooko O'Whielacronx
6c4019ec33 immutable: storage servers accept any size shares now 
Nathan Wilcox observed that the storage server can rely on the size of the share file combined with the count of leases to unambiguously identify the location of the leases.  This means that it can hold any size share data, even though the field nominally used to hold the size of the share data is only 32 bits wide.

With this patch, the storage server still writes the "size of the share data" field (just in case the server gets downgraded to an earlier version which requires that field, or the share file gets moved to another server which is of an earlier vintage), but it doesn't use it.  Also, with this patch, the server no longer rejects requests to write shares which are >= 2^32 bytes in size, and it no longer rejects attempts to read such shares.

This fixes http://allmydata.org/trac/tahoe/ticket/346 (increase share-size field to 8 bytes, remove 12GiB filesize limit), although there remains open a question of how clients know that a given server can handle large shares (by using the new versioning scheme, probably).

Note that share size is also limited by another factor -- how big of a file we can store on the local filesystem on the server.  Currently allmydata.com typically uses ext3 and I think we typically have block size = 4 KiB, which means that the largest file is about 2 TiB.  Also, the hard drives themselves are only 1 TB, so the largest share is definitely slightly less than 1 TB, which means (when K == 3), the largest file is less than 3 TB.

This patch also refactors the creation of new sharefiles so that only a single fopen() is used.

This patch also helps with the unit-testing of repairer, since formerly it was unclear what repairer should expect to find if the "share data size" field was corrupted (some corruptions would have no effect, others would cause failure to download).  Now it is clear that repairer is not required to notice if this field is corrupted since it has no effect on download.  :-)
 2008-12-31 15:42:26 -07:00
Zooko O'Whielacronx
0687f692b0 trivial: "M-x whitespace-cleanup" on immutable/layout.py 2008-12-31 15:07:02 -07:00
Zooko O'Whielacronx
b8904e023c trivial: remove unused import -- thanks, pyflakes 2008-12-31 15:25:56 -07:00
Zooko O'Whielacronx
c00d20361f rrefutil: generically wrap any errback from callRemote() in a ServerFailure instance 
This facilitates client code to easily catch ServerFailures without also catching exceptions arising from client-side code.
See also:
http://foolscap.lothar.com/trac/ticket/105 # make it easy to distinguish server-side failures/exceptions from client-side
 2008-12-31 14:28:30 -07:00
Zooko O'Whielacronx
3a47031a51 immutable: more detailed tests for checker/verifier/repairer 
There are a lot of different ways that a share could be corrupted, or that attempting to download it might fail.  These tests attempt to exercise many of those ways and require the checker/verifier/repairer to handle each kind of failure well.
 2008-12-31 14:18:38 -07:00
Zooko O'Whielacronx
8d83699783 cli: make startstop_node wait 20 seconds instead of 5 for a process to go away after we signalled it to go away 
Because the unit tests on the VirtualZooko buildslave failed when it took 16 seconds for a process to go away.
Perhaps getting notification after only 5 seconds instead of 20 seconds is desirable, and we should change the unit tests and set this back to 5, but I don't know exactly how to change the unit tests.  Perhaps match this particular warning message about the shutdown taking a while and allow the code under test to pass if the only stderr that it emits is this warning.
 2008-12-30 01:20:22 -07:00
Zooko O'Whielacronx
872e4fc84d doc: sundry amendments to docs and in-line code comments 2008-12-28 16:59:54 -07:00
Zooko O'Whielacronx
9f117dbe8f cli: mark unicode filenames as unsupported -- see #534 for details 2008-12-24 13:28:02 -07:00
Zooko O'Whielacronx
883e51b02d cli: undo the effects of [http://allmydata.org/trac/tahoe/changeset/20081222235453-92b7f-f841e18afb94e1fd95e6dafb799a3d876dd85c69] 
We're just going to mark unicode in the cli as unsupported for tahoe-lafs-1.3.0.  Unicode filenames on the command-line do actually work for some platforms and probably only if the platform encoding is utf-8, but I'm not sure, and in any case for it to be marked as "supported" it would have to work on all platforms, be thoroughly tested, and also we would have to understand why it worked.  :-)
 2008-12-24 09:53:17 -07:00
Zooko O'Whielacronx
25d962c67d test: extend timeout on the hotline file that prevents the client from stopping itself 
The 20-second timeout was apparently tripped on my Powerbook G4 "draco".
 2008-12-21 21:06:29 -07:00
Zooko O'Whielacronx
36df38e58b cli: decode all cli arguments, assuming that they are utf-8 encoded 
Also encode all args to urllib as utf-8 because urllib doesn't handle unicode objects.
I'm not sure if it is appropriate to *assume* utf-8 encoding of cli args.  Perhaps the Right thing to do is to detect the platform encoding.  Any ideas?
This patch is mostly due to François Deppierraz.
 2008-12-22 17:54:53 -07:00
Zooko O'Whielacronx
ce5effbedf util/base32: the identity trans table needn't have any contents -- we are using string.translate solely to delete known chars 2008-12-22 17:48:08 -07:00
Zooko O'Whielacronx
3a26a4bd58 util/base32: allow unicode inputs to a2b() or could_be_base32_encoded(), and encode them with utf-8 before processing them 2008-12-22 17:47:13 -07:00
Zooko O'Whielacronx
dde2376c4e util/base32: loosen the precondition forbidding unicode and requiring str -- now it requires either unicode or str 
Hopefully this will make it so that tests pass with François Deppierraz's patch to fix the tahoe cli's handling of unicode argument.
 2008-12-22 16:22:37 -07:00
Zooko O'Whielacronx
c54783f5e1 immutable: don't catch all exception when downloading, catch only DeadReferenceError and IntegrityCheckReject 2008-12-21 17:41:35 -07:00
Zooko O'Whielacronx
ad58f8b693 immutable: invent download.BadOrMissingHashError which is raised if either hashtree.BadHashError, hashtree.NotEnoughHashesError, and which is a subclass of IntegrityCheckReject 2008-12-21 17:41:30 -07:00
Zooko O'Whielacronx
f1fbd4feae dirnode: don't check MAC on entries in dirnodes 
In an ancient version of directories, we needed a MAC on each entry.  In modern times, the entire dirnode comes with a digital signature, so the MAC on each entry is redundant.
With this patch, we no longer check those MACs when reading directories, but we still produce them so that older readers will accept directories that we write.
 2008-12-21 17:35:18 -07:00
Zooko O'Whielacronx
8b7ce325d7 immutable, checker, and tests: improve docstrings, assertions, tests 
No functional changes, but remove unused code, improve or fix docstrings, etc.
 2008-12-21 15:07:52 -07:00
Zooko O'Whielacronx
c79230c854 cli: if response code from wapi server is not 200 then stop instead of proceeding 
Also, include the data that failed to json parse in an exception raised by the json parser.
 2008-12-20 07:49:18 -07:00
Zooko O'Whielacronx
ec86563326 immutable: when downloading an immutable file, use primary shares if they are available 
Primary shares require no erasure decoding so the more primary shares you have, the less CPU is used.
 2008-12-20 07:14:56 -07:00
Zooko O'Whielacronx
a71a68b31e trivial: remove unused import (thanks, pyflakes) 2008-12-19 13:46:29 -07:00
Zooko O'Whielacronx
471e1f1b9b try to tidy up uri-as-string vs. uri-as-object 
I get confused about whether a given argument or return value is a uri-as-string or uri-as-object.  This patch adds a lot of assertions that it is one or the other, and also changes CheckerResults to take objects not strings.
In the future, I hope that we generally use Python objects except when importing into or exporting from the Python interpreter e.g. over the wire, the UI, or a stored file.
 2008-12-19 08:39:24 -07:00
Zooko O'Whielacronx
7b285ebcb1 immutable: remove the last bits of code (only test code or unused code) which did something with plaintext hashes or plaintext hash trees 2008-12-19 08:18:07 -07:00
Zooko O'Whielacronx
d67a3fe4b1 immutable: use new logging mixins to simplify logging 2008-12-16 18:04:50 -07:00
Zooko O'Whielacronx
d511941136 immutable: refactor ReadBucketProxy a little 2008-12-16 17:53:25 -07:00
Zooko O'Whielacronx
82ee44ed5b debug: pass empty optional arguments to ReadBucketProxy 
because those arguments are about to become non-optional (for other code than test/debug code)
 2008-12-16 17:51:45 -07:00
Zooko O'Whielacronx
8e48a0eeef uri: generalize regexp that recognizes tahoe URLs to work for any host and port 2008-12-16 17:49:30 -07:00
Zooko O'Whielacronx
2b2c2a566b util: logging: refactor some common logging behavior into mixins 2008-12-16 17:38:07 -07:00
Zooko O'Whielacronx
5fb44a3e79 pyutil: assertutil: copy in simplified assertutil from pyutil 2008-12-16 17:37:45 -07:00
Zooko O'Whielacronx
e29e4c04d9 pyutil: assertutil: simplify handling of exception during formatting of precondition message, and reduce dependency to just the Python Standard Library's logging module 2008-12-10 07:10:57 -07:00
Zooko O'Whielacronx
1d84606d52 client: add get_servers() 2008-12-08 17:04:00 -07:00
Brian Warner
278c47b9bd mutable publish: if we are surprised by shares that match what we would have written anyways, don't be surprised. This should fix one of the two #546 problems, in which we re-use a server and forget that we already sent them a share. 2008-12-09 22:44:49 -07:00
Zooko O'Whielacronx
db566db31a immutable: remove unused code to produce plaintext hashes 2008-12-09 16:45:46 -07:00
Zooko O'Whielacronx
c3edae5158 finish renaming 'subshare' to 'block' in immutable/encode.py and in docs/ 2008-12-09 16:33:18 -07:00
Zooko O'Whielacronx
09b7151b6f introducer: fix bug in recent simplification caught by Brian's sharp code-reviewing eye 2008-12-08 17:16:34 -07:00
Zooko O'Whielacronx
a84581fcff introducer: simplify get_permuted_peers() implementation and add get_peers() 2008-12-08 16:57:25 -07:00
Zooko O'Whielacronx
c456ff8591 rename "get_verifier()" to "get_verify_cap()" 2008-12-08 12:44:11 -07:00
Zooko O'Whielacronx
16b20794dd mutable: merge renaming with test patches 2008-12-07 08:45:19 -07:00
Zooko O'Whielacronx
b58875fe43 mutable: rename mutable/node.py to mutable/filenode.py and mutable/repair.py to mutable/repairer.py 
To be more consistent with the immutable layout that I am working on.
 2008-12-07 08:20:08 -07:00
Brian Warner
735b3493f1 web/directory.py: really really fix #553. Unfortunately it's tricky to simulate the behavior of a brower's relative-url handling in a unit test. 2008-12-05 23:14:12 -07:00
francois
db7ad6da12 filenode.py: Fix partial HTTP Range header handling according to RFC2616 
Tahoe webapi was failing on HTTP request containing a partial Range header.
This change allows movies players like mplayer to seek in movie files stored in
tahoe.

Associated tests for GET and HEAD methods are also included
 2008-11-18 07:41:35 -07:00
Brian Warner
ffb5985146 mutable.modify(): after UCWE, publish even if the second invocation of the modifier didn't modify anything. For #551. 2008-12-05 22:49:23 -07:00
Brian Warner
7a0afb59a4 dirnode.py: dirnode.delete which hits UCWE should not fail with NoSuchChildError. Fixes #550. 2008-12-05 22:08:37 -07:00
Brian Warner
fb9af2c7a0 MutableFileNode.modify: pass first_time= and servermap= to the modifier callback 2008-12-05 22:07:10 -07:00
Zooko O'Whielacronx
21d88a0cba web: fix more info links again 
Really, *really* closes #553.
 2008-12-05 15:39:39 -07:00
Zooko O'Whielacronx
dd834cf7e9 web: fix moreinfo link 2008-12-05 15:29:39 -07:00
Zooko O'Whielacronx
93cd0f58a9 web: "More Info" link describes the same file that the "file" link points to, rather than to the file under the same name in this directory 
It's a subtle but real difference.
Fixes #553 -- "More Info" link should point to a file/dir, not a dir+childname .
 2008-12-05 15:05:02 -07:00
Zooko O'Whielacronx
60bbc46a53 minor: fix unused imports -- thanks, pyflakes 2008-12-05 13:07:23 -07:00
Zooko O'Whielacronx
b315619d6b download: refactor handling of URI Extension Block and crypttext hash tree, simplify things 
Refactor into a class the logic of asking each server in turn until one of them gives an answer 
that validates.  It is called ValidatedThingObtainer.

Refactor the downloading and verification of the URI Extension Block into a class named 
ValidatedExtendedURIProxy.

The new logic of validating UEBs is minimalist: it doesn't require the UEB to contain any 
unncessary information, but of course it still accepts such information for backwards 
compatibility (so that this new download code is able to download files uploaded with old, and 
for that matter with current, upload code).

The new logic of validating UEBs follows the practice of doing all validation up front.  This 
practice advises one to isolate the validation of incoming data into one place, so that all of 
the rest of the code can assume only valid data.

If any redundant information is present in the UEB+URI, the new code cross-checks and asserts 
that it is all fully consistent.  This closes some issues where the uploader could have 
uploaded inconsistent redundant data, which would probably have caused the old downloader to 
simply reject that download after getting a Python exception, but perhaps could have caused 
greater harm to the old downloader.

I removed the notion of selecting an erasure codec from codec.py based on the string that was 
passed in the UEB.  Currently "crs" is the only such string that works, so 
"_assert(codec_name == 'crs')" is simpler and more explicit.  This is also in keeping with the 
"validate up front" strategy -- now if someone sets a different string than "crs" in their UEB, 
the downloader will reject the download in the "validate this UEB" function instead of in a 
separate "select the codec instance" function.

I removed the code to check plaintext hashes and plaintext Merkle Trees.  Uploaders do not 
produce this information any more (since it potentially exposes confidential information about 
the file), and the unit tests for it were disabled.  The downloader before this patch would 
check that plaintext hash or plaintext merkle tree if they were present, but not complain if 
they were absent.  The new downloader in this patch complains if they are present and doesn't 
check them.  (We might in the future re-introduce such hashes over the plaintext, but encrypt 
the hashes which are stored in the UEB to preserve confidentiality.  This would be a double-
check on the correctness of our own source code -- the current Merkle Tree over the ciphertext 
is already sufficient to guarantee the integrity of the download unless there is a bug in our 
Merkle Tree or AES implementation.) 

This patch increases the lines-of-code count by 8 (from 17,770 to 17,778), and reduces the 
uncovered-by-tests lines-of-code count by 24 (from 1408 to 1384).  Those numbers would be more 
meaningful if we omitted src/allmydata/util/ from the test-coverage statistics.
 2008-12-05 08:17:54 -07:00
Brian Warner
a0b5b5ab2b test_web: add get_permuted_peers, to unbreak recent checker_results change 2008-12-05 02:12:10 -07:00
Brian Warner
42f8aece05 web checker_results: include a table of servers in permuted order, so you can see the places where new servers have been inserted 2008-12-05 02:03:09 -07:00
Brian Warner
e942ab141b test_system.py: assert less about the stats we get, since shares (and thus allocate() calls) are distributed randomly 2008-12-04 17:27:04 -07:00
Brian Warner
7c4edac582 stats: don't return booleans: it violates the schema. Add a test. 2008-12-04 15:01:24 -07:00
Brian Warner
7cfc74bcc9 test_system.py: don't ask the stats-gatherer to poll: it tolerates failures, so it isn't really giving us enough test coverage. Removing the call will make it more clear that we need to improve the tests later 2008-12-04 15:00:53 -07:00
Peter Secor
a8e515a8bb confwiz.py - removing hardcoded version number 2008-12-02 20:38:31 -07:00
Brian Warner
a2ce315074 CLI: check for pre-existing aliases in 'tahoe create-alias' and 'tahoe add-alias' 2008-12-02 20:20:22 -07:00
Brian Warner
cdbb72d5ff test_cli: pass rc out of do_cli() too 2008-12-02 20:08:28 -07:00
Brian Warner
6958b7fa90 test_storage.py: more windows-vs-readonly-storage fixes 2008-12-02 19:41:02 -07:00
Brian Warner
fe9c3e9529 storage.py: oops, fix windows again, readonly_storage wasn't getting picked up properly 2008-12-02 19:03:17 -07:00
Brian Warner
d3783475b4 test_download.py: remove extra base32 import 2008-12-02 18:31:26 -07:00
Brian Warner
b997e6ca7c test_download: test both mutable and immutable pre-generated shares 2008-12-02 18:30:07 -07:00
Brian Warner
2d70f7fc0b test_download.py: added 'known-answer-tests', to make sure current code can download a file that was created by earlier code 2008-12-02 18:22:08 -07:00
Brian Warner
d295a7afb9 storage.py: unbreak readonly_storage=True on windows 2008-12-01 19:49:46 -07:00
Brian Warner
71c62f8fcb #542 'tahoe create-key-generator': fix the .tac file this creates to be compatible with modern code, add a test 2008-12-01 17:47:21 -07:00
Brian Warner
c5222558f4 storage.py: fix minor typo in comment 2008-12-01 17:25:40 -07:00
Brian Warner
cfba882b30 storage: replace sizelimit with reserved_space, make the stats 'disk_avail' number incorporate this reservation 2008-12-01 17:24:21 -07:00
Brian Warner
39a41f1d85 util/abbreviate: add abbreviated-size parser 2008-12-01 17:24:12 -07:00
Zooko O'Whielacronx
fe6abac87b wui/wapi: change the default port number from 8123 to 3456 to avoid conflict with TorButton 
See ticket #536 for details.
 2008-11-25 17:57:37 -07:00
Brian Warner
7ea0a1316a mutable publish: reinstate the foolscap-reference-token-bug workaround, both for the original reasons and because of an apparent new foolscap bug that's triggered by reference tokens. See #541 for details. 2008-11-25 14:27:35 -07:00
Zooko O'Whielacronx
51c7580fc8 setup: fix missing import -- thanks, pyflakes 2008-11-25 09:55:28 -07:00
Zooko O'Whielacronx
de9edde8f5 setup: correctly detect Arch Linux in platform description 2008-11-25 09:51:18 -07:00
Brian Warner
bc53c24003 dirnode manifest: add verifycaps, both to internal API and to webapi. This will give the manual-GC tools more to work with, so they can estimate how much space will be freed. 2008-11-24 14:40:46 -07:00
Brian Warner
2308440fb6 control.py: use get_buckets() instead of get_version() to measure ping time, because the latter changed recently 2008-11-22 23:13:23 -07:00
Brian Warner
3e25efc010 upload: when using a Helper, insist that it provide protocols/helper/v1 . Related to #538. 2008-11-21 20:29:32 -07:00
Brian Warner
0fab511be5 upload: don't use servers which can't support the share size we need. This ought to avoid #439 problems. Some day we'll have a storage server which advertises support for a larger share size. No tests yet. 2008-11-21 20:28:12 -07:00
Brian Warner
bf06492a90 #538: fetch version and attach to the rref. Make IntroducerClient demand v1 support. 2008-11-21 20:07:27 -07:00
Brian Warner
0eb6b324a4 #538: add remote_get_version() to four main Referenceable objects: Introducer Service, Storage Server, Helper, CHK Upload Helper. Remove unused storage-server get_versions(). 2008-11-21 17:43:52 -07:00
Brian Warner
4ada26101b test_dirnode: add an explainError call 2008-11-19 16:02:12 -07:00
Brian Warner
b84c2c6541 manifest: add storage-index strings to the json results 2008-11-19 16:00:27 -07:00
Brian Warner
815e0673e6 manifest: include stats in results. webapi is unchanged. 2008-11-19 15:03:47 -07:00
Brian Warner
6f6556da22 mutable: respect the new tahoe.cfg 'shares.needed' and 'shares.total' settings 2008-11-19 14:05:01 -07:00
Brian Warner
82fe7ba360 oops, update tests to match 'tahoe stats' change 2008-11-18 20:32:59 -07:00
Brian Warner
677928a1da cli: tahoe stats: abbreviate total sizes too 2008-11-18 20:28:16 -07:00
Brian Warner
9871f3f1b7 cli: 'tahoe stats': add abbreviated size to the histogram. Not sure this actually improves things. 2008-11-18 20:17:36 -07:00
Brian Warner
7391f63fc1 util/abbreviate: little utility to abbreviate seconds and bytes 2008-11-18 20:11:42 -07:00
Brian Warner
0cdf526b50 cli: add 'tahoe check' and 'tahoe deep-check' commands, with primitive reporting code 2008-11-18 19:12:10 -07:00
Brian Warner
9c866ada71 cli: factor out slow-http-operation to a separate module 2008-11-18 19:11:13 -07:00
Brian Warner
5c3e153d0e cli: tahoe stats/manifest: change --verbose to --raw, since I want -v for --verify for check/deep-check/repair 2008-11-18 18:36:08 -07:00
Brian Warner
d657d22c0c test_system: make 'where' strings more helpful, to track down test failures better 2008-11-18 18:29:50 -07:00
Brian Warner
7932fadb5e webapi: add 'summary' string to checker results JSON 2008-11-18 18:28:26 -07:00
Brian Warner
994d97c644 webapi: introducer stats: add 'announcement_distinct_hosts' to the t=json form, to show how many distinct hosts are providing e.g. storage services 2008-11-18 15:30:15 -07:00
Brian Warner
918970b63a 'tahoe create-key-generator': fix help text 2008-11-18 01:47:58 -07:00
Brian Warner
8473a96ada #330: convert stats-gatherer into a .tac file service, add 'tahoe create-stats-gatherer' 2008-11-18 01:46:20 -07:00
Brian Warner
cae54ab118 tahoe.cfg: add controls for k and N (and shares-of-happiness) 2008-11-18 00:29:44 -07:00
Brian Warner
ead0e4d6ca cli: add tests for 'tahoe stats --verbose' 2008-11-17 22:11:14 -07:00
Brian Warner
94de73ddc0 cli: add --verbose to 'tahoe manifest', to show the raw JSON data 2008-11-17 22:02:19 -07:00
Brian Warner
d6a67cd566 dirnode manifest/stats: process more than one LIT file per tree; we were accidentally ignoring all but the first 2008-11-14 22:50:49 -07:00
Brian Warner
fc7cd23bd1 limiter.py: fix stack blowout by inserting an eventual-send between _done and maybe_start_task. This was causing failures during a 'tahoe manifest' of a large set of directories 2008-11-14 21:11:44 -07:00
francois
1a3a464534 test_cli.py: Ensure that we can read our uploaded files back 2008-11-14 07:44:58 -07:00
francois
32b5bc6330 test_cli.py: use str objects instead of unicode ones 
This will hopefully fix failing tests with LC_ALL=C
 2008-11-14 07:41:37 -07:00
Brian Warner
5a60086dbc CLI: add 'tahoe stats', to run start-deep-stats and print the results 2008-11-13 19:43:50 -07:00
Brian Warner
9c9994300e test_system.py: fix new 'tahoe manifest' tests to not break on windows, by providing --node-directory instead of --node-url 2008-11-13 15:27:48 -07:00
francois
5c0c5bfc81 test for bug #534, unicode filenames 
This test assure that uploading a file whose name contains unicode character
doesn't prevent further uploads in the same directory.
 2008-11-13 05:19:51 -07:00
francois
c1f639d230 Fix an filename encoding issue with "tahoe cp" 2008-11-11 14:08:03 -07:00
Brian Warner
0d5cc38a35 web/info.py: use 128-bit ophandles instead of 64-bit 2008-11-12 20:18:42 -07:00
Brian Warner
0d93d6244e CLI: add 'tahoe manifest', which takes a directory and returns a list of things you can reach from it 2008-11-12 20:17:25 -07:00
Brian Warner
c9175de0f2 create_node.py: also remove now-unused import of pkg_resources 2008-11-12 18:47:16 -07:00
Brian Warner
9976bd439a tahoe.cfg: add tub.location, to override the location hints we include in our FURL. This replaces advertised_ip_addresses, which doesn't remain useful enough to retain it. Helps with #517 (Tor). 2008-11-12 18:44:58 -07:00
Zooko O'Whielacronx
74ce1cdc00 setup: remove pkg_resources.require() from create_node.py and add it to runner.py 
Brian correctly points out that the latter is an entry point.
 2008-11-12 15:25:03 -07:00
Zooko O'Whielacronx
14444dbcc0 tests: fix comment 1970-01-05 04:10:55 -07:00
Zooko O'Whielacronx
3a0dc8e4ec tests: add tahoe_cp to the list of scripts that we don't actually have tests for yet 1970-01-05 04:00:58 -07:00
Zooko O'Whielacronx
1aed9fcfa1 setup: the .tac files created by create_node.py call pkg_resources.require() so that they can load tahoe and twisted packages which were installed with setuptools multi-version mode 
Also the create_node.py script itself uses pkg_resources.require() for the same reason.
 1970-01-01 17:50:05 -07:00
Brian Warner
68f366b584 web/info: don't let an unrecoverable file break the page (show ? instead of a size) 2008-11-06 22:51:17 -07:00
Brian Warner
dfa2408157 checker: add is_recoverable() to checker results, make our stub immutable-verifier not throw an exception on unrecoverable files, add tests 2008-11-06 22:35:47 -07:00
Brian Warner
b7f7b2f43e monitor: update interface definition: get_status() can return a Failure 2008-11-06 21:54:52 -07:00
Brian Warner
d7f1f9fd63 web/operations.py: if the operation failed, render the Failure 2008-11-06 21:53:09 -07:00
Peter Secor
ccd029bf15 undoing test change for native_client.php 2008-11-06 16:03:10 -07:00
Brian Warner
926f3958a2 client.py: oops, update FTP/SFTP config names to match current docs 2008-11-05 19:34:42 -07:00
Brian Warner
7c4856c222 SFTP/FTP: merge user/account code, merge docs 2008-11-05 19:25:58 -07:00
Brian Warner
c489a05235 ftp/sftp: move to a new frontends/ directory in preparation for factoring out password-auth component 2008-11-05 14:07:33 -07:00
Brian Warner
4e68faebb4 sftpd: minor debug-logging tweak 2008-11-05 13:45:11 -07:00
Peter Secor
510cb6d8e6 confwiz.py - trying out a new configuration site 2008-11-04 19:18:30 -07:00
Brian Warner
3fc69308eb ftpd: include an (unused) avatar logout callback 2008-11-04 18:01:04 -07:00
Brian Warner
9f908de9e2 #531: implement an SFTP frontend. Mostly works, still lots of debug messages. Still needs tests and auth-by-pubkey in accounts.file 2008-11-04 18:00:22 -07:00
Zooko O'Whielacronx
eae5403e5a shebang: replace "/usr/bin/python" with "/usr/bin/env python" 2008-11-04 18:03:06 -07:00
Zooko O'Whielacronx
9c35ca76a4 util: copy in pyutil.fileutil.ReopenableNamedTemporaryFile 2008-11-04 17:47:15 -07:00
Brian Warner
6fa41e738b immutable: tolerate filenode.read() with a size= that's too big, rather than hanging 2008-11-04 15:29:19 -07:00
Zooko O'Whielacronx
cef24792e7 util: copy in nummedobj from pyutil 2008-11-04 13:55:50 -07:00
Zooko O'Whielacronx
fb85a6280f util: copy in dictutil from pyutil 2008-11-04 13:53:27 -07:00
Brian Warner
a2e195df3e debug catalog-shares: tolerate even more errors on bad files/directories 2008-10-30 14:54:47 -07:00
Brian Warner
ba019bfd3a #527: expire the cached files that are used to support Range: headers, every hour, when the file is unused and older than an hour 2008-10-30 13:39:09 -07:00
Brian Warner
c205a54965 util/cachedir.py: add a cache-directory manager class, which expires+deletes unused files after a while 2008-10-30 13:01:20 -07:00
Brian Warner
1419db6ac7 test_cli: try to fix windows again 2008-10-30 12:32:04 -07:00
Brian Warner
d97c34d622 debug/test_cli: fix error handling for catalog-shares, to make the test stop failing on windows 2008-10-30 12:06:51 -07:00
Brian Warner
b1db6d9ff2 web: add 'Repair' button to checker results when they indicate unhealthyness. Also add the object's uri to the CheckerResults instance. 2008-10-29 18:09:17 -07:00
Brian Warner
6a7d5b893b create_node.py: add 'web.static = public_html' to the initial tahoe.cfg 2008-10-29 17:13:36 -07:00
Brian Warner
d3f2df00be webapi: serve the /static URL tree from /public_html (configurable) 2008-10-29 15:34:31 -07:00
Brian Warner
a9101112a4 catalog-shares command: tolerate errors, log them to stderr, handle v2-immutable shares 2008-10-29 15:10:10 -07:00
Brian Warner
186b64b633 test_web.py: one more line of test coverage 2008-10-28 22:00:15 -07:00
Brian Warner
0350994133 test_web: improve test coverage of PUT DIRURL t=uri replace=false 2008-10-28 21:57:44 -07:00
Brian Warner
a5aebf0cbd web: test (and fix) PUT DIRURL t=uri, which replaces a directory in-place with some other cap 2008-10-28 21:54:46 -07:00
Brian Warner
e1d9169ba3 web/directory.py: slight shuffle to improve test coverage 2008-10-28 21:54:06 -07:00
Brian Warner
248033d78c test_client.py: improve test coverage a bit 2008-10-28 21:43:35 -07:00
Brian Warner
67ae5c1b2f node.py: remove unused old_log() function 2008-10-28 21:35:58 -07:00
Brian Warner
ba26239879 node.py: remove support for the old BASEDIR/authorized_keys.PORT file 2008-10-28 21:34:20 -07:00
Brian Warner
b73c380cdb move testutil into test/common_util.py, since it doesn't count as 'code under test' for our pyflakes numbers 2008-10-28 21:28:31 -07:00
Brian Warner
294e3fb682 util: move PollMixin to a separate file (pollmixin.py), so testutil can be moved into test/ 2008-10-28 21:15:48 -07:00
Brian Warner
3dd9b79e3f control.py: removed unused testutil.PollMixin 2008-10-28 21:03:59 -07:00
Brian Warner
4c2235fcef web/filenode: oops, fix test failures, not everything has a storage index 2008-10-28 18:17:20 -07:00
Brian Warner
6e1db34817 web/filenode: add Accept-Ranges and ETag (for immutable files) headers to GET responses 2008-10-28 18:01:03 -07:00
Brian Warner
b1ca238176 #527: respond to GETs with early ranges quickly, without waiting for the whole file to download. Fixes the alacrity problems with the earlier code. Still needs cache expiration. 2008-10-28 17:56:18 -07:00
Brian Warner
37e3d8e47c #527: support HTTP 'Range:' requests, using a cachefile. Adds filenode.read(consumer, offset, size) method. Still needs: cache expiration, reduced alacrity. 2008-10-28 13:41:04 -07:00
Brian Warner
2d4e895096 iputil.py: avoid a DNS lookup at startup (which may timeout tests when run on a partially-offline host) by using 198.41.0.4 instead of A.ROOT-SERVERS.NET 2008-10-28 13:36:46 -07:00
Brian Warner
914655c52b interfaces.py: promote immutable.encode.NotEnoughSharesError.. it isn't just for immutable files any more 2008-10-27 13:34:49 -07:00
Brian Warner
4b48d94c52 interfaces.IMutableFileNode.download_best_version(): fix return value 2008-10-27 13:20:46 -07:00
Brian Warner
fca158e83a dirnode lookup: use distinct NoSuchChildError instead of the generic KeyError when a child can't be found 2008-10-27 13:15:25 -07:00
Brian Warner
5fd6633289 storage: don't use colons in the corruption-advisory filename, since windows can't tolerate them 2008-10-25 19:46:33 -07:00
Brian Warner
9f21f7cf65 mutable: call remove_advise_corrupt_share when we see share corruption in mapupdate/download/check, tolerate servers that do not implement it 2008-10-24 13:21:28 -07:00
Brian Warner
db37c14ab7 storage: add remote_advise_corrupt_share, for clients to tell storage servers about share corruption that they've discovered. The server logs the report. 2008-10-24 11:52:48 -07:00
Brian Warner
6dbef907ac mutable/servermap.py: fix needs_merge(), it was incorrectly claiming that mixed shares with distinct seqnums needed a merge, causing repair(force=False) to fail 2008-10-23 21:00:24 -07:00
Brian Warner
0383ec5a49 test_web.test_POST_DIRURL_deepcheck: confirm that /operations/HANDLE/ works with or without the slash 2008-10-23 19:17:59 -07:00
Brian Warner
750ab8fa59 web/checker_results.py: remove dead code 2008-10-23 17:17:17 -07:00
Brian Warner
cadb045288 test_web: more test coverage 2008-10-23 17:11:18 -07:00
Brian Warner
a1cfac89f9 webapi: fix t=rename from==to, it used to delete the file 2008-10-23 16:32:36 -07:00
Brian Warner
d4b4cd8ab8 test_system: update test to match web checker results 2008-10-23 16:32:02 -07:00