Zooko O'Whielacronx
6801d5f572
setup: emit the version of python in the list of versions
2008-08-28 15:04:54 -07:00
Brian Warner
3b06aa6a79
servermap: don't log late arrivals, and don't log DeadReferenceError at log.WEIRD
2008-08-26 17:37:29 -07:00
Brian Warner
1668401c16
mutable: make mutable-repair work for non-verifier runs, add tests
2008-08-26 16:34:54 -07:00
Zooko O'Whielacronx
f7b4c45d46
mutable: remove work-around for a flaw in an older version of foolscap
...
We now require "foolscap[secure_connections] >= 0.3.0", per [source:_auto_deps.py].
2008-08-26 08:50:55 -07:00
Brian Warner
a94af879ff
logging: add 'unique-message-ids' (or 'umids') to each WEIRD-or-higher log.msg call, to make it easier to correlate log message with source code
2008-08-25 18:57:59 -07:00
Brian Warner
735aa895b9
logging cleanups: lower DeadReferenceError from WEIRD (which provokes Incidents) to merely UNUSUAL, don't pre-format Failures in others
2008-08-25 17:51:55 -07:00
Zooko O'Whielacronx
def9fc8cf0
checker: make the log() function of SimpleCHKFileVerifier compatible with the log() function of its superclasses and subclasses
2008-08-25 14:44:07 -07:00
Brian Warner
eb695f1c58
mutable/checker: log a WEIRD-level event when we see a hash failure, to trigger an Incident
2008-08-12 20:50:20 -07:00
Brian Warner
97852cd626
immutable checker: add a status_report field
2008-08-12 20:35:30 -07:00
Brian Warner
e82d77c3cf
mutable/servermap: lower the priority of many log messages
2008-08-12 20:35:06 -07:00
Brian Warner
e5252fd5b2
web/deep-check: show the webapi runtime at the bottom of the page
2008-08-12 20:34:26 -07:00
Brian Warner
7a206421b4
CLI: tolerate blank lines in the aliases file
2008-08-12 19:50:50 -07:00
Brian Warner
bcbb51fe4a
test_web: workaround broken HEAD behavior in twisted-2.5.0 and earlier
2008-08-12 19:45:20 -07:00
Brian Warner
6af4619928
test_web: oops, actually use HEAD (instead of GET) in the HEAD test
2008-08-12 19:04:51 -07:00
Brian Warner
4d6b504883
web: use get_size_of_best_version for HEAD requests, provide correct content-type
2008-08-12 19:04:10 -07:00
Brian Warner
d43baa2ad7
mutable: add get_size_of_best_version to the interface, to simplify the web HEAD code, and tests
2008-08-12 19:02:52 -07:00
Brian Warner
014c9b5969
CLI: add 'tahoe debug corrupt-share', and use it for deep-verify tests, and fix non-deep web checker API to pass verify=true into node
2008-08-12 17:05:01 -07:00
Brian Warner
c80e352951
IFilesystemNode: add get_storage_index(), it makes tests easier
2008-08-12 16:14:07 -07:00
Brian Warner
5c0bfd7681
test_system: rename Checker to ImmutableChecker, to make room for a mutable one
2008-08-12 15:59:32 -07:00
Brian Warner
67e7333955
'tahoe debug dump-share': add --offsets, to show section offsets
2008-08-12 14:46:56 -07:00
Brian Warner
022fcf3db8
test_cli: oops, fix tests after recent stdout/stderr cleanup
2008-08-12 14:46:34 -07:00
Brian Warner
acb0297806
scripts/debug: split out dump_immutable_share
2008-08-12 13:55:17 -07:00
Brian Warner
533afd039f
scripts/debug: clean up use of stdout/stderr
2008-08-12 13:52:42 -07:00
Brian Warner
1711b5591f
CLI: move the 'repl' command to 'tahoe debug repl'
2008-08-12 13:40:17 -07:00
Brian Warner
bb33e3e4c2
CLI: move all debug commands (dump-share, dump-cap, find-shares, catalog-shares) into a 'debug' subcommand, and improve --help output
2008-08-12 13:37:32 -07:00
Brian Warner
3cf70697e8
hush a pyflakes warning
2008-08-11 21:24:23 -07:00
Brian Warner
08e229f29a
web/directory: enable verify=true in t=deep-check
2008-08-11 21:24:09 -07:00
Brian Warner
1306b11a76
dirnode: add some deep-check logging
2008-08-11 21:23:38 -07:00
Brian Warner
79576cf75b
checker_results.problems: don't str the whole Failure, just extract the reason string
2008-08-11 21:23:06 -07:00
Brian Warner
d106e411af
checker: add information to results, add some deep-check tests, fix a bug in which unhealthy files were not counted
2008-08-11 21:03:26 -07:00
Brian Warner
17d7f7d983
mutable/checker: rearrange a bit, change checker-results to have a status_report string
2008-08-11 20:20:33 -07:00
Brian Warner
a15ae52122
mutable/servermap: add summarize_version
2008-08-11 20:19:30 -07:00
Brian Warner
376df2eb5a
CLI: make 'tahoe webopen' command accept aliases like 'tahoe ls'
2008-08-11 18:20:23 -07:00
Brian Warner
0d5c8468e9
storage: include disk-free information in the stats-gatherer output
2008-08-06 14:06:02 -07:00
Brian Warner
c7c57bd85c
mutable: more repair tests, one with force=True to check out merging
2008-08-06 12:06:07 -07:00
Brian Warner
a68c9364b8
test/common: add ShouldFailMixin
2008-08-06 12:05:52 -07:00
Brian Warner
572115a405
test_mutable: add comment about minimal-bandwidth repairer, comma lack of
2008-08-06 10:38:50 -07:00
Brian Warner
cdbfccd23e
test_mutable: factor out common setup code
2008-08-06 10:38:04 -07:00
Brian Warner
dd6ec73efa
mutable: start adding Repair tests, fix a simple bug
2008-08-05 23:12:39 -07:00
Brian Warner
3ad0168fbf
test_system: factor out find_shares/replace_shares to a common class, so they can be used by other tests
2008-08-05 18:49:58 -07:00
Brian Warner
56943bc5e2
web: add 'report incident' button at the bottom of the welcome page
2008-08-05 12:09:21 -07:00
Brian Warner
45d12baba2
test_cli: more coverage for 'tahoe put' modifying a mutable file in-place, by filename, closes #441
2008-08-04 13:26:43 -07:00
Brian Warner
4fb03c2510
check_grid.py: update to match new CLI: 'put - TARGET' instead of 'put TARGET'
2008-08-01 19:48:56 -07:00
Brian Warner
b9f4006b96
test_cli: remove windows-worrying newlines from test data
2008-08-01 19:47:34 -07:00
Brian Warner
35afe726c1
test_cli.py: factor out CLITestMixin
2008-08-01 19:29:38 -07:00
Brian Warner
f7d2fcc233
CLI: change one-arg forms of 'tahoe put' to make an unlinked file, fix replace-mutable #441
2008-08-01 19:27:29 -07:00
Brian Warner
2eec38279d
CLI: add create-alias command, to merge mkdir and add-alias into a single (secure-from-argv-snooping) step
2008-08-01 19:10:41 -07:00
Brian Warner
dc57ab4d36
test_cli: add system-based tests for PUT, including a mutable put that fails/todo ( #441 )
2008-08-01 15:10:09 -07:00
Brian Warner
071cbec865
tests: simplify CLI tests that use stdin, now that runner supports it
2008-08-01 15:05:14 -07:00
Brian Warner
f181a0458a
CLI: simplify argument-passing, use options= for everthing, including stdout
2008-08-01 11:46:24 -07:00
Zooko O'Whielacronx
6d3ec390b0
tests: add test that verifier notices any (randomly chosen) bit flipped in the verifiable part of any (randomly chosen) share
...
The currently verifier doesn't (usually) pass this randomized test, hence the TODO.
2008-07-30 17:20:15 -07:00
Zooko O'Whielacronx
86160c3fc9
tests: test that checker doesn't cause reads on the storage servers
...
It would still pass the test if it noticed a corrupted share. (It won't
notice, of course.) But it is required to do its work without causing storage
servers to read blocks from the filesystem.
2008-07-30 16:54:20 -07:00
Zooko O'Whielacronx
29255568df
storage: make storage servers declare oldest supported version == 1.0, and storage clients declare oldest supported version == 1.0
...
See comments in patch for intended semantics.
2008-07-30 15:51:07 -07:00
Zooko O'Whielacronx
dbd1c2397e
tests: use the handy dandy TestCase.mktemp() function from trial to give unique and nicely named directories for each testcase
2008-07-30 15:49:20 -07:00
Zooko O'Whielacronx
6aabaa0797
tests: don't use SignalMixin
...
It seems like we no longer need it, and it screws up something internal in
trial which causes trial's TestCase.mktemp() method to exhibit wrong behavior
(always using a certain test method name instead of using the current test
method name), and I wish to use TestCase.mktemp().
Of course, it is possible that the buildbot is about to tell me that we do
still require SignalMixin on some of our platforms...
2008-07-30 15:35:36 -07:00
Zooko O'Whielacronx
56282613c5
tests: add test_system.Checker which tests basic checking (without verification) functionality
2008-07-28 16:43:17 -07:00
Zooko O'Whielacronx
9d14341c7d
test: add testutil.flip_one_bit which flips a randomly chosen bit of the input string
2008-07-28 16:42:17 -07:00
Zooko O'Whielacronx
fe522e31a4
tests: make it so that you can use common.py's SystemTestMixin.set_up_nodes() more than once with the same introducer
2008-07-28 16:40:29 -07:00
Brian Warner
cd4d3cdeea
download.py: set up self._paused before registering the producer, since they might call pauseProducing right away
2008-07-28 14:57:31 -07:00
Brian Warner
9e22353773
test/common.py: use pre-computed Tub certificates for the system-test mixin, to speed such tests up by maybe 15%. The goal is to encourage more full-grid tests.
2008-07-28 12:44:21 -07:00
Brian Warner
914783acdf
web: add /status/?t=json, with active upload/download ops. Addresses #493 .
2008-07-25 17:41:10 -07:00
Brian Warner
f8221c2c66
web: make t=json stats pages use text/plain, instead of leaving it at text/html
2008-07-25 17:24:27 -07:00
Brian Warner
48c64b7a29
test_system.py: factor SystemTestMixin out of SystemTest
2008-07-25 15:33:49 -07:00
Brian Warner
10bbc740e9
test_system.py: modify system-test setup code in preparation for merge with common.SystemTestMixin
2008-07-25 15:29:31 -07:00
Brian Warner
766cdd9c52
test_system.py: move SystemTestMixin out into common.py, where further improvements will occur
2008-07-25 15:17:58 -07:00
Brian Warner
e317fcc08c
test_system.py: create SystemTestMixin, with less cruft, for faster system-like tests
2008-07-25 15:13:00 -07:00
Brian Warner
afda2a43e4
storage: remove update_write_enabler method, it won't serve the desired purpose, and I have a better scheme in mind. See #489 for details
2008-07-21 17:28:28 -07:00
Zooko O'Whielacronx
9461887e0a
immutable file download: make the ciphertext hash tree mandatory
...
This fixes #491 (URIs do not refer to unique files in Allmydata Tahoe).
Fortunately all of the versions of Tahoe currently in use are already producing
this ciphertext hash tree when uploading, so there is no
backwards-compatibility problem with having the downloader require it to be
present.
2008-07-21 09:31:02 -07:00
Brian Warner
edec917060
test_web: test that save=true filename=unicode doesn't crash
2008-07-18 18:58:57 -07:00
Brian Warner
572c848d98
web: for GET save=true, don't interpret the filename= arg with any character set, just copy the bytes back into the Content-Disposition header. This seems to make it maximally compatible with Firefox and IE7
2008-07-18 18:06:50 -07:00
Brian Warner
879fefe5f3
first pass at a mutable repairer. not tested at all yet, but of course all existing tests pass
2008-07-17 21:09:23 -07:00
Brian Warner
3e95681bad
interfaces: add IRepairable
2008-07-17 17:32:17 -07:00
Brian Warner
923c9c242a
oops, fix import/pyflakes problems
2008-07-17 17:06:20 -07:00
Brian Warner
67db0a4967
deep-check: add webapi, add 'DEEP-CHECK' button to wui, add tests, rearrange checker API a bit
2008-07-17 16:47:09 -07:00
Brian Warner
69156aeb28
dirnode deep-check: add tests of cycles, fix failures
2008-07-17 14:37:04 -07:00
Brian Warner
acf3180fac
dirnode deep-check: rearrange traversal approach, simplify code a bit
2008-07-17 14:25:04 -07:00
Brian Warner
9289433ba3
first pass at deep-checker, no webapi yet, probably big problems with it, only minimal tests
2008-07-16 18:20:57 -07:00
Brian Warner
7b9fff388d
immutable download: remove dead LiteralDownloader, now that we use filenodes for download
2008-07-16 16:31:47 -07:00
Brian Warner
3e9322bcb6
checker: re-enable checker web results (although they just say 'Healthy' right now)
2008-07-16 15:42:56 -07:00
Brian Warner
7394607141
move encode/upload/download/checker.py into a new immutable/ directory. No behavior changes expected.
2008-07-16 13:14:39 -07:00
Brian Warner
94e619c1f6
overhaul checker invocation
...
Removed the Checker service, removed checker results storage (both in-memory
and the tiny stub of sqlite-based storage). Added ICheckable, all
check/verify is now done by calling the check() method on filenodes and
dirnodes (immutable files, literal files, mutable files, and directory
instances).
Checker results are returned in a Results instance, with an html() method for
display. Checker results have been temporarily removed from the wui directory
listing until we make some other fixes.
Also fixed client.create_node_from_uri() to create LiteralFileNodes properly,
since they have different checking behavior. Previously we were creating full
FileNodes with LIT uris inside, which were downloadable but not checkable.
2008-07-15 17:23:25 -07:00
Brian Warner
fd465b4aaf
download: fix stopProducing failure ('self._paused_at not defined'), add tests
2008-07-14 15:25:21 -07:00
Brian Warner
c46f5bc634
storage: rename the latency key names so they sort properly
2008-07-11 21:51:02 -07:00
Brian Warner
bffe14ef3e
add 1%,10% percentiles to the storage server latency output
2008-07-11 21:34:36 -07:00
Brian Warner
60725ed065
storage: add add_lease/update_write_enabler to remote API, revamp lease handling
2008-07-09 18:06:55 -07:00
Brian Warner
de8ce0cb5b
welcome page: add link to statistics page
2008-07-09 17:37:22 -07:00
Brian Warner
a7aa6f8686
implement a mutable checker+verifier. No repair yet. Part of #205 .
2008-07-07 17:36:00 -07:00
Brian Warner
a0de01e971
servermap.shares_available: report N too, not just k
2008-07-07 15:06:15 -07:00
Brian Warner
45405d85c4
interfaces: add verify= and repair= args to check()
2008-07-07 14:37:36 -07:00
Brian Warner
9b6f23ac05
'tahoe dump-share': show verify-cap too
2008-07-07 14:11:02 -07:00
Brian Warner
2074c92dd1
test_mutable.py: add tests for no-servers conditions, closes #463 .
2008-07-07 12:18:10 -07:00
Brian Warner
bbc63e5d3f
web/root.py: fix minor typo
2008-07-07 00:18:16 -07:00
Brian Warner
91c7e0f689
mutable: if there are no servers available, return an error, rather than hanging forever
2008-07-07 00:17:41 -07:00
Brian Warner
7dc8ff0263
re-enable incident-reporting, but disable it in unit tests, because they take 150% longer, and the leftover trailing timers cause errors
2008-07-06 23:49:08 -07:00
Brian Warner
13d0ba72a4
disable incident reporting: unit tests are failing because of the leftover post-incident timer
2008-07-02 18:24:00 -07:00
Brian Warner
d58591ceba
start using Foolscap's 'incident-logging' feature, which requires foolscap-0.2.9
2008-07-02 17:40:29 -07:00
Brian Warner
dfe235bcba
test_storage: oops, update the test to match the leave-incoming/ change
2008-06-26 11:36:17 -07:00
Brian Warner
02ce46cfb4
catalog-shares: ignore incoming/
2008-06-26 11:23:44 -07:00
Brian Warner
edabd2f004
storage.py: leave the storage/shares/incoming/ directory in place when the bucket is closed
2008-06-26 11:07:57 -07:00
Brian Warner
af4e4dc847
storage: record latency stats in a flat dict, not nested, to conform to RIStatsProvider
2008-06-24 17:21:18 -07:00
Brian Warner
dba0ad3d28
test_mutable.py: add more tests of post-mapupdate corruption, to support #474 testing
2008-06-24 11:08:10 -07:00
Brian Warner
4e5b9ee63e
introducer: move the relevant interfaces out to introducer/interfaces.py
2008-06-18 17:04:41 -07:00
Brian Warner
fa453c9b5e
introducer: add old (V1) introducer code, add test framework for compatibility testing
2008-06-18 16:58:34 -07:00
Brian Warner
9ed9eb2048
check_grid.py: make it work, move node start/stop responsibility from Makefile to script
2008-06-19 12:39:52 -07:00
Brian Warner
50d945ea8f
start work on 'check-grid' target, for use in an automated prodnet tester. Just a docstring so far.
2008-06-18 12:57:47 -07:00
Brian Warner
28f4652b96
break introducer up into separate modules in the new allmydata.introducer package
2008-06-18 12:24:16 -07:00
Brian Warner
5bdff74e5b
web: stop using absolute links (or url.here) in forms and pages, since they break behind proxies. Partially addresses #461
2008-06-17 19:49:40 -07:00
Brian Warner
c09c342718
test_storage: yet more coverage
2008-06-17 17:44:10 -07:00
Brian Warner
6b55b8b022
test_storage.py: improve test coverage
2008-06-17 17:01:42 -07:00
Brian Warner
8e52b3d97d
storage.ReadBucketProxy: avoid double-start, this only affected tests
2008-06-17 17:01:29 -07:00
Brian Warner
b62b3974d3
storage: remove unused method
2008-06-17 11:36:32 -07:00
Brian Warner
2154b5751b
test_storage: add coverage for discard_storage
2008-06-16 17:52:40 -07:00
Brian Warner
75e662cc46
test_storage: add coverage for readonly_storage
2008-06-16 17:52:13 -07:00
Brian Warner
166e68f2a3
web /statistics : show 0 instead of None for mutable files
2008-06-16 16:37:04 -07:00
Brian Warner
1ce3b77dde
storage: improve stats, make them accessible via webport /statistics
2008-06-16 16:35:59 -07:00
Brian Warner
6b7ff02e36
storage: measure latency-per-operation, calculate mean/median/percentiles
2008-06-16 15:21:55 -07:00
robk-tahoe
212c19fc45
webish: re-fix the addSlash issue in directory rendering
...
having fixed the relevant unit test not to assert incorrect behaviour of
the node when rendering directories, it's now safe to fix the latter.
2008-06-11 15:35:27 -07:00
robk-tahoe
b803158de7
test_web: fix webapi test around redirection issues
...
this fixes the test_web test test_POST_upload_mutable which chdir's into
a /foo subdirectory, but then later asserts that the web ui redirects the
user back to /foo, which should really be /foo/ since it's a directory.
2008-06-11 15:19:17 -07:00
Brian Warner
4c681847f8
directory.py: revert that addSlash, it causes a test to fail and we don't understand it yet, and we don't need the fix quite yet
2008-06-11 12:59:28 -07:00
robk-tahoe
021a9e370f
webish: fix directory url redirection
...
when rendering a directory in webish, if the url did not correctly include
a trailing slash '/' then the browser will be redirected to a url which
does. this causes any relative links (e.g. the 'other representations'
which are links to files relative to the current directory) to be correctly
intrpreted as relative to the directory in question by the browser.
uses twisted's addSlash mechanism, which is designed for this purpose :-)
2008-06-11 12:46:28 -07:00
Zooko O'Whielacronx
65618ba5a2
wapi/wui: add a trailing slash to the targets of hyperlinks of children of the current directory when those targets are directories
...
This is intended to fix #458 '"other representations" broken in webish ui'.
2008-06-10 19:59:38 -07:00
Brian Warner
baf3b6868e
storage.py: remove unused import
2008-06-10 13:05:44 -07:00
Brian Warner
bc04b8528a
upload: abort the bucket upon any write error, and do it with callRemoteOnly to avoid double errors
2008-06-10 11:55:28 -07:00
Brian Warner
814922a9a1
storage: ignore shares in incoming/, to make clients use other servers during simultaneous uploads
2008-06-10 11:53:10 -07:00
Brian Warner
0aa6d24d62
web: add test for unicode POST when the name comes from name=, not the filename attribute
2008-06-03 17:09:39 -07:00
Brian Warner
9f5075e83d
web: make nickname more visible in the welcome page, closes #361
2008-06-03 15:02:10 -07:00
Brian Warner
9f59ecafbb
webish: make /cap/ equivalent to /uri/, accepting both with the same meanings. Closes #428
2008-06-03 14:34:00 -07:00
Brian Warner
32c89a8d59
web: transform FileTooLargeError into a friendlier '413 Request Entity Too Large' error
2008-06-03 00:03:16 -07:00
Brian Warner
f4496bd553
mutable/publish.py: raise FileTooLargeError instead of an ugly assertion when the SDMF restrictions are exceeded
2008-06-03 00:02:10 -07:00
Brian Warner
5289064dcf
move FileTooLargeError out to a common location
2008-06-03 00:01:15 -07:00
Brian Warner
beb502ac18
CLI: make 'tahoe webopen' use the 'tahoe:' alias properly, instead of the old --dir-cap option
2008-06-02 17:55:54 -07:00
Brian Warner
3e8d58767f
CLI: improve docs w.r.t. aliases, add examples to 'tahoe put' and 'tahoe get' help output. Addresses part of #431
2008-06-02 17:54:56 -07:00
Brian Warner
8c37b8e3af
Don't allow uploads of large files (about 12GiB or larger), since they're doomed to be corrupted. Closes #439
2008-06-02 16:57:01 -07:00
Brian Warner
b6c2f80170
test_system: even more 'cp -r' coverage
2008-05-21 18:40:49 -07:00
Brian Warner
4824d058cc
test_system.py: improve 'cp -r' coverage: exercise copy from tahoe to local disk
2008-05-21 18:36:25 -07:00
Brian Warner
0f8f213baf
CLI: remove the '-r' shortcut for --dir-cap, to make it available for cp -r
2008-05-21 17:55:32 -07:00
Brian Warner
457248f023
tahoe_cp: rewrite, make --recursive work
2008-05-21 17:35:21 -07:00
Brian Warner
15fbe05092
tahoe_ls: improve error message when the target is missing
2008-05-21 17:34:52 -07:00
Brian Warner
282e22abcf
scripts/common_http.py: add comment about why we need .seek
2008-05-21 17:31:19 -07:00
Brian Warner
7ae432abf8
tahoe_cp.py: fix pyflakes complaint
2008-05-21 12:09:13 -07:00
Brian Warner
7afd869854
cli: initial implementation of 'cp -r', probably doesn't work yet
2008-05-21 11:49:22 -07:00
Brian Warner
fe44b8fb4b
util.dictutil: move DictOfSets out to a separate utility module
2008-05-21 09:43:49 -07:00
Brian Warner
0df663b7ae
CLI: implement the easy part of cp (no -r, only two arguments)
2008-05-20 16:56:03 -07:00
Brian Warner
3ac4a734e5
web t=json: add 'mutable' key to the information dict
2008-05-20 15:40:49 -07:00
Brian Warner
9c992c0d06
test_web: improve test coverage of web.common utility code
2008-05-20 15:21:46 -07:00
Brian Warner
8dded7819c
web: fix JSON output for mutable files
2008-05-20 15:14:19 -07:00
Brian Warner
1d8e452184
CLI: add 'list-aliases', factor out get_aliases
2008-05-20 14:36:04 -07:00
Brian Warner
1236bc3408
CLI: add 'ln', just like move but without the delete
2008-05-20 13:30:31 -07:00
Brian Warner
f09f92c371
test_system.py: remove dead code
2008-05-20 13:20:48 -07:00
Brian Warner
3803d591cf
tahoe_ls: improve output formatting
2008-05-20 13:07:50 -07:00
Brian Warner
abe4addbdb
CLI mv: if we can't created the new child, don't delete the old one
2008-05-20 12:49:47 -07:00
Brian Warner
a8ed3208ba
CLI ls: add --readonly-uri to display readonly URIs for all children
2008-05-20 12:49:11 -07:00
Brian Warner
e889463f0c
CLI: add put --mutable, enhance ls to show mutable vs immutable as rw/r-
2008-05-20 12:36:55 -07:00
Brian Warner
e323019708
web: handle PUT mutable=true properly
2008-05-20 12:36:02 -07:00
Brian Warner
d72f75019f
test_web: add HEAD coverage
2008-05-20 11:47:43 -07:00
Brian Warner
b0e306a753
web: more mutable-file coverage
2008-05-20 11:35:47 -07:00
Brian Warner
6b51841ec8
web: even more test coverage
2008-05-20 11:33:14 -07:00
Brian Warner
2eadabbf82
web: improve test coverage, remove some dead code
2008-05-20 11:13:12 -07:00
Brian Warner
88457fbc32
test_web/test_system: improve test coverage
2008-05-19 23:28:52 -07:00
Brian Warner
506ce2dfd0
test_system: fix windows test failure, open files with binary mode
2008-05-19 20:24:12 -07:00
Brian Warner
8e92dfcb50
overhaul CLI: not quite complete but it works a lot better than it used to. The new scheme uses 'tahoe add-alias' and rsync/scp-style 'alias:foo/bar.txt' arguments
2008-05-19 19:28:50 -07:00
Brian Warner
9662e6d986
test/common: oops, forgot the FakeMutableFileNode.get_readonly fix
2008-05-19 18:52:19 -07:00
Brian Warner
ef929e51f9
web: improve test coverage
2008-05-19 18:38:39 -07:00
Brian Warner
fe8fdfac64
web/filenode: remove unreachable code
2008-05-19 18:38:28 -07:00
Brian Warner
daebb42060
web/directory: fix rw_uri output in t=JSON to reflect mutable files properly
2008-05-19 18:37:28 -07:00
Brian Warner
d0685d42ff
web: factor out identical renderHTTP methods
2008-05-19 15:19:25 -07:00
Brian Warner
242b98f750
webish: censor URLs that start with /file and /named too
2008-05-19 14:34:13 -07:00
Brian Warner
fdb4149bad
test_web: remove leftover import to hush pyflakes
2008-05-19 14:28:39 -07:00
Brian Warner
b1d2922ea3
test_system: match webapi change, making new files now returns 201 Created, not 200 OK
2008-05-19 14:09:31 -07:00
Brian Warner
40172047a0
filenode: add is_mutable to non-MutableFileNode classes
2008-05-19 13:03:00 -07:00
Brian Warner
f9cd30d9bc
webish: complete rewrite, break into smaller pieces, auto-create directories, improve error handling
2008-05-19 12:57:04 -07:00
Brian Warner
b8c0217ad5
test_web: implement API changes from the recent webapi.txt overhaul
2008-05-19 12:56:02 -07:00
Brian Warner
869e1ed41a
test_web.py: minor cleanups, improved error reporting
2008-05-19 12:33:39 -07:00
Brian Warner
3360fd24fc
test_web.py: localdir=/localfile= is going away, so remove the tests that exercise it
2008-05-19 12:32:09 -07:00
Brian Warner
b2f8c22fb8
debug dump-share: print size of encprivkey and signature too
2008-05-19 14:24:41 -07:00
Brian Warner
154dca10a0
servermap.py: add size_of_version(), to retrieve the size of a mutable file version
2008-05-19 14:16:02 -07:00
Brian Warner
87c1e8e066
dirnode: add overwrite= to most API calls, defaulting to True. When False, this raises ExistingChildError rather than overwriting an existing child
2008-05-16 16:09:47 -07:00
Brian Warner
1121df6664
testutil.shouldFail: mention the 'which' string in substring failures too
2008-05-16 16:08:38 -07:00
Brian Warner
304abfee32
webish: add /file links, change directory page to use them. This fixes filenames in wget. Closes #221 .
2008-05-14 14:32:21 -07:00
Brian Warner
5098297a2b
introweb.py: tolerate foolscap>=0.2.6, which changed the internals of .locationHints
2008-05-14 13:55:47 -07:00
Brian Warner
ff0b9e2549
test_mutable.py: test replacing a file that has one new outlier share present: closes #272
2008-05-14 13:10:41 -07:00
Brian Warner
26187bfc81
use a weakref cache in the client to manage singleton filenodes/dirnodes, to avoid autocollision. Should close #391 .
2008-05-08 18:02:55 -07:00
Brian Warner
8f4e409677
check_memory: oops, fix HTTP-based modes: trailing slash in node.url vs no slash
2008-05-08 18:01:31 -07:00
Brian Warner
7323ced8e4
test_dirnode: improve coverage of not-mutable-error a bit
2008-05-08 16:53:35 -07:00
Brian Warner
411f578c19
check_memory: don't use fixed webports, they cause occasional test failures
2008-05-08 16:49:27 -07:00
Brian Warner
fabdc28c06
deep-stats: add file-size histogram
2008-05-08 16:19:42 -07:00
Brian Warner
a1ea3d9b37
test_dirnode: the size of the directories varies, tolerate it as low as 500 bytes
2008-05-08 13:42:42 -07:00
Brian Warner
e6ae7a2c60
dirnode: refactor deep-stats a bit
2008-05-08 13:33:07 -07:00
Brian Warner
6c00a70dbc
dirnode: add a deep_stats(), like deep-size but with more information. webish adds t=deeps-size too.
2008-05-08 13:21:14 -07:00
Brian Warner
462ef2a0ac
run a stats provider even if there's no gatherer, since the HTTP /statistics page is then useful. Only run the once-per-second load-monitor if there is a gatherer configured
2008-05-08 11:37:30 -07:00
Brian Warner
3cb361e233
dirnode: use the concurrency limiter in t=manifest and t=deep-size, allow 10 retrievals in parallel
2008-05-07 18:36:37 -07:00
Brian Warner
d76cf6f748
limiter: fix bug that ignores the limit= argument
2008-05-07 18:35:53 -07:00
Brian Warner
15a9f3d01b
test_util: improve ConcurrencyLimiter tests
2008-05-07 17:00:47 -07:00
Brian Warner
c7e441309d
add a basic concurrency limiter utility
2008-05-07 16:53:30 -07:00
Brian Warner
1e4b904b1a
web status: abbreviate file size in the upload+download summary page
2008-05-07 13:53:58 -07:00
Zooko O'Whielacronx
c1184f51e4
docs: fix a few stale comments in code
2008-05-07 08:39:03 -07:00
Brian Warner
fc774803ce
test_stats.py: improve test coverage
2008-04-30 11:52:31 -07:00
Brian Warner
a5a7ba24ef
stats: add tests for CPUUsageMonitor, modify it a bit to facilitate testing
2008-04-30 11:39:13 -07:00
Zooko O'Whielacronx
fc271a0ee9
setup: don't catch ImportError when importing _auto_deps in allmydata/__init__.py
...
Nowadays pkg_resources is a runtime requirement, and if there is something screwed up in the installation, we want an explicit ImportError exception as early as possible.
2008-04-30 13:22:04 -07:00
Brian Warner
88951548ee
test_mutable: update notify_publish() to match new signature
2008-04-29 18:24:57 -07:00
Brian Warner
4b3adb3fcf
mutable stats: track mutable bytes published too
2008-04-29 18:20:05 -07:00
Brian Warner
3be921174b
stats: add CPU-percentage monitor, with 1min/5min/15min moving-window averages, using time.clock()
2008-04-29 18:12:53 -07:00
Zooko O'Whielacronx
f842656ec2
wui: reorganize the welcome.xhtml page
...
Jake Edge tried Tahoe out and didn't notice the /status page. Hopefully with this new organization people like he will see that link more easily. This also addresses drewp's suggestion that the controls appear above the list of servers instead of below. (I think that was his suggestion.) I also reordered the controls.
2008-04-29 15:10:14 -07:00
Brian Warner
351a7f4632
offloaded: add 'resumes' counter to stats: how many upload requests are resuming an interrupted upload?
2008-04-28 13:27:55 -07:00
ben
c85869fc08
support freebsd 6
2008-04-28 00:41:40 -07:00
Brian Warner
186492e620
introducer: only record one announcement per (tubid,service) tuple. Fixes #343 .
2008-04-23 15:05:39 -07:00
Brian Warner
55dfb697a4
client: don't start the IntroducerClient until the Tub is ready, otherwise we will sometimes connect to the introducer (or other clients) before we've done Tub.setLocation, which loses some information on the introducer status page
2008-04-23 14:52:34 -07:00
Brian Warner
a557d44877
test_mutable: test that all servers refusing our share means a publish fails
2008-04-22 18:53:20 -07:00
Brian Warner
13eb00145e
mutable: improve test coverage in Retrieve, when shares change after mapupdate
2008-04-22 17:25:14 -07:00
Brian Warner
acdd98432d
hush pyflakes warning about code that got moved in the recent StallMixin refactoring
2008-04-22 17:14:26 -07:00
Brian Warner
a7632a345e
mutable/servermap: improve test coverage
2008-04-22 16:47:52 -07:00
Brian Warner
81ab1ec785
testutil: factor stall() out into a common location
2008-04-22 16:47:15 -07:00
Brian Warner
57d5341715
test_introducer.py: don't log nodeids as binary goop
2008-04-22 12:54:16 -07:00
Brian Warner
9ad6ed3777
web/status: remove trailing whitespace
2008-04-22 11:51:14 -07:00
Brian Warner
e6074f5dfc
mutable: test write failures, uncoordinated write detection
2008-04-22 11:49:53 -07:00
Brian Warner
7e1f9761e8
mutable: remove some dead code, rearrange use of populate_pubkey
2008-04-21 17:51:22 -07:00
Brian Warner
db4ddfc1bd
mutable: improve testing to exercise the non-cached case
2008-04-21 17:50:50 -07:00
Brian Warner
f39f8ae032
check_speed: rearrange a bit to make it easy to disable certain portions
2008-04-21 17:29:59 -07:00
Brian Warner
8377244c46
mutable status: make the 'total' timing value optional: unit tests fail otherwise
2008-04-21 17:29:12 -07:00
Brian Warner
6af124dc3e
mutable read: enable the cache (written during mapupdate, read during retrieve). This speeds up small-file reads by about 30% over a link with an average 25ms RTT
2008-04-21 17:27:50 -07:00
Brian Warner
e8b0f3d3cb
mutable status: use google-chart-API to draw server response times for servermap update
2008-04-21 16:16:55 -07:00
Brian Warner
a04c2866b1
mutable: improve test coverage slightly
2008-04-21 15:10:50 -07:00
Brian Warner
f16f48a909
download status: add time spent paused by the client (when we're serving over a slow HTTP link)
2008-04-21 12:19:17 -07:00
Zooko O'Whielacronx
78829dcf08
key_generator: make default key size be a constructor argument instead of a class variable, pass default key size of 522 (the smallest that we can do) in unit tests to make them faster
2008-04-22 12:28:18 -07:00
Brian Warner
c375b1503a
mutable/servermap: improve test coverage
2008-04-18 20:13:28 -07:00
Brian Warner
09dcfeae22
mutable: improve test coverage, fix bug in privkey fetching, add .finished to stats, remove dead code
2008-04-18 19:55:12 -07:00
Brian Warner
fc7b75e55f
mutable.node: avoid reentrancy problems in Deferred code on twisted2.5, by adding an eventual-send call
2008-04-18 00:43:29 -07:00
Zooko O'Whielacronx
a504f3395f
let MutableFileNode.__repr__() return something useful even if it isn't not initialize yet
...
This is because I sometimes log or examine a repr of an object from a function which is being called from that object's own __init__()...
(I'm committing this patch in order to test our patch management infrastructure.)
2008-04-17 21:32:38 -07:00
Brian Warner
d7b82f73c5
dirnode: return to 'delete fails if the child wasn't actually there' semantics, to make tests pass. There's a switch to enable/disable this
2008-04-17 20:06:06 -07:00
Brian Warner
d5b182c183
test_mutable: hush pyflakes
2008-04-17 19:57:55 -07:00
Brian Warner
1bff08a7df
dirnode: update to use MutableFileNode.modify
2008-04-17 19:57:04 -07:00
Brian Warner
b6b3b6b1bd
testutil: minor reformatting
2008-04-17 19:56:45 -07:00
Brian Warner
945003ea0f
test_mutable: factor out ShouldFailMixin
2008-04-17 19:55:51 -07:00
Brian Warner
004d3c14bf
mutable: implement MutableFileNode.modify, plus tests
2008-04-17 19:12:42 -07:00
Brian Warner
a379690b04
mutable: replace MutableFileNode API, update tests. Changed all callers to use overwrite(), but that will change soon
2008-04-17 17:51:38 -07:00
Brian Warner
157073d8d8
mutable WIP: merge in patches from current trunk
2008-04-17 13:09:22 -07:00
Brian Warner
ee218c262d
mutable WIP: use fireOnOneErrback when using a DeferredList
2008-04-17 13:11:48 -07:00
Brian Warner
e1838ba217
mutable WIP: clean up status handling, shrink the code a lot, improve test coverage
2008-04-17 13:02:22 -07:00
Brian Warner
a1670497a8
mutable WIP: add servermap update status pages
2008-04-16 19:05:41 -07:00
Brian Warner
842b1f1a70
mutable WIP: oops, fix test_mutable
2008-04-16 18:06:54 -07:00
Brian Warner
f35f20e10b
mutable WIP: publish status doesn't know its size early enough to update the stats_provider
2008-04-16 17:55:17 -07:00
Brian Warner
749c42fa2c
mutable WIP: re-enable publish/retrieve status
2008-04-16 17:49:06 -07:00
Brian Warner
4f07d96667
mutable WIP: improve logging a bit
2008-04-16 15:22:30 -07:00
Brian Warner
c0c8d72b44
mutable WIP: rewrite ServerMap data structure, add tests
2008-04-16 14:49:47 -07:00
Brian Warner
1b4b4cbd4a
mutable WIP: rename NotEnoughPeersError to NotEnoughSharesError
2008-04-15 16:08:32 -07:00
Brian Warner
ca14b99397
mutable WIP: if corrupt shares cause a retrieve to fail, restart it once, ignoring those shares and using different ones
2008-04-15 15:58:02 -07:00
Brian Warner
cf84bae850
mutable WIP: merge conflicts in test_system.py
2008-04-15 15:57:28 -07:00
Brian Warner
d4230d1781
mutable WIP: split mutable.py into separate files. All tests pass.
2008-04-11 14:31:16 -07:00
Brian Warner
0515b1099c
mutable WIP: all tests pass, but publish/retrieve status is still stubbed out
2008-04-10 19:26:35 -07:00
Brian Warner
418407ee5c
mutable: WIP. make Publish work, remove some test scaffolding. test_system still fails.
2008-04-10 18:44:06 -07:00
Brian Warner
b53bbaa43c
mutable.py: checkpointing #303 work part 2, Publish is sketched out
2008-04-07 19:01:28 -07:00
Brian Warner
2c939bfdd3
mutable.py: checkpointing #303 work: retrieve does what I want, now starting in on publish
2008-04-04 17:09:26 -07:00
Brian Warner
68527b25fc
client: add 'node.uptime' to the stats we collect
2008-04-17 11:13:39 -07:00
Brian Warner
7385ddaf30
download: DownloadStopped isn't SCARY, lower the log severity
2008-04-15 16:06:09 -07:00
Brian Warner
944f8babec
test_mutable.py: remove spurious Retrieve during a publish test
2008-04-15 11:20:38 -07:00
Brian Warner
b5d19456f4
web/statistics: fix typo that make immutable-download stats missing
2008-04-15 11:20:04 -07:00
Brian Warner
aebd80feec
upload.py: add some notes: extensions are more like 419 than 1000 bytes
2008-04-15 11:14:58 -07:00
Brian Warner
f153dafaa0
web: return a proper error upon POST with a bad t= value
2008-04-15 11:11:29 -07:00
Brian Warner
6de2a0794f
web: don't break status page when there is no helper running
2008-04-14 23:42:20 -07:00
Brian Warner
a0fa21e442
helper: log SI in ciphertext-read-failed message
2008-04-14 19:41:23 -07:00
Brian Warner
a719b5060f
helper: add SI to logged progress messages
2008-04-14 19:26:53 -07:00
Brian Warner
a2dd4d14a6
helper status: include percentage fetched+pushed, add helper-uploads to the upload/download list
2008-04-14 18:36:27 -07:00
Brian Warner
86578bc3cd
webish: fix 'not running helper' status indicator on welcome page, add tests
2008-04-14 16:28:11 -07:00
Brian Warner
5b8320442a
stats: add /statistics web page to show them, add tests
2008-04-14 14:17:08 -07:00
Brian Warner
8bea663fb2
offloaded: add fetched-percentage to the log message
2008-04-14 14:16:38 -07:00
Brian Warner
500934b72f
helper stats: fix the /helper_status page, the recent conflict merging missed some uses. Added tests, updated the munin plugins to match
2008-04-14 13:18:53 -07:00
Brian Warner
1334a251ca
remove size constraint on ShareData: large directories caused errors which triggered massive memory usage. See #379 for details
2008-04-11 22:51:54 -07:00
Brian Warner
d82f90faaa
mutable.py: catch errors during publish.. previously they were ignored. oops.
2008-04-11 22:51:02 -07:00
Zooko O'Whielacronx
bdda08df8c
copy pyutil.time_format into src/allmydata/util
...
time_format() provides a good format for logging timestamps
2007-08-14 09:33:49 -07:00
Zooko O'Whielacronx
0a6b2b5ff2
doc: slightly clarify an error message
2008-04-14 11:43:05 -07:00
robk-tahoe
a32275771a
stats_gatherer: fix typo in helper stats gathering.
2008-04-10 17:41:42 -07:00
robk-tahoe
766deaa9b6
stats_gatherer: reconcile helper stats gathering
...
I'd implemented stats gathering hooks in the helper a while back.
Brian did the same without reference to my changes. This reconciles
those two changes, encompassing all the stats in both changes,
implemented through the stats_provider interface.
this also provide templates for all 10 helper graphs in the
tahoe-stats munin plugin.
2008-04-10 17:25:44 -07:00
robk-tahoe
f47ebd0814
conflict reconciliation (part 1, stats gathering in helper)
2008-03-27 17:25:16 -07:00
robk-tahoe
5446ea5d67
stats: added stats reporting to the upload helper
...
adds a stats_producer for the upload helper, which provides a series of counters
to the stats gatherer, under the name 'chk_upload_helper'.
it examines both the 'incoming' directory, and the 'encoding' dir, providing
inc_count inc_size inc_size_old enc_count enc_size enc_size_old, respectively
the number of files in each dir, the total size thereof, and the aggregate
size of all files older than 48hrs
2008-03-25 18:19:08 -07:00
robk-tahoe
8620c7271d
key_generator: fix edge case found on windows
...
the windows (cygwin) buildslave has been failing the key generator test
it turns out that the time check on whether to refill the pool, and the
reactor, are interacting such that when the maybe_refill_pool call posted
on the reactor fires, the test on whether to fill the pool fails.
this adds a loop in the failure case to retry each 1s until it is time
to refill the pool, thus mitigating this timing accuracy problem on
windows.
2008-04-10 14:27:35 -07:00
robk-tahoe
35319c3380
stats gathering: fix storage server stats if not tracking consumed
...
the RIStatsProvider interface requires that counter and stat values be
ChoiceOf(float, int, long) the recent changes to storage server to not
track 'consumed' led to returning None as the value of a counter.
this causes violations to be experienced by nodes whose stats are being
gathered.
this patch simply omits that stat if 'consumed' is not being tracked.
2008-04-09 18:23:06 -07:00
robk-tahoe
ef0f523fe0
stats gathering: added counters to upload,download,mutablewatcher
...
counting number of operations, and for immutable files, bytes transferred
2008-04-09 18:08:59 -07:00
robk-tahoe
0d2eb1edf6
stats_gatherer: verbose debug logging
...
one of the storage servers is throwing foolscap violations about the
return value of get_stats(). this adds a log of the data returned
to the foolscap log event stream at the debug level '12' (between
NOISY(10) and OPERATIONAL(20)) hopefully this will facilitate
finding the cause of this problem.
2008-04-09 16:10:53 -07:00
robk-tahoe
27ac71458f
key_generator: remove poll timeouts from test
...
the timeouts on uses of 'poll' were there purely to make sure a test doesn't
poll indefinitely. however having such timeouts makes tests susceptible
to premature timeouts under high load, or on slow machines. (e.g. cygwin
slaves running in virtual machines on loaded hosts)
purportedly trial by default applies a timeout to tests to prevent them
hanging out indefinitely, so these poll timeouts are redundant and cause
intermittent failures on slow hosts. hence they're more bother than they're
worth, and should be culled.
2008-04-08 11:06:58 -07:00
robk-tahoe
8e9e56fdd0
key_generator: fix a typo in the .tac generating create-key-generator
...
verbose is an object attribute, no longer settable via init args
2008-04-08 11:06:06 -07:00
Zooko O'Whielacronx
8783eabf5a
don't do a du on startup if there is no size limit configured
...
This also turns off the production of the "space measurement done" log message, if there is no size limit configured.
2008-04-08 11:36:56 -07:00
robk-tahoe
30eaaa7862
key_generator: fix timing, make tests more robust
...
previously there was an edge case in the timing of expected behaviour
of the key_generator (w.r.t. the refresh delay and twisted/foolscap
delivery). if it took >6s for a key to be generated, then it was
possible for the pool refresh delay to transpire _during_ the
synchronous creation of a key in remote_get_rsa_key_pair. this could
lead to the timer elapsing during key creation and hence the pool
being refilled before control returned to the client.
this change ensures that the time window from a get key request
until the key gen reactor blocks to refill the pool is the time
since a request was answered, not since a request was asked.
this causes the behaviour to match expectations, as embodied in
test_keygen, even if the delay window is dropped to 0.1s
2008-04-03 18:43:46 -07:00
robk-tahoe
c838cfef44
key_generator: up timeouts on keygen test
...
in both these cases, the timeout only serves to abort a stuck test, and
the key_generator should respond more quickly, but seeing test failures
in buildbot on some platforms suggests that the test is too susceptible
to timing issues on loaded buildslaves.
2008-04-03 16:26:24 -07:00
robk-tahoe
1ae2c39862
key_generator: service related cleanups, incorporation into system test
...
this cleans up KeyGenerator to be a service (a subservice of the
KeyGeneratorService as instantiated by the key-generator.tac app)
this means that the timer which replenishes the keypool will be
shutdown cleanly when the service is stopped.
adds checks on the key_generator service and client into the system
test 'test_mutable' such that one of the nodes (clients[3]) uses
the key_generator service, and checks that mutable file creation
in that node, via a variety of means, are all consuming keys from
the key_generator.
2008-04-03 15:57:07 -07:00
robk-tahoe
ccda06b061
key_generator: added a unit test
...
implemented a unit test of basic KeyGenService functionality,
fixed a bug in the timing of pool refreshes
2008-04-03 13:01:43 -07:00
Brian Warner
06dae8fd67
trial_figleaf.py: make our figleaf code compatible with both Twisted-8.x and Twisted-2.5.x
2008-04-02 17:48:55 -07:00
robk-tahoe
5578559b85
added offloaded key generation
...
this adds a new service to pre-generate RSA key pairs. This allows
the expensive (i.e. slow) key generation to be placed into a process
outside the node, so that the node's reactor will not block when it
needs a key pair, but instead can retrieve them from a pool of already
generated key pairs in the key-generator service.
it adds a tahoe create-key-generator command which initialises an
empty dir with a tahoe-key-generator.tac file which can then be run
via twistd. it stashes its .pem and portnum for furl stability and
writes the furl of the key gen service to key_generator.furl, also
printing it to stdout.
by placing a key_generator.furl file into the nodes config directory
(e.g. ~/.tahoe) a node will attempt to connect to such a service, and
will use that when creating mutable files (i.e. directories) whenever
possible. if the keygen service is unavailable, it will perform the
key generation locally instead, as before.
2008-04-01 18:45:13 -07:00
Brian Warner
f9e261d939
introducer.py: accelerate reconnection after being offline. Closes #374 .
...
When we establish any new connection, reset the delays on all the other
Reconnectors. This will trigger a new batch of connection attempts. The idea
is to detect when we (the client) have been offline for a while, and to
connect to all servers when we get back online. By accelerating the timers
inside the Reconnectors, we try to avoid spending a long time in a
partially-connected state (which increases the chances of causing problems
with mutable files, by not updating all the shares that we ought to).
2008-03-31 15:28:45 -07:00
Brian Warner
d6be5116f5
storage: emit log messages on bucket allocate/read and mutable writev
2008-03-27 17:33:58 -07:00
Brian Warner
2c96a32633
helper: add more stats to webapi, at /helper_status
2008-03-27 16:46:08 -07:00
Brian Warner
36f5c025a6
helper: add stats for the gatherer, show some on the webish welcome page
2008-03-27 15:55:32 -07:00
Brian Warner
9b3a32d0b3
add GET /uri/URI/?t=deep-size, to compute the total size of immutable files reachable from a given directory
2008-03-27 11:33:42 -07:00
Brian Warner
3f5fed0bd4
node.py: make twistd.pid world-readable, so stats-gathering tools (specifically memory-measuring munin plugins) can see it
2008-03-26 18:37:54 -07:00
Brian Warner
04b690ac9e
web-status: client methods like list_all_uploads() return Upload instances,
...
not status instances. Fix this. The symptom was that following a link like
'up-123' that referred to an old operation (no longer in memory) while an
upload was active would get an ugly traceback instead of a "no such resource"
message.
2008-03-26 18:20:07 -07:00
Brian Warner
4531d1e953
introweb.py: add ?t=json, to provide machine-readable subscriber counts
2008-03-25 12:56:12 -07:00
Brian Warner
4b46f1cd53
encode.py: also record the size, along with plaintext_hash and SI
2008-03-24 19:08:15 -07:00
Brian Warner
15ef9f624d
encode: log a plaintext hash and SI for each upload. This will allow the log gatherer to correlate the two, to better measure the benefits of convergence
2008-03-24 18:55:37 -07:00
robk-tahoe
7f14b93341
confwiz: set a convergence domain based on root_dir upon config
...
when the confwiz configures a node (i.e. typically once on mac, once per
install on windows) in addition to writing the root_dir.cap retrieved from
the native_client backend into a config file, it additionally writes a hash
thereof into the 'convergence' config file.
this causes uploads from this node to use a consistent 'convergence' hashing
value matching any other nodes with the same configured root_dir, i.e. for
the most part other systems installed and configured on the same account.
2008-03-24 15:47:12 -07:00
robk-tahoe
d25d3065d1
mutable: revise a couple of error messages
...
at brian and zooko's suggestion, reword an error message encountered when
multiple writers are racing to make overlapping changes to a directory
2008-03-24 15:46:28 -07:00
Brian Warner
1869d7c29b
upload.py: remove spurious trailing quotes that confuse syntax-highlighting
2008-03-24 15:51:19 -07:00
Brian Warner
1e097766c9
disable plaintext hashes in shares, but leave a switch to turn it back on
2008-03-24 13:39:51 -07:00
Brian Warner
7b21054c33
UNDO: upload: stop putting plaintext and ciphertext hashes in shares.
...
This removes the guess-partial-information attack vector, and reduces
the amount of overhead that we consume with each file. It also introduces
a forwards-compability break: older versions of the code (before the
previous download-time "make hashes optional" patch) will be unable
to read files uploaded by this version, as they will complain about the
missing hashes. This patch is experimental, and is being pushed into
trunk to obtain test coverage. We may undo it before releasing 1.0.
2008-03-23 15:35:54 -07:00
Zooko O'Whielacronx
dfdbb95648
fix check-memory to use new upload API (which requires a "convergence" argument), and change it to measure convergence instead of random-key, since convergence is the use case we care about more
2008-03-24 15:28:04 -07:00
Zooko O'Whielacronx
fc3bd0c987
use added secret to protect convergent encryption
...
Now upload or encode methods take a required argument named "convergence" which can be either None, indicating no convergent encryption at all, or a string, which is the "added secret" to be mixed in to the content hash key. If you want traditional convergent encryption behavior, set the added secret to be the empty string.
This patch also renames "content hash key" to "convergent encryption" in a argument names and variable names. (A different and larger renaming is needed in order to clarify that Tahoe supports immutable files which are not encrypted content-hash-key a.k.a. convergent encryption.)
This patch also changes a few unit tests to use non-convergent encryption, because it doesn't matter for what they are testing and non-convergent encryption is slightly faster.
2008-03-24 09:46:06 -07:00
Brian Warner
7996131a0a
upload: stop putting plaintext and ciphertext hashes in shares.
...
This removes the guess-partial-information attack vector, and reduces
the amount of overhead that we consume with each file. It also introduces
a forwards-compability break: older versions of the code (before the
previous download-time "make hashes optional" patch) will be unable
to read files uploaded by this version, as they will complain about the
missing hashes. This patch is experimental, and is being pushed into
trunk to obtain test coverage. We may undo it before releasing 1.0.
2008-03-23 15:35:54 -07:00
Brian Warner
553367d567
download: make plaintext and ciphertext hashes in the UEB optional.
...
Removing the plaintext hashes can help with the guess-partial-information
attack. This does not affect compatibility, but if and when we actually
remove any hashes from the share, that will introduce a
forwards-compatibility break: tahoe-0.9 will not be able to read such files.
2008-03-23 14:46:49 -07:00
robk-tahoe
f14c498e4f
confwiz: refine the descriptive text on the conf wiz pages
2008-03-18 17:06:47 -07:00
robk-tahoe
605d693fa3
confwiz: reworked confwiz look and feel
...
this changes the confwiz to have a look and feel much more consistent
with that of the innosetup installer it is launched within the context
of. this applies, naturally, primarily to windows.
2008-03-18 16:15:36 -07:00
robk-tahoe
e75c417bb5
webish mkdir-p: added unit test
...
added a test for the simple mkdir-p hack I added yesterday
checks that mkdir-p can create a directory hierarchy, and that resubmitting
a request for the same path yields the existing dir's uri
2008-03-18 13:11:08 -07:00
robk-tahoe
08fd16ef78
add a mkdir-p POST handler
...
this adds a t=mkdir-p call to directories (accessed by their uri as
/uri/<URI>?t=mkdir=p&path=/some/path) which returns the uri for a
directory at a specified path before the given uri, regardless of
whether the directory exists or whether intermediate directories
need to be created to satisfy the request.
this is used by the migration code in MV to optimise the work of
path traversal which was other wise done on every file PUT
2008-03-17 18:13:01 -07:00
Brian Warner
75b52cda5c
web/status.py: guard against file_size=None, closes #354
2008-03-17 13:31:10 -07:00
Zooko O'Whielacronx
7bc06872ac
refactor one of the dispatch routines in webish.py
...
The behavior is intended to be unchanged by this refactoring. Unit tests show no change in behavior.
2008-03-20 12:11:09 -07:00
Brian Warner
2ef70ab814
mutable.py: split replace() into update() and overwrite(). Addresses #328 .
2008-03-12 18:00:43 -07:00
Zooko O'Whielacronx
0aa0efa123
storage servers announce that they will support clients as old as v0.8.0
...
Not that anyone pays attention to what storage servers claim about what versions they will support.
2008-03-13 09:10:11 -07:00
Brian Warner
45a1e655c7
introducer: record a timestamp with each announcement, and display it on the introducer's web page
2008-03-11 19:33:19 -07:00
Brian Warner
29e23626ad
introducer: record a timestamp with each subscriber, and display it on the introducer's web page
2008-03-11 19:28:37 -07:00
Brian Warner
89be2e1bea
introweb: combine announcement and subscriber information to show version+nickname for each client
2008-03-11 19:21:29 -07:00
Brian Warner
c21d30c320
client: publish a 'stub client' announcement to the introducer, to provide version/nickname information for each client
2008-03-11 19:20:10 -07:00
Brian Warner
0c40513e75
webish: put a trailing slash in the node.url file, since our unit tests use it, and some versions of twisted or nevow don't automatically add one
2008-03-11 19:01:18 -07:00
Brian Warner
fd5a922cad
introweb.py: tolerate non-setLocationed client tubs
2008-03-11 18:09:13 -07:00
Brian Warner
3f5abe5cba
oops, add introducer.xhtml
2008-03-11 17:41:03 -07:00
Brian Warner
810ba68343
add a webserver for the Introducer, showing service announcements and subscriber lists
2008-03-11 17:36:25 -07:00
Brian Warner
10d3ea5045
increase remote-interface size limits to 16EiB by not casually using 'int' as a constraint
2008-03-11 10:50:31 -07:00
Brian Warner
c727348d85
mutable: tolerate multiple encodings, using whichever version is recoverable first. Closes #312
2008-03-11 00:26:00 -07:00
Brian Warner
791482cf8d
test_mutable: improve multiple-encodings test coverage
2008-03-10 23:47:35 -07:00
Brian Warner
9ca55b8b79
test_mutable: make test-multiple-encodings work
2008-03-10 23:16:28 -07:00
Brian Warner
57bd23f35f
test_mutable: more test coverage, building up a framework to cause reads to occur in a specific order
2008-03-10 22:15:43 -07:00
Brian Warner
2464899f9a
mutable: increase max segsize to 3.5MB, to allow dirnodes with about 10k entries
2008-03-10 20:39:55 -07:00
Zooko O'Whielacronx
a8a85da326
make the nodeurl start with '127.0.0.1' instead of with 'localhost'
...
This is because there exist in the wild computers that are misconfigured so that 'localhost' doesn't resolve to 127.0.0.1. On those computers, using 'localhost' for the nodeurl is a security problem, because the user commonly sends valuable caps to the nodeurl.
2008-03-11 06:18:03 -07:00
Brian Warner
11445b5109
test_mutable: exercise short reads too
2008-03-10 18:08:23 -07:00
robk-tahoe
a1f2c9d624
confwiz: write accountname to a file
...
motivated simply by a desire to be able to identify 'noderoot' directories for
debugging and testing, the confwiz now writes an 'accountname' files based on
what account was used when the node was configured. this is not currently read
by or used by any code in the system, but helps identify directories from testing.
2008-03-10 17:50:44 -07:00
Brian Warner
be5a6147b4
test_mutable: test all hash-failure cases except a corrupted encrypted private key
2008-03-10 17:46:52 -07:00
Brian Warner
647734cd3b
test_mutable: add Roundtrip test, suitable for new share-mangling tests
2008-03-10 16:14:08 -07:00
Brian Warner
9e460cd22e
mutable: cosmetic changes
2008-03-10 15:44:05 -07:00
Brian Warner
d6cde55c63
mutable: minor refactoring of _do_read, to make other tests easier
2008-03-10 15:42:56 -07:00
Zooko O'Whielacronx
cad26e000d
fix pyflakes warning
2008-03-10 16:14:03 -07:00
Brian Warner
567979ba85
lower max_segsize for immutable files to 128KiB, to make download-progress smoother and improve alacrity. Closes #252 . Note that mutable files continue to use 1MiB segments.
2008-03-07 19:24:51 -07:00
robk-tahoe
bd58f6f799
macapp: new mac icon
...
this provides a new icon for the .app bundle
also removes the setting of the dock icon from within wx
(which previously used a different icon)
2008-03-07 18:48:28 -07:00
Brian Warner
8815b30b9b
webish: show storage sizelimit, abbreviate current usage
2008-03-06 21:16:38 -07:00
robk-tahoe
6177a958d0
macapp: simplify node startup failure reporting
...
1. changed the node's exit-on-error behaviour. rather than logging debug and
then delegating to self for _abort_process() instead simply delegate to self
_service_startup_failed(failure) to report failures in the startup deferred
chain. subclasses then have complete control of handling and reporting any
failures in node startup.
2. replace the convoluted wx.PostEvent() glue for posting an event into the
gui thread with the simpler expedient of wx.CallAfter() which is much like
foolscap's eventually() but also thread safe for inducing a call back on the
gui thread.
2008-03-06 15:09:04 -07:00
robk-tahoe
ee67d788cd
macapp: report failure of node startup to the user
...
in certain cases (e.g. the node.pem changed but old .furls are in private/)
the node will abort upon startup. previously it used os.abort() which in these
cases caused the mac gui app to crash on startup with no explanation.
this changes that behaviour from calling os.abort() to calling
node._abort_process(failure) which by default calls os.abort(). this allows
that method to be overridden in subclasses.
the mac app now provides and uses such a subclass of Client, so that failures
are reported to the user in a message dialog before the process exits.
this uses wx.PostEvent() with a custom event type to signal from the reactor
thread into the gui thread.
2008-03-06 13:53:21 -07:00
Brian Warner
3a8bd1b772
webish: add more share information to upload status, including assisted uploads
2008-03-05 19:51:51 -07:00
Brian Warner
dd4a951770
webish: add publish status
2008-03-05 18:41:10 -07:00
robk-tahoe
2c81988ed3
confwiz: add nickname determination
...
the confwiz now uses socket.gethostname() if a 'nickname' file doesn't already
exist, and passes that nickname into the 'record_install' method on the backend,
so that the moniker can be recorded in the system table.
2008-03-05 16:34:18 -07:00
Brian Warner
3a331f6822
webish: split out 'unlinked' operations
2008-03-05 15:12:42 -07:00
Brian Warner
4fa622f9b1
webish: this file is too big, start breaking it into pieces, beginning with status
2008-03-05 14:59:56 -07:00
Brian Warner
f46cb5cd6d
mutable-retrieve: only record server response times for queries that we don't ignore
2008-03-04 21:10:52 -07:00
Brian Warner
8160a4f2f2
mutable: when retrieving, don't try to grab the encprivkey, to save a roundtrip
2008-03-04 21:08:24 -07:00
Brian Warner
ec23da28a6
webish: add more mutable-retrieve timing status
2008-03-04 21:04:36 -07:00
Brian Warner
ca1a1762e2
web: status: add 'started' timestamps to all operations
2008-03-04 18:50:44 -07:00
