ExternalVendorCode/tahoe-lafs
1
0
Fork 0
mirror of https://github.com/tahoe-lafs/tahoe-lafs.git synced 2025-01-03 19:54:09 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
3,404 Commits 185 Branches 90 Tags 89 MiB
c7cd3f38e7
Commit Graph

1940 Commits

Author SHA1 Message Date
robk-tahoe
4d9f499de1 macapp: changes to support aliases, updated tahoefuse command line options 
the tahoefuse command line options changed to support the runtests harness,
and as part of that gained support for named aliases via --alias

this changes the mac app's invocation of tahoefuse to match that, and also
changes the gui to present the list of defined aliases as valid mounts

this replaces the previous logic which examined the ~/.tahoe/private directory
looking for files ending in '.cap' - an ad-hoc alias mechanism.

if a file is found matching ~/.tahoe/private/ALIASNAME.icns then that will still
be passed to tahoefuse as the icon to display for that filesystem. if no such
file is found, the allmydata icon will be used by default.

the '-olocal' option is passed to tahoefuse.  this is potentially contentious.
specifically this is telling the OS that this is a 'local' filesystem, which is
intended to be used to locally attached devices.  however leopard (OSX 10.5)
will only display non-local filesystems in the Finder's side bar if they are of
fs types specifically known by Finder to be network file systems (nfs, cifs, 
webdav, afp)  hence the -olocal flag is the only way on leopard to cause finder
to display the mounted filesystem in the sidebar, but it displays as a 'device'.
there is a potential (i.e. the fuse docs carry warnings) that this may cause
vague and unspecified undesirable behaviour.
(c.f. http://code.google.com/p/macfuse/wiki/FAQ specifically Q4.3 and Q4.1)
 2008-09-24 18:01:28 -07:00
robk-tahoe
ed9873c2a9 CLI: rework webopen, and moreover its tests w.r.t. path handling 
in the recent reconciliation of webopen patches, I wound up adjusting
webopen to 'pass through' the state of the trailing slash on the given
argument to the resultant url passed to the browser.  this change 
removes the requirement that arguments must be directories, and allows
webopen to be used with files.  it also broke the tests that assumed
that webopen would always normalise the url to have a trailing slash.

in fixing the tests, I realised that, IMHO, there's something deeply
awry with the way tahoe handles paths; specifically in the combination
of '/' being the name of the root path within an alias, but a leading
slash on paths, e.g. 'alias:/path', is catagorically incorrect. i.e.
 'tahoe:' == 'tahoe:/' == '/' 
but 'tahoe:/foo' is an invalid path, and must be 'tahoe:foo'

I wound up making the internals of webopen simply spot a 'path' of
'/' and smash it to '', which 'fixes' webopen to match the behaviour
of tahoe's path handling elsewhere, but that special case sort of
points to the weirdness.

(fwiw, I personally found the fact that the leading / in a path was
disallowed to be weird - I'm just used to seeing paths qualified by
the leading / I guess - so in a debate about normalising path handling
I'd vote to include the /)
 2008-09-24 09:45:23 -07:00
robk-tahoe
2c9a854f10 CLI: reconcile webopen changes 
I think this is largely attributable to a cleanup patch I'd made
which never got committed upstream somehow, but at any rate various
conflicting changes to webopen had been made. This cleans up the
conflicts therein, and hopefully brings 'tahoe webopen' in line with
other cli commands.
 2008-09-24 08:20:02 -07:00
robk-tahoe
8f28d64d77 cli: cleanup webopen command 
moved the body of webopen out of cli.py into tahoe_webopen.py

made its invocation consistent with the other cli commands, most
notably replacing its 'vdrive path' with the same alias parsing,
allowing usage such as 'tahoe webopen private:Pictures/xti'
 2008-06-18 13:19:40 -07:00
robk-tahoe
0042881736 macapp: changed to remove 'Tahoe' from .app name 
Change the build product from 'Allmydata Tahoe' to 'Allmydata'
more inkeeping with the branding of the Allmydata product
 2008-06-10 17:31:45 -07:00
Brian Warner
53f53b7c82 add --syslog argument to 'tahoe start' and 'tahoe restart', used to pass --syslog to twistd for non-Tahoe nodes (like cpu-watcher) 2008-09-24 18:03:02 -07:00
Brian Warner
54fce5c066 webapi: survive slashes in filenames better: make t=info and t=delete to work, and let t=rename fix the problem 2008-09-24 13:35:05 -07:00
Zooko O'Whielacronx
7339e09c72 setup: when detecting platform, ask the Python Standard Library's platform.dist() before executing lsb_release, and cache the result in global (module) variables 
This should make it sufficiently fast, while still giving a better answer on Ubuntu than platform.dist() currently does, and also falling back to lsb_release if platform.dist() says that it doesn't know.
 2008-09-24 11:09:22 -07:00
Brian Warner
dd9171eb72 node.py: add BASEDIR/keepalive_timeout and BASEDIR/disconnect_timeout, to set/enable the foolscap timers, for #521 2008-09-24 10:51:12 -07:00
Zooko O'Whielacronx
ff08bab0c4 immutable: remove unused imports (thanks, pyflakes) 2008-09-23 12:26:10 -07:00
Zooko O'Whielacronx
fa302544fa immutable: refactor immutable filenodes and comparison thereof 
* the two kinds of immutable filenode now have a common base class
* they store only an instance of their URI, not both an instance and a string
* they delegate comparison to that instance
 2008-09-23 11:52:49 -07:00
Zooko O'Whielacronx
2ef977d85a setup: try parsing /etc/lsb-release first, then invoking lsb_release, because the latter takes half-a-second on my workstation, which is too long 
Also because in some cases the former will work and the latter won't.
This patch also tightens the regexes so it won't match random junk.
 2008-09-23 10:14:31 -07:00
Zooko O'Whielacronx
37dd91238b setup: fix a cut-and-paste error in the fallback to parsing /etc/lsb-release 2008-09-23 09:55:51 -07:00
Zooko O'Whielacronx
8a35d2ed83 setup: if executing lsb_release doesn't work, fall back to parsing /etc/lsb-release before falling back to platform.dist() 
An explanatio of why we do it this way is in the docstring.
 2008-09-23 09:28:58 -07:00
Zooko O'Whielacronx
e2b1ce13f5 setup: if invoking lsb_release doesn't work (which it doesn't on our etch buildslave), then fall back to the Python Standard Library's platform.dist() function 2008-09-23 08:48:20 -07:00
Zooko O'Whielacronx
8874e27da0 setup: fix bug in recent patch to use allmydata.get_package_versions() to tell the foolscap app-version-tracking what's what 2008-09-22 17:13:47 -07:00
Zooko O'Whielacronx
992339b97e setup: when using the foolscap "what versions are here?" feature, use allmydata.get_package_versions() instead of specifically importing allmydata, pycryptopp, and zfec 2008-09-22 17:03:51 -07:00
Zooko O'Whielacronx
5075c12725 setup: simplify the implementation of allmydata.get_package_versions() and add "platform" which is a human-oriented summary of the underlying operating system and machine 2008-09-22 16:53:54 -07:00
Brian Warner
c6999d219e use foolscap's new app_versions API, require foolscap-0.3.1 2008-09-20 11:38:53 -07:00
Brian Warner
063f85d157 BASEDIR/nickname is now UTF-8 encoded 2008-09-20 11:37:13 -07:00
Brian Warner
0a59991ce6 various: use util.log.err instead of twisted.log.err, so we get both Incidents and trial-test-flunking 2008-09-20 10:35:45 -07:00
Brian Warner
8854d0c1b5 util.log: send log.err to Twisted too, so that Trial tests are flunked 2008-09-20 10:34:27 -07:00
Brian Warner
0799e09910 test_cli: disable generate-keypair test on OS-X, pycryptopp still has a bug 2008-09-19 12:38:55 -07:00
Brian Warner
6607fdc586 CLI: add 'tahoe admin generate-keypair' command 2008-09-18 17:11:33 -07:00
Brian Warner
99d5a8d8b9 web: add 'more info' pages for files and directories, move URI/checker-buttons/deep-size/etc off to them 2008-09-17 22:00:41 -07:00
Peter Secor
c253655f66 conf_wiz.py - updating version numbers in file, should really get these from a TAG or conf file 2008-09-16 17:45:47 -07:00
Brian Warner
51612cd1be webish: add an extra newline to JSON output 2008-09-15 13:43:14 -07:00
Brian Warner
f00d1961d2 startstop_node.py: find twistd in our supportlib if we had to build Twisted as a setuptools dependency. This is a form of cgalvan's #505 patch, simplified because now 'setup.py trial' takes care of sys.path and PYTHONPATH 2008-09-11 19:51:38 -07:00
Brian Warner
ca273eeaf1 web: fix output=JSON, add buttons for repair/json to the 'run deep-check' form 2008-09-10 14:11:37 -07:00
Brian Warner
f570ad7ba5 disallow deep-check on non-directories, simplifies the code a bit 2008-09-10 13:44:58 -07:00
Brian Warner
4bb88fd2ee dirnode: refactor recursive-traversal methods, add stats to deep_check() method results and t=deep-check webapi 2008-09-10 01:45:04 -07:00
Brian Warner
f6eeb3161f dirnode: cleanup, make get_verifier() always return a URI instance, not a string 2008-09-10 01:37:55 -07:00
Brian Warner
28a9f8f076 test_system: check t=deep-stats too 2008-09-09 23:54:57 -07:00
Brian Warner
12ff8a6eae test_system: add deep-check-JSON tests, fix a bug 2008-09-09 23:14:16 -07:00
Brian Warner
ce7fcbde36 test_system: oops, re-enable some tests that got bypassed 2008-09-09 23:02:45 -07:00
Brian Warner
ba336aed3e test_system: add deep-stats test 2008-09-09 22:56:34 -07:00
Brian Warner
80d8f3e862 hush pyflakes 2008-09-09 19:50:17 -07:00
Brian Warner
1d2d6a35a6 checker results: add output=JSON to webapi, add tests, clean up APIs 
to make the internal ones use binary strings (nodeid, storage index) and
the web/JSON ones use base32-encoded strings. The immutable verifier is
still incomplete (it returns imaginary healty results).
 2008-09-09 19:45:17 -07:00
Brian Warner
04513e3ac5 immutable verifier: provide some dummy results so deep-check works, make the tests ignore these results until we finish it off 2008-09-09 18:08:27 -07:00
Brian Warner
7fb3308498 mutable checker: even more tests. Everything in ICheckerResults should be covered now, except for immutable-verify which is incomplete 2008-09-09 17:57:06 -07:00
Brian Warner
84a5778507 checker results: more tests, update interface docs 2008-09-09 17:30:10 -07:00
Brian Warner
9676586732 mutable checker: oops, fix redefinition of 'healthy' (numshares < N, not numshares < k, which is 'recoverable' not 'healthy') 2008-09-09 17:28:53 -07:00
Brian Warner
f895e39d48 checker results: more tests, more results. immutable verifier tests are disabled until they emit more complete results 2008-09-09 17:15:46 -07:00
Brian Warner
90b934eb71 checker: add tests, add stub for immutable check_and_repair 2008-09-09 16:34:49 -07:00
Brian Warner
137750eca6 interfaces.py: minor improvement to IDirectoryNode.set_node 2008-09-09 16:34:16 -07:00
Brian Warner
918b0543b9 test_system: make log() tolerate the format= form 2008-09-07 20:03:36 -07:00
Brian Warner
af2231563e immutable/checker: make log() tolerate the format= form 2008-09-07 20:03:08 -07:00
Brian Warner
3408d552cd checker: overhaul checker results, split check/check_and_repair into separate methods, improve web displays 2008-09-07 12:44:56 -07:00
Brian Warner
e2c484a761 introducer: add get_nickname_for_peerid 2008-09-05 22:07:00 -07:00
Brian Warner
bab799297f testutil.PollMixin: use a custom exception (and convert it) to avoid the ugly 'stash' cycle 2008-09-02 20:32:51 -07:00
Zooko O'Whielacronx
037c9157dc setup: simplify parsing of python version number 2008-08-28 17:00:45 -07:00
Zooko O'Whielacronx
6801d5f572 setup: emit the version of python in the list of versions 2008-08-28 15:04:54 -07:00
Brian Warner
3b06aa6a79 servermap: don't log late arrivals, and don't log DeadReferenceError at log.WEIRD 2008-08-26 17:37:29 -07:00
Brian Warner
1668401c16 mutable: make mutable-repair work for non-verifier runs, add tests 2008-08-26 16:34:54 -07:00
Zooko O'Whielacronx
f7b4c45d46 mutable: remove work-around for a flaw in an older version of foolscap 
We now require "foolscap[secure_connections] >= 0.3.0", per [source:_auto_deps.py].
 2008-08-26 08:50:55 -07:00
Brian Warner
a94af879ff logging: add 'unique-message-ids' (or 'umids') to each WEIRD-or-higher log.msg call, to make it easier to correlate log message with source code 2008-08-25 18:57:59 -07:00
Brian Warner
735aa895b9 logging cleanups: lower DeadReferenceError from WEIRD (which provokes Incidents) to merely UNUSUAL, don't pre-format Failures in others 2008-08-25 17:51:55 -07:00
Zooko O'Whielacronx
def9fc8cf0 checker: make the log() function of SimpleCHKFileVerifier compatible with the log() function of its superclasses and subclasses 2008-08-25 14:44:07 -07:00
Brian Warner
eb695f1c58 mutable/checker: log a WEIRD-level event when we see a hash failure, to trigger an Incident 2008-08-12 20:50:20 -07:00
Brian Warner
97852cd626 immutable checker: add a status_report field 2008-08-12 20:35:30 -07:00
Brian Warner
e82d77c3cf mutable/servermap: lower the priority of many log messages 2008-08-12 20:35:06 -07:00
Brian Warner
e5252fd5b2 web/deep-check: show the webapi runtime at the bottom of the page 2008-08-12 20:34:26 -07:00
Brian Warner
7a206421b4 CLI: tolerate blank lines in the aliases file 2008-08-12 19:50:50 -07:00
Brian Warner
bcbb51fe4a test_web: workaround broken HEAD behavior in twisted-2.5.0 and earlier 2008-08-12 19:45:20 -07:00
Brian Warner
6af4619928 test_web: oops, actually use HEAD (instead of GET) in the HEAD test 2008-08-12 19:04:51 -07:00
Brian Warner
4d6b504883 web: use get_size_of_best_version for HEAD requests, provide correct content-type 2008-08-12 19:04:10 -07:00
Brian Warner
d43baa2ad7 mutable: add get_size_of_best_version to the interface, to simplify the web HEAD code, and tests 2008-08-12 19:02:52 -07:00
Brian Warner
014c9b5969 CLI: add 'tahoe debug corrupt-share', and use it for deep-verify tests, and fix non-deep web checker API to pass verify=true into node 2008-08-12 17:05:01 -07:00
Brian Warner
c80e352951 IFilesystemNode: add get_storage_index(), it makes tests easier 2008-08-12 16:14:07 -07:00
Brian Warner
5c0bfd7681 test_system: rename Checker to ImmutableChecker, to make room for a mutable one 2008-08-12 15:59:32 -07:00
Brian Warner
67e7333955 'tahoe debug dump-share': add --offsets, to show section offsets 2008-08-12 14:46:56 -07:00
Brian Warner
022fcf3db8 test_cli: oops, fix tests after recent stdout/stderr cleanup 2008-08-12 14:46:34 -07:00
Brian Warner
acb0297806 scripts/debug: split out dump_immutable_share 2008-08-12 13:55:17 -07:00
Brian Warner
533afd039f scripts/debug: clean up use of stdout/stderr 2008-08-12 13:52:42 -07:00
Brian Warner
1711b5591f CLI: move the 'repl' command to 'tahoe debug repl' 2008-08-12 13:40:17 -07:00
Brian Warner
bb33e3e4c2 CLI: move all debug commands (dump-share, dump-cap, find-shares, catalog-shares) into a 'debug' subcommand, and improve --help output 2008-08-12 13:37:32 -07:00
Brian Warner
3cf70697e8 hush a pyflakes warning 2008-08-11 21:24:23 -07:00
Brian Warner
08e229f29a web/directory: enable verify=true in t=deep-check 2008-08-11 21:24:09 -07:00
Brian Warner
1306b11a76 dirnode: add some deep-check logging 2008-08-11 21:23:38 -07:00
Brian Warner
79576cf75b checker_results.problems: don't str the whole Failure, just extract the reason string 2008-08-11 21:23:06 -07:00
Brian Warner
d106e411af checker: add information to results, add some deep-check tests, fix a bug in which unhealthy files were not counted 2008-08-11 21:03:26 -07:00
Brian Warner
17d7f7d983 mutable/checker: rearrange a bit, change checker-results to have a status_report string 2008-08-11 20:20:33 -07:00
Brian Warner
a15ae52122 mutable/servermap: add summarize_version 2008-08-11 20:19:30 -07:00
Brian Warner
376df2eb5a CLI: make 'tahoe webopen' command accept aliases like 'tahoe ls' 2008-08-11 18:20:23 -07:00
Brian Warner
0d5c8468e9 storage: include disk-free information in the stats-gatherer output 2008-08-06 14:06:02 -07:00
Brian Warner
c7c57bd85c mutable: more repair tests, one with force=True to check out merging 2008-08-06 12:06:07 -07:00
Brian Warner
a68c9364b8 test/common: add ShouldFailMixin 2008-08-06 12:05:52 -07:00
Brian Warner
572115a405 test_mutable: add comment about minimal-bandwidth repairer, comma lack of 2008-08-06 10:38:50 -07:00
Brian Warner
cdbfccd23e test_mutable: factor out common setup code 2008-08-06 10:38:04 -07:00
Brian Warner
dd6ec73efa mutable: start adding Repair tests, fix a simple bug 2008-08-05 23:12:39 -07:00
Brian Warner
3ad0168fbf test_system: factor out find_shares/replace_shares to a common class, so they can be used by other tests 2008-08-05 18:49:58 -07:00
Brian Warner
56943bc5e2 web: add 'report incident' button at the bottom of the welcome page 2008-08-05 12:09:21 -07:00
Brian Warner
45d12baba2 test_cli: more coverage for 'tahoe put' modifying a mutable file in-place, by filename, closes #441 2008-08-04 13:26:43 -07:00
Brian Warner
4fb03c2510 check_grid.py: update to match new CLI: 'put - TARGET' instead of 'put TARGET' 2008-08-01 19:48:56 -07:00
Brian Warner
b9f4006b96 test_cli: remove windows-worrying newlines from test data 2008-08-01 19:47:34 -07:00
Brian Warner
35afe726c1 test_cli.py: factor out CLITestMixin 2008-08-01 19:29:38 -07:00
Brian Warner
f7d2fcc233 CLI: change one-arg forms of 'tahoe put' to make an unlinked file, fix replace-mutable #441 2008-08-01 19:27:29 -07:00
Brian Warner
2eec38279d CLI: add create-alias command, to merge mkdir and add-alias into a single (secure-from-argv-snooping) step 2008-08-01 19:10:41 -07:00
Brian Warner
dc57ab4d36 test_cli: add system-based tests for PUT, including a mutable put that fails/todo (#441) 2008-08-01 15:10:09 -07:00
Brian Warner
071cbec865 tests: simplify CLI tests that use stdin, now that runner supports it 2008-08-01 15:05:14 -07:00
Brian Warner
f181a0458a CLI: simplify argument-passing, use options= for everthing, including stdout 2008-08-01 11:46:24 -07:00
Zooko O'Whielacronx
6d3ec390b0 tests: add test that verifier notices any (randomly chosen) bit flipped in the verifiable part of any (randomly chosen) share 
The currently verifier doesn't (usually) pass this randomized test, hence the TODO.
 2008-07-30 17:20:15 -07:00
Zooko O'Whielacronx
86160c3fc9 tests: test that checker doesn't cause reads on the storage servers 
It would still pass the test if it noticed a corrupted share.  (It won't
notice, of course.)  But it is required to do its work without causing storage
servers to read blocks from the filesystem.
 2008-07-30 16:54:20 -07:00
Zooko O'Whielacronx
29255568df storage: make storage servers declare oldest supported version == 1.0, and storage clients declare oldest supported version == 1.0 
See comments in patch for intended semantics.
 2008-07-30 15:51:07 -07:00
Zooko O'Whielacronx
dbd1c2397e tests: use the handy dandy TestCase.mktemp() function from trial to give unique and nicely named directories for each testcase 2008-07-30 15:49:20 -07:00
Zooko O'Whielacronx
6aabaa0797 tests: don't use SignalMixin 
It seems like we no longer need it, and it screws up something internal in
trial which causes trial's TestCase.mktemp() method to exhibit wrong behavior
(always using a certain test method name instead of using the current test
method name), and I wish to use TestCase.mktemp().

Of course, it is possible that the buildbot is about to tell me that we do
still require SignalMixin on some of our platforms...
 2008-07-30 15:35:36 -07:00
Zooko O'Whielacronx
56282613c5 tests: add test_system.Checker which tests basic checking (without verification) functionality 2008-07-28 16:43:17 -07:00
Zooko O'Whielacronx
9d14341c7d test: add testutil.flip_one_bit which flips a randomly chosen bit of the input string 2008-07-28 16:42:17 -07:00
Zooko O'Whielacronx
fe522e31a4 tests: make it so that you can use common.py's SystemTestMixin.set_up_nodes() more than once with the same introducer 2008-07-28 16:40:29 -07:00
Brian Warner
cd4d3cdeea download.py: set up self._paused before registering the producer, since they might call pauseProducing right away 2008-07-28 14:57:31 -07:00
Brian Warner
9e22353773 test/common.py: use pre-computed Tub certificates for the system-test mixin, to speed such tests up by maybe 15%. The goal is to encourage more full-grid tests. 2008-07-28 12:44:21 -07:00
Brian Warner
914783acdf web: add /status/?t=json, with active upload/download ops. Addresses #493. 2008-07-25 17:41:10 -07:00
Brian Warner
f8221c2c66 web: make t=json stats pages use text/plain, instead of leaving it at text/html 2008-07-25 17:24:27 -07:00
Brian Warner
48c64b7a29 test_system.py: factor SystemTestMixin out of SystemTest 2008-07-25 15:33:49 -07:00
Brian Warner
10bbc740e9 test_system.py: modify system-test setup code in preparation for merge with common.SystemTestMixin 2008-07-25 15:29:31 -07:00
Brian Warner
766cdd9c52 test_system.py: move SystemTestMixin out into common.py, where further improvements will occur 2008-07-25 15:17:58 -07:00
Brian Warner
e317fcc08c test_system.py: create SystemTestMixin, with less cruft, for faster system-like tests 2008-07-25 15:13:00 -07:00
Brian Warner
afda2a43e4 storage: remove update_write_enabler method, it won't serve the desired purpose, and I have a better scheme in mind. See #489 for details 2008-07-21 17:28:28 -07:00
Zooko O'Whielacronx
9461887e0a immutable file download: make the ciphertext hash tree mandatory 
This fixes #491 (URIs do not refer to unique files in Allmydata Tahoe).
Fortunately all of the versions of Tahoe currently in use are already producing
this ciphertext hash tree when uploading, so there is no
backwards-compatibility problem with having the downloader require it to be
present.
 2008-07-21 09:31:02 -07:00
Brian Warner
edec917060 test_web: test that save=true filename=unicode doesn't crash 2008-07-18 18:58:57 -07:00
Brian Warner
572c848d98 web: for GET save=true, don't interpret the filename= arg with any character set, just copy the bytes back into the Content-Disposition header. This seems to make it maximally compatible with Firefox and IE7 2008-07-18 18:06:50 -07:00
Brian Warner
879fefe5f3 first pass at a mutable repairer. not tested at all yet, but of course all existing tests pass 2008-07-17 21:09:23 -07:00
Brian Warner
3e95681bad interfaces: add IRepairable 2008-07-17 17:32:17 -07:00
Brian Warner
923c9c242a oops, fix import/pyflakes problems 2008-07-17 17:06:20 -07:00
Brian Warner
67db0a4967 deep-check: add webapi, add 'DEEP-CHECK' button to wui, add tests, rearrange checker API a bit 2008-07-17 16:47:09 -07:00
Brian Warner
69156aeb28 dirnode deep-check: add tests of cycles, fix failures 2008-07-17 14:37:04 -07:00
Brian Warner
acf3180fac dirnode deep-check: rearrange traversal approach, simplify code a bit 2008-07-17 14:25:04 -07:00
Brian Warner
9289433ba3 first pass at deep-checker, no webapi yet, probably big problems with it, only minimal tests 2008-07-16 18:20:57 -07:00
Brian Warner
7b9fff388d immutable download: remove dead LiteralDownloader, now that we use filenodes for download 2008-07-16 16:31:47 -07:00
Brian Warner
3e9322bcb6 checker: re-enable checker web results (although they just say 'Healthy' right now) 2008-07-16 15:42:56 -07:00
Brian Warner
7394607141 move encode/upload/download/checker.py into a new immutable/ directory. No behavior changes expected. 2008-07-16 13:14:39 -07:00
Brian Warner
94e619c1f6 overhaul checker invocation 
Removed the Checker service, removed checker results storage (both in-memory
and the tiny stub of sqlite-based storage). Added ICheckable, all
check/verify is now done by calling the check() method on filenodes and
dirnodes (immutable files, literal files, mutable files, and directory
instances).

Checker results are returned in a Results instance, with an html() method for
display. Checker results have been temporarily removed from the wui directory
listing until we make some other fixes.

Also fixed client.create_node_from_uri() to create LiteralFileNodes properly,
since they have different checking behavior. Previously we were creating full
FileNodes with LIT uris inside, which were downloadable but not checkable.
 2008-07-15 17:23:25 -07:00
Brian Warner
fd465b4aaf download: fix stopProducing failure ('self._paused_at not defined'), add tests 2008-07-14 15:25:21 -07:00
Brian Warner
c46f5bc634 storage: rename the latency key names so they sort properly 2008-07-11 21:51:02 -07:00
Brian Warner
bffe14ef3e add 1%,10% percentiles to the storage server latency output 2008-07-11 21:34:36 -07:00
Brian Warner
60725ed065 storage: add add_lease/update_write_enabler to remote API, revamp lease handling 2008-07-09 18:06:55 -07:00
Brian Warner
de8ce0cb5b welcome page: add link to statistics page 2008-07-09 17:37:22 -07:00
Brian Warner
a7aa6f8686 implement a mutable checker+verifier. No repair yet. Part of #205. 2008-07-07 17:36:00 -07:00
Brian Warner
a0de01e971 servermap.shares_available: report N too, not just k 2008-07-07 15:06:15 -07:00
Brian Warner
45405d85c4 interfaces: add verify= and repair= args to check() 2008-07-07 14:37:36 -07:00
Brian Warner
9b6f23ac05 'tahoe dump-share': show verify-cap too 2008-07-07 14:11:02 -07:00
Brian Warner
2074c92dd1 test_mutable.py: add tests for no-servers conditions, closes #463. 2008-07-07 12:18:10 -07:00
Brian Warner
bbc63e5d3f web/root.py: fix minor typo 2008-07-07 00:18:16 -07:00
Brian Warner
91c7e0f689 mutable: if there are no servers available, return an error, rather than hanging forever 2008-07-07 00:17:41 -07:00
Brian Warner
7dc8ff0263 re-enable incident-reporting, but disable it in unit tests, because they take 150% longer, and the leftover trailing timers cause errors 2008-07-06 23:49:08 -07:00
Brian Warner
13d0ba72a4 disable incident reporting: unit tests are failing because of the leftover post-incident timer 2008-07-02 18:24:00 -07:00
Brian Warner
d58591ceba start using Foolscap's 'incident-logging' feature, which requires foolscap-0.2.9 2008-07-02 17:40:29 -07:00
Brian Warner
dfe235bcba test_storage: oops, update the test to match the leave-incoming/ change 2008-06-26 11:36:17 -07:00
Brian Warner
02ce46cfb4 catalog-shares: ignore incoming/ 2008-06-26 11:23:44 -07:00
Brian Warner
edabd2f004 storage.py: leave the storage/shares/incoming/ directory in place when the bucket is closed 2008-06-26 11:07:57 -07:00
Brian Warner
af4e4dc847 storage: record latency stats in a flat dict, not nested, to conform to RIStatsProvider 2008-06-24 17:21:18 -07:00
Brian Warner
dba0ad3d28 test_mutable.py: add more tests of post-mapupdate corruption, to support #474 testing 2008-06-24 11:08:10 -07:00
Brian Warner
4e5b9ee63e introducer: move the relevant interfaces out to introducer/interfaces.py 2008-06-18 17:04:41 -07:00
Brian Warner
fa453c9b5e introducer: add old (V1) introducer code, add test framework for compatibility testing 2008-06-18 16:58:34 -07:00
Brian Warner
9ed9eb2048 check_grid.py: make it work, move node start/stop responsibility from Makefile to script 2008-06-19 12:39:52 -07:00
Brian Warner
50d945ea8f start work on 'check-grid' target, for use in an automated prodnet tester. Just a docstring so far. 2008-06-18 12:57:47 -07:00
Brian Warner
28f4652b96 break introducer up into separate modules in the new allmydata.introducer package 2008-06-18 12:24:16 -07:00
Brian Warner
5bdff74e5b web: stop using absolute links (or url.here) in forms and pages, since they break behind proxies. Partially addresses #461 2008-06-17 19:49:40 -07:00
Brian Warner
c09c342718 test_storage: yet more coverage 2008-06-17 17:44:10 -07:00
Brian Warner
6b55b8b022 test_storage.py: improve test coverage 2008-06-17 17:01:42 -07:00
Brian Warner
8e52b3d97d storage.ReadBucketProxy: avoid double-start, this only affected tests 2008-06-17 17:01:29 -07:00
Brian Warner
b62b3974d3 storage: remove unused method 2008-06-17 11:36:32 -07:00
Brian Warner
2154b5751b test_storage: add coverage for discard_storage 2008-06-16 17:52:40 -07:00
Brian Warner
75e662cc46 test_storage: add coverage for readonly_storage 2008-06-16 17:52:13 -07:00
Brian Warner
166e68f2a3 web /statistics : show 0 instead of None for mutable files 2008-06-16 16:37:04 -07:00
Brian Warner
1ce3b77dde storage: improve stats, make them accessible via webport /statistics 2008-06-16 16:35:59 -07:00
Brian Warner
6b7ff02e36 storage: measure latency-per-operation, calculate mean/median/percentiles 2008-06-16 15:21:55 -07:00
robk-tahoe
212c19fc45 webish: re-fix the addSlash issue in directory rendering 
having fixed the relevant unit test not to assert incorrect behaviour of
the node when rendering directories, it's now safe to fix the latter.
 2008-06-11 15:35:27 -07:00
robk-tahoe
b803158de7 test_web: fix webapi test around redirection issues 
this fixes the test_web test test_POST_upload_mutable which chdir's into
a /foo subdirectory, but then later asserts that the web ui redirects the
user back to /foo, which should really be /foo/ since it's a directory.
 2008-06-11 15:19:17 -07:00
Brian Warner
4c681847f8 directory.py: revert that addSlash, it causes a test to fail and we don't understand it yet, and we don't need the fix quite yet 2008-06-11 12:59:28 -07:00
robk-tahoe
021a9e370f webish: fix directory url redirection 
when rendering a directory in webish, if the url did not correctly include
a trailing slash '/' then the browser will be redirected to a url which 
does.  this causes any relative links (e.g. the 'other representations'
which are links to files relative to the current directory) to be correctly
intrpreted as relative to the directory in question by the browser.

uses twisted's addSlash mechanism, which is designed for this purpose :-)
 2008-06-11 12:46:28 -07:00
Zooko O'Whielacronx
65618ba5a2 wapi/wui: add a trailing slash to the targets of hyperlinks of children of the current directory when those targets are directories 
This is intended to fix #458 '"other representations" broken in webish ui'.
 2008-06-10 19:59:38 -07:00
Brian Warner
baf3b6868e storage.py: remove unused import 2008-06-10 13:05:44 -07:00
Brian Warner
bc04b8528a upload: abort the bucket upon any write error, and do it with callRemoteOnly to avoid double errors 2008-06-10 11:55:28 -07:00
Brian Warner
814922a9a1 storage: ignore shares in incoming/, to make clients use other servers during simultaneous uploads 2008-06-10 11:53:10 -07:00
Brian Warner
0aa6d24d62 web: add test for unicode POST when the name comes from name=, not the filename attribute 2008-06-03 17:09:39 -07:00
Brian Warner
9f5075e83d web: make nickname more visible in the welcome page, closes #361 2008-06-03 15:02:10 -07:00
Brian Warner
9f59ecafbb webish: make /cap/ equivalent to /uri/, accepting both with the same meanings. Closes #428 2008-06-03 14:34:00 -07:00
Brian Warner
32c89a8d59 web: transform FileTooLargeError into a friendlier '413 Request Entity Too Large' error 2008-06-03 00:03:16 -07:00
Brian Warner
f4496bd553 mutable/publish.py: raise FileTooLargeError instead of an ugly assertion when the SDMF restrictions are exceeded 2008-06-03 00:02:10 -07:00
Brian Warner
5289064dcf move FileTooLargeError out to a common location 2008-06-03 00:01:15 -07:00
Brian Warner
beb502ac18 CLI: make 'tahoe webopen' use the 'tahoe:' alias properly, instead of the old --dir-cap option 2008-06-02 17:55:54 -07:00
Brian Warner
3e8d58767f CLI: improve docs w.r.t. aliases, add examples to 'tahoe put' and 'tahoe get' help output. Addresses part of #431 2008-06-02 17:54:56 -07:00
Brian Warner
8c37b8e3af Don't allow uploads of large files (about 12GiB or larger), since they're doomed to be corrupted. Closes #439 2008-06-02 16:57:01 -07:00
Brian Warner
b6c2f80170 test_system: even more 'cp -r' coverage 2008-05-21 18:40:49 -07:00
Brian Warner
4824d058cc test_system.py: improve 'cp -r' coverage: exercise copy from tahoe to local disk 2008-05-21 18:36:25 -07:00
Brian Warner
0f8f213baf CLI: remove the '-r' shortcut for --dir-cap, to make it available for cp -r 2008-05-21 17:55:32 -07:00
Brian Warner
457248f023 tahoe_cp: rewrite, make --recursive work 2008-05-21 17:35:21 -07:00
Brian Warner
15fbe05092 tahoe_ls: improve error message when the target is missing 2008-05-21 17:34:52 -07:00
Brian Warner
282e22abcf scripts/common_http.py: add comment about why we need .seek 2008-05-21 17:31:19 -07:00
Brian Warner
7ae432abf8 tahoe_cp.py: fix pyflakes complaint 2008-05-21 12:09:13 -07:00
Brian Warner
7afd869854 cli: initial implementation of 'cp -r', probably doesn't work yet 2008-05-21 11:49:22 -07:00
Brian Warner
fe44b8fb4b util.dictutil: move DictOfSets out to a separate utility module 2008-05-21 09:43:49 -07:00
Brian Warner
0df663b7ae CLI: implement the easy part of cp (no -r, only two arguments) 2008-05-20 16:56:03 -07:00
Brian Warner
3ac4a734e5 web t=json: add 'mutable' key to the information dict 2008-05-20 15:40:49 -07:00
Brian Warner
9c992c0d06 test_web: improve test coverage of web.common utility code 2008-05-20 15:21:46 -07:00
Brian Warner
8dded7819c web: fix JSON output for mutable files 2008-05-20 15:14:19 -07:00
Brian Warner
1d8e452184 CLI: add 'list-aliases', factor out get_aliases 2008-05-20 14:36:04 -07:00
Brian Warner
1236bc3408 CLI: add 'ln', just like move but without the delete 2008-05-20 13:30:31 -07:00
Brian Warner
f09f92c371 test_system.py: remove dead code 2008-05-20 13:20:48 -07:00
Brian Warner
3803d591cf tahoe_ls: improve output formatting 2008-05-20 13:07:50 -07:00
Brian Warner
abe4addbdb CLI mv: if we can't created the new child, don't delete the old one 2008-05-20 12:49:47 -07:00
Brian Warner
a8ed3208ba CLI ls: add --readonly-uri to display readonly URIs for all children 2008-05-20 12:49:11 -07:00
Brian Warner
e889463f0c CLI: add put --mutable, enhance ls to show mutable vs immutable as rw/r- 2008-05-20 12:36:55 -07:00
Brian Warner
e323019708 web: handle PUT mutable=true properly 2008-05-20 12:36:02 -07:00
Brian Warner
d72f75019f test_web: add HEAD coverage 2008-05-20 11:47:43 -07:00
Brian Warner
b0e306a753 web: more mutable-file coverage 2008-05-20 11:35:47 -07:00
Brian Warner
6b51841ec8 web: even more test coverage 2008-05-20 11:33:14 -07:00
Brian Warner
2eadabbf82 web: improve test coverage, remove some dead code 2008-05-20 11:13:12 -07:00
Brian Warner
88457fbc32 test_web/test_system: improve test coverage 2008-05-19 23:28:52 -07:00
Brian Warner
506ce2dfd0 test_system: fix windows test failure, open files with binary mode 2008-05-19 20:24:12 -07:00
Brian Warner
8e92dfcb50 overhaul CLI: not quite complete but it works a lot better than it used to. The new scheme uses 'tahoe add-alias' and rsync/scp-style 'alias:foo/bar.txt' arguments 2008-05-19 19:28:50 -07:00
Brian Warner
9662e6d986 test/common: oops, forgot the FakeMutableFileNode.get_readonly fix 2008-05-19 18:52:19 -07:00
Brian Warner
ef929e51f9 web: improve test coverage 2008-05-19 18:38:39 -07:00
Brian Warner
fe8fdfac64 web/filenode: remove unreachable code 2008-05-19 18:38:28 -07:00
Brian Warner
daebb42060 web/directory: fix rw_uri output in t=JSON to reflect mutable files properly 2008-05-19 18:37:28 -07:00
Brian Warner
d0685d42ff web: factor out identical renderHTTP methods 2008-05-19 15:19:25 -07:00
Brian Warner
242b98f750 webish: censor URLs that start with /file and /named too 2008-05-19 14:34:13 -07:00
Brian Warner
fdb4149bad test_web: remove leftover import to hush pyflakes 2008-05-19 14:28:39 -07:00
Brian Warner
b1d2922ea3 test_system: match webapi change, making new files now returns 201 Created, not 200 OK 2008-05-19 14:09:31 -07:00
Brian Warner
40172047a0 filenode: add is_mutable to non-MutableFileNode classes 2008-05-19 13:03:00 -07:00
Brian Warner
f9cd30d9bc webish: complete rewrite, break into smaller pieces, auto-create directories, improve error handling 2008-05-19 12:57:04 -07:00
Brian Warner
b8c0217ad5 test_web: implement API changes from the recent webapi.txt overhaul 2008-05-19 12:56:02 -07:00
Brian Warner
869e1ed41a test_web.py: minor cleanups, improved error reporting 2008-05-19 12:33:39 -07:00
Brian Warner
3360fd24fc test_web.py: localdir=/localfile= is going away, so remove the tests that exercise it 2008-05-19 12:32:09 -07:00
Brian Warner
b2f8c22fb8 debug dump-share: print size of encprivkey and signature too 2008-05-19 14:24:41 -07:00
Brian Warner
154dca10a0 servermap.py: add size_of_version(), to retrieve the size of a mutable file version 2008-05-19 14:16:02 -07:00
Brian Warner
87c1e8e066 dirnode: add overwrite= to most API calls, defaulting to True. When False, this raises ExistingChildError rather than overwriting an existing child 2008-05-16 16:09:47 -07:00
Brian Warner
1121df6664 testutil.shouldFail: mention the 'which' string in substring failures too 2008-05-16 16:08:38 -07:00
Brian Warner
304abfee32 webish: add /file links, change directory page to use them. This fixes filenames in wget. Closes #221. 2008-05-14 14:32:21 -07:00
Brian Warner
5098297a2b introweb.py: tolerate foolscap>=0.2.6, which changed the internals of .locationHints 2008-05-14 13:55:47 -07:00
Brian Warner
ff0b9e2549 test_mutable.py: test replacing a file that has one new outlier share present: closes #272 2008-05-14 13:10:41 -07:00
Brian Warner
26187bfc81 use a weakref cache in the client to manage singleton filenodes/dirnodes, to avoid autocollision. Should close #391. 2008-05-08 18:02:55 -07:00
Brian Warner
8f4e409677 check_memory: oops, fix HTTP-based modes: trailing slash in node.url vs no slash 2008-05-08 18:01:31 -07:00
Brian Warner
7323ced8e4 test_dirnode: improve coverage of not-mutable-error a bit 2008-05-08 16:53:35 -07:00
Brian Warner
411f578c19 check_memory: don't use fixed webports, they cause occasional test failures 2008-05-08 16:49:27 -07:00
Brian Warner
fabdc28c06 deep-stats: add file-size histogram 2008-05-08 16:19:42 -07:00
Brian Warner
a1ea3d9b37 test_dirnode: the size of the directories varies, tolerate it as low as 500 bytes 2008-05-08 13:42:42 -07:00
Brian Warner
e6ae7a2c60 dirnode: refactor deep-stats a bit 2008-05-08 13:33:07 -07:00
Brian Warner
6c00a70dbc dirnode: add a deep_stats(), like deep-size but with more information. webish adds t=deeps-size too. 2008-05-08 13:21:14 -07:00
Brian Warner
462ef2a0ac run a stats provider even if there's no gatherer, since the HTTP /statistics page is then useful. Only run the once-per-second load-monitor if there is a gatherer configured 2008-05-08 11:37:30 -07:00
Brian Warner
3cb361e233 dirnode: use the concurrency limiter in t=manifest and t=deep-size, allow 10 retrievals in parallel 2008-05-07 18:36:37 -07:00
Brian Warner
d76cf6f748 limiter: fix bug that ignores the limit= argument 2008-05-07 18:35:53 -07:00
Brian Warner
15a9f3d01b test_util: improve ConcurrencyLimiter tests 2008-05-07 17:00:47 -07:00
Brian Warner
c7e441309d add a basic concurrency limiter utility 2008-05-07 16:53:30 -07:00
Brian Warner
1e4b904b1a web status: abbreviate file size in the upload+download summary page 2008-05-07 13:53:58 -07:00
Zooko O'Whielacronx
c1184f51e4 docs: fix a few stale comments in code 2008-05-07 08:39:03 -07:00
Brian Warner
fc774803ce test_stats.py: improve test coverage 2008-04-30 11:52:31 -07:00
Brian Warner
a5a7ba24ef stats: add tests for CPUUsageMonitor, modify it a bit to facilitate testing 2008-04-30 11:39:13 -07:00
Zooko O'Whielacronx
fc271a0ee9 setup: don't catch ImportError when importing _auto_deps in allmydata/__init__.py 
Nowadays pkg_resources is a runtime requirement, and if there is something screwed up in the installation, we want an explicit ImportError exception as early as possible.
 2008-04-30 13:22:04 -07:00
Brian Warner
88951548ee test_mutable: update notify_publish() to match new signature 2008-04-29 18:24:57 -07:00
Brian Warner
4b3adb3fcf mutable stats: track mutable bytes published too 2008-04-29 18:20:05 -07:00
Brian Warner
3be921174b stats: add CPU-percentage monitor, with 1min/5min/15min moving-window averages, using time.clock() 2008-04-29 18:12:53 -07:00
Zooko O'Whielacronx
f842656ec2 wui: reorganize the welcome.xhtml page 
Jake Edge tried Tahoe out and didn't notice the /status page.  Hopefully with this new organization people like he will see that link more easily.  This also addresses drewp's suggestion that the controls appear above the list of servers instead of below.  (I think that was his suggestion.)  I also reordered the controls.
 2008-04-29 15:10:14 -07:00
Brian Warner
351a7f4632 offloaded: add 'resumes' counter to stats: how many upload requests are resuming an interrupted upload? 2008-04-28 13:27:55 -07:00
ben
c85869fc08 support freebsd 6 2008-04-28 00:41:40 -07:00
Brian Warner
186492e620 introducer: only record one announcement per (tubid,service) tuple. Fixes #343. 2008-04-23 15:05:39 -07:00
Brian Warner
55dfb697a4 client: don't start the IntroducerClient until the Tub is ready, otherwise we will sometimes connect to the introducer (or other clients) before we've done Tub.setLocation, which loses some information on the introducer status page 2008-04-23 14:52:34 -07:00
Brian Warner
a557d44877 test_mutable: test that all servers refusing our share means a publish fails 2008-04-22 18:53:20 -07:00
Brian Warner
13eb00145e mutable: improve test coverage in Retrieve, when shares change after mapupdate 2008-04-22 17:25:14 -07:00
Brian Warner
acdd98432d hush pyflakes warning about code that got moved in the recent StallMixin refactoring 2008-04-22 17:14:26 -07:00
Brian Warner
a7632a345e mutable/servermap: improve test coverage 2008-04-22 16:47:52 -07:00
Brian Warner
81ab1ec785 testutil: factor stall() out into a common location 2008-04-22 16:47:15 -07:00
Brian Warner
57d5341715 test_introducer.py: don't log nodeids as binary goop 2008-04-22 12:54:16 -07:00
Brian Warner
9ad6ed3777 web/status: remove trailing whitespace 2008-04-22 11:51:14 -07:00
Brian Warner
e6074f5dfc mutable: test write failures, uncoordinated write detection 2008-04-22 11:49:53 -07:00
Brian Warner
7e1f9761e8 mutable: remove some dead code, rearrange use of populate_pubkey 2008-04-21 17:51:22 -07:00
Brian Warner
db4ddfc1bd mutable: improve testing to exercise the non-cached case 2008-04-21 17:50:50 -07:00
Brian Warner
f39f8ae032 check_speed: rearrange a bit to make it easy to disable certain portions 2008-04-21 17:29:59 -07:00
Brian Warner
8377244c46 mutable status: make the 'total' timing value optional: unit tests fail otherwise 2008-04-21 17:29:12 -07:00
Brian Warner
6af124dc3e mutable read: enable the cache (written during mapupdate, read during retrieve). This speeds up small-file reads by about 30% over a link with an average 25ms RTT 2008-04-21 17:27:50 -07:00
Brian Warner
e8b0f3d3cb mutable status: use google-chart-API to draw server response times for servermap update 2008-04-21 16:16:55 -07:00
Brian Warner
a04c2866b1 mutable: improve test coverage slightly 2008-04-21 15:10:50 -07:00
Brian Warner
f16f48a909 download status: add time spent paused by the client (when we're serving over a slow HTTP link) 2008-04-21 12:19:17 -07:00
Zooko O'Whielacronx
78829dcf08 key_generator: make default key size be a constructor argument instead of a class variable, pass default key size of 522 (the smallest that we can do) in unit tests to make them faster 2008-04-22 12:28:18 -07:00
Brian Warner
c375b1503a mutable/servermap: improve test coverage 2008-04-18 20:13:28 -07:00
Brian Warner
09dcfeae22 mutable: improve test coverage, fix bug in privkey fetching, add .finished to stats, remove dead code 2008-04-18 19:55:12 -07:00
Brian Warner
fc7b75e55f mutable.node: avoid reentrancy problems in Deferred code on twisted2.5, by adding an eventual-send call 2008-04-18 00:43:29 -07:00
Zooko O'Whielacronx
a504f3395f let MutableFileNode.__repr__() return something useful even if it isn't not initialize yet 
This is because I sometimes log or examine a repr of an object from a function which is being called from that object's own __init__()...
(I'm committing this patch in order to test our patch management infrastructure.)
 2008-04-17 21:32:38 -07:00
Brian Warner
d7b82f73c5 dirnode: return to 'delete fails if the child wasn't actually there' semantics, to make tests pass. There's a switch to enable/disable this 2008-04-17 20:06:06 -07:00
Brian Warner
d5b182c183 test_mutable: hush pyflakes 2008-04-17 19:57:55 -07:00
Brian Warner
1bff08a7df dirnode: update to use MutableFileNode.modify 2008-04-17 19:57:04 -07:00
Brian Warner
b6b3b6b1bd testutil: minor reformatting 2008-04-17 19:56:45 -07:00
Brian Warner
945003ea0f test_mutable: factor out ShouldFailMixin 2008-04-17 19:55:51 -07:00
Brian Warner
004d3c14bf mutable: implement MutableFileNode.modify, plus tests 2008-04-17 19:12:42 -07:00
Brian Warner
a379690b04 mutable: replace MutableFileNode API, update tests. Changed all callers to use overwrite(), but that will change soon 2008-04-17 17:51:38 -07:00
Brian Warner
157073d8d8 mutable WIP: merge in patches from current trunk 2008-04-17 13:09:22 -07:00
Brian Warner
ee218c262d mutable WIP: use fireOnOneErrback when using a DeferredList 2008-04-17 13:11:48 -07:00
Brian Warner
e1838ba217 mutable WIP: clean up status handling, shrink the code a lot, improve test coverage 2008-04-17 13:02:22 -07:00
Brian Warner
a1670497a8 mutable WIP: add servermap update status pages 2008-04-16 19:05:41 -07:00
Brian Warner
842b1f1a70 mutable WIP: oops, fix test_mutable 2008-04-16 18:06:54 -07:00
Brian Warner
f35f20e10b mutable WIP: publish status doesn't know its size early enough to update the stats_provider 2008-04-16 17:55:17 -07:00
Brian Warner
749c42fa2c mutable WIP: re-enable publish/retrieve status 2008-04-16 17:49:06 -07:00
Brian Warner
4f07d96667 mutable WIP: improve logging a bit 2008-04-16 15:22:30 -07:00
Brian Warner
c0c8d72b44 mutable WIP: rewrite ServerMap data structure, add tests 2008-04-16 14:49:47 -07:00
Brian Warner
1b4b4cbd4a mutable WIP: rename NotEnoughPeersError to NotEnoughSharesError 2008-04-15 16:08:32 -07:00
Brian Warner
ca14b99397 mutable WIP: if corrupt shares cause a retrieve to fail, restart it once, ignoring those shares and using different ones 2008-04-15 15:58:02 -07:00
Brian Warner
cf84bae850 mutable WIP: merge conflicts in test_system.py 2008-04-15 15:57:28 -07:00
Brian Warner
d4230d1781 mutable WIP: split mutable.py into separate files. All tests pass. 2008-04-11 14:31:16 -07:00
Brian Warner
0515b1099c mutable WIP: all tests pass, but publish/retrieve status is still stubbed out 2008-04-10 19:26:35 -07:00
Brian Warner
418407ee5c mutable: WIP. make Publish work, remove some test scaffolding. test_system still fails. 2008-04-10 18:44:06 -07:00
Brian Warner
b53bbaa43c mutable.py: checkpointing #303 work part 2, Publish is sketched out 2008-04-07 19:01:28 -07:00
Brian Warner
2c939bfdd3 mutable.py: checkpointing #303 work: retrieve does what I want, now starting in on publish 2008-04-04 17:09:26 -07:00
Brian Warner
68527b25fc client: add 'node.uptime' to the stats we collect 2008-04-17 11:13:39 -07:00
Brian Warner
7385ddaf30 download: DownloadStopped isn't SCARY, lower the log severity 2008-04-15 16:06:09 -07:00
Brian Warner
944f8babec test_mutable.py: remove spurious Retrieve during a publish test 2008-04-15 11:20:38 -07:00
Brian Warner
b5d19456f4 web/statistics: fix typo that make immutable-download stats missing 2008-04-15 11:20:04 -07:00
Brian Warner
aebd80feec upload.py: add some notes: extensions are more like 419 than 1000 bytes 2008-04-15 11:14:58 -07:00
Brian Warner
f153dafaa0 web: return a proper error upon POST with a bad t= value 2008-04-15 11:11:29 -07:00
Brian Warner
6de2a0794f web: don't break status page when there is no helper running 2008-04-14 23:42:20 -07:00
Brian Warner
a0fa21e442 helper: log SI in ciphertext-read-failed message 2008-04-14 19:41:23 -07:00
Brian Warner
a719b5060f helper: add SI to logged progress messages 2008-04-14 19:26:53 -07:00
Brian Warner
a2dd4d14a6 helper status: include percentage fetched+pushed, add helper-uploads to the upload/download list 2008-04-14 18:36:27 -07:00
Brian Warner
86578bc3cd webish: fix 'not running helper' status indicator on welcome page, add tests 2008-04-14 16:28:11 -07:00
Brian Warner
5b8320442a stats: add /statistics web page to show them, add tests 2008-04-14 14:17:08 -07:00
Brian Warner
8bea663fb2 offloaded: add fetched-percentage to the log message 2008-04-14 14:16:38 -07:00
Brian Warner
500934b72f helper stats: fix the /helper_status page, the recent conflict merging missed some uses. Added tests, updated the munin plugins to match 2008-04-14 13:18:53 -07:00
Brian Warner
1334a251ca remove size constraint on ShareData: large directories caused errors which triggered massive memory usage. See #379 for details 2008-04-11 22:51:54 -07:00
Brian Warner
d82f90faaa mutable.py: catch errors during publish.. previously they were ignored. oops. 2008-04-11 22:51:02 -07:00
Zooko O'Whielacronx
bdda08df8c copy pyutil.time_format into src/allmydata/util 
time_format() provides a good format for logging timestamps
 2007-08-14 09:33:49 -07:00
Zooko O'Whielacronx
0a6b2b5ff2 doc: slightly clarify an error message 2008-04-14 11:43:05 -07:00
robk-tahoe
a32275771a stats_gatherer: fix typo in helper stats gathering. 2008-04-10 17:41:42 -07:00
robk-tahoe
766deaa9b6 stats_gatherer: reconcile helper stats gathering 
I'd implemented stats gathering hooks in the helper a while back.
Brian did the same without reference to my changes.  This reconciles
those two changes, encompassing all the stats in both changes,
implemented through the stats_provider interface.

this also provide templates for all 10 helper graphs in the 
tahoe-stats munin plugin.
 2008-04-10 17:25:44 -07:00
robk-tahoe
f47ebd0814 conflict reconciliation (part 1, stats gathering in helper) 2008-03-27 17:25:16 -07:00
robk-tahoe
5446ea5d67 stats: added stats reporting to the upload helper 
adds a stats_producer for the upload helper, which provides a series of counters
to the stats gatherer, under the name 'chk_upload_helper'.

it examines both the 'incoming' directory, and the 'encoding' dir, providing
inc_count inc_size inc_size_old enc_count enc_size enc_size_old, respectively
the number of files in each dir, the total size thereof, and the aggregate
size of all files older than 48hrs
 2008-03-25 18:19:08 -07:00
robk-tahoe
8620c7271d key_generator: fix edge case found on windows 
the windows (cygwin) buildslave has been failing the key generator test

it turns out that the time check on whether to refill the pool, and the
reactor, are interacting such that when the maybe_refill_pool call posted
on the reactor fires, the test on whether to fill the pool fails.

this adds a loop in the failure case to retry each 1s until it is time
to refill the pool, thus mitigating this timing accuracy problem on
windows.
 2008-04-10 14:27:35 -07:00
robk-tahoe
35319c3380 stats gathering: fix storage server stats if not tracking consumed 
the RIStatsProvider interface requires that counter and stat values be
ChoiceOf(float, int, long)  the recent changes to storage server to not
track 'consumed' led to returning None as the value of a counter.
this causes violations to be experienced by nodes whose stats are being
gathered.

this patch simply omits that stat if 'consumed' is not being tracked.
 2008-04-09 18:23:06 -07:00
robk-tahoe
ef0f523fe0 stats gathering: added counters to upload,download,mutablewatcher 
counting number of operations, and for immutable files, bytes transferred
 2008-04-09 18:08:59 -07:00
robk-tahoe
0d2eb1edf6 stats_gatherer: verbose debug logging 
one of the storage servers is throwing foolscap violations about the
return value of get_stats().  this adds a log of the data returned
to the foolscap log event stream at the debug level '12' (between
NOISY(10) and OPERATIONAL(20))  hopefully this will facilitate
finding the cause of this problem.
 2008-04-09 16:10:53 -07:00
robk-tahoe
27ac71458f key_generator: remove poll timeouts from test 
the timeouts on uses of 'poll' were there purely to make sure a test doesn't
poll indefinitely.  however having such timeouts makes tests susceptible 
to premature timeouts under high load, or on slow machines. (e.g. cygwin 
slaves running in virtual machines on loaded hosts)

purportedly trial by default applies a timeout to tests to prevent them
hanging out indefinitely, so these poll timeouts are redundant and cause
intermittent failures on slow hosts.  hence they're more bother than they're
worth, and should be culled.
 2008-04-08 11:06:58 -07:00
robk-tahoe
8e9e56fdd0 key_generator: fix a typo in the .tac generating create-key-generator 
verbose is an object attribute, no longer settable via init args
 2008-04-08 11:06:06 -07:00
Zooko O'Whielacronx
8783eabf5a don't do a du on startup if there is no size limit configured 
This also turns off the production of the "space measurement done" log message, if there is no size limit configured.
 2008-04-08 11:36:56 -07:00
robk-tahoe
30eaaa7862 key_generator: fix timing, make tests more robust 
previously there was an edge case in the timing of expected behaviour
of the key_generator (w.r.t. the refresh delay and twisted/foolscap
delivery).  if it took >6s for a key to be generated, then it was 
possible for the pool refresh delay to transpire _during_ the 
synchronous creation of a key in remote_get_rsa_key_pair.  this could
lead to the timer elapsing during key creation and hence the pool
being refilled before control returned to the client.

this change ensures that the time window from a get key request
until the key gen reactor blocks to refill the pool is the time
since a request was answered, not since a request was asked.
this causes the behaviour to match expectations, as embodied in
test_keygen, even if the delay window is dropped to 0.1s
 2008-04-03 18:43:46 -07:00
robk-tahoe
c838cfef44 key_generator: up timeouts on keygen test 
in both these cases, the timeout only serves to abort a stuck test, and
the key_generator should respond more quickly, but seeing test failures
in buildbot on some platforms suggests that the test is too susceptible
to timing issues on loaded buildslaves.
 2008-04-03 16:26:24 -07:00
robk-tahoe
1ae2c39862 key_generator: service related cleanups, incorporation into system test 
this cleans up KeyGenerator to be a service (a subservice of the
KeyGeneratorService as instantiated by the key-generator.tac app)
this means that the timer which replenishes the keypool will be
shutdown cleanly when the service is stopped.

adds checks on the key_generator service and client into the system
test 'test_mutable' such that one of the nodes (clients[3]) uses
the key_generator service, and checks that mutable file creation
in that node, via a variety of means, are all consuming keys from
the key_generator.
 2008-04-03 15:57:07 -07:00
robk-tahoe
ccda06b061 key_generator: added a unit test 
implemented a unit test of basic KeyGenService functionality,
fixed a bug in the timing of pool refreshes
 2008-04-03 13:01:43 -07:00
Brian Warner
06dae8fd67 trial_figleaf.py: make our figleaf code compatible with both Twisted-8.x and Twisted-2.5.x 2008-04-02 17:48:55 -07:00
robk-tahoe
5578559b85 added offloaded key generation 
this adds a new service to pre-generate RSA key pairs.  This allows
the expensive (i.e. slow) key generation to be placed into a process
outside the node, so that the node's reactor will not block when it
needs a key pair, but instead can retrieve them from a pool of already
generated key pairs in the key-generator service.

it adds a tahoe create-key-generator command which initialises an 
empty dir with a tahoe-key-generator.tac file which can then be run
via twistd.  it stashes its .pem and portnum for furl stability and
writes the furl of the key gen service to key_generator.furl, also
printing it to stdout.

by placing a key_generator.furl file into the nodes config directory
(e.g. ~/.tahoe) a node will attempt to connect to such a service, and
will use that when creating mutable files (i.e. directories) whenever
possible.  if the keygen service is unavailable, it will perform the
key generation locally instead, as before.
 2008-04-01 18:45:13 -07:00
Brian Warner
f9e261d939 introducer.py: accelerate reconnection after being offline. Closes #374. 
When we establish any new connection, reset the delays on all the other
Reconnectors. This will trigger a new batch of connection attempts. The idea
is to detect when we (the client) have been offline for a while, and to
connect to all servers when we get back online. By accelerating the timers
inside the Reconnectors, we try to avoid spending a long time in a
partially-connected state (which increases the chances of causing problems
with mutable files, by not updating all the shares that we ought to).
 2008-03-31 15:28:45 -07:00
Brian Warner
d6be5116f5 storage: emit log messages on bucket allocate/read and mutable writev 2008-03-27 17:33:58 -07:00
Brian Warner
2c96a32633 helper: add more stats to webapi, at /helper_status 2008-03-27 16:46:08 -07:00
Brian Warner
36f5c025a6 helper: add stats for the gatherer, show some on the webish welcome page 2008-03-27 15:55:32 -07:00
Brian Warner
9b3a32d0b3 add GET /uri/URI/?t=deep-size, to compute the total size of immutable files reachable from a given directory 2008-03-27 11:33:42 -07:00
Brian Warner
3f5fed0bd4 node.py: make twistd.pid world-readable, so stats-gathering tools (specifically memory-measuring munin plugins) can see it 2008-03-26 18:37:54 -07:00
Brian Warner
04b690ac9e web-status: client methods like list_all_uploads() return Upload instances, 
not status instances. Fix this. The symptom was that following a link like
'up-123' that referred to an old operation (no longer in memory) while an
upload was active would get an ugly traceback instead of a "no such resource"
message.
 2008-03-26 18:20:07 -07:00
Brian Warner
4531d1e953 introweb.py: add ?t=json, to provide machine-readable subscriber counts 2008-03-25 12:56:12 -07:00
Brian Warner
4b46f1cd53 encode.py: also record the size, along with plaintext_hash and SI 2008-03-24 19:08:15 -07:00
Brian Warner
15ef9f624d encode: log a plaintext hash and SI for each upload. This will allow the log gatherer to correlate the two, to better measure the benefits of convergence 2008-03-24 18:55:37 -07:00
robk-tahoe
7f14b93341 confwiz: set a convergence domain based on root_dir upon config 
when the confwiz configures a node (i.e. typically once on mac, once per
install on windows) in addition to writing the root_dir.cap retrieved from
the native_client backend into a config file, it additionally writes a hash
thereof into the 'convergence' config file.

this causes uploads from this node to use a consistent 'convergence' hashing
value matching any other nodes with the same configured root_dir, i.e. for
the most part other systems installed and configured on the same account.
 2008-03-24 15:47:12 -07:00
robk-tahoe
d25d3065d1 mutable: revise a couple of error messages 
at brian and zooko's suggestion, reword an error message encountered when
multiple writers are racing to make overlapping changes to a directory
 2008-03-24 15:46:28 -07:00