30 Commits

Author SHA1 Message Date
Kevan Carstensen
f80a7fdf18 mutable: train checker and repairer to work with MDMF mutable files 2011-08-01 18:51:40 -07:00
david-sarah
e76092e16c Change relative imports to absolute 2010-02-26 01:14:33 -07:00
Brian Warner
ba0690c9d7 mutable repair: return successful=False when numshares<k (thus repair fails),
instead of weird errors. Closes #874 and #786.

Previously, if the file had 0 shares, this would raise TypeError as it tried
to call download_version(None). If the file had some shares but fewer than
'k', it would incorrectly raise MustForceRepairError.

Added get_successful() to the IRepairResults API, to give repair() a place to
report non-code-bug problems like this.
2009-12-29 15:37:46 -08:00
Brian Warner
0d5dc51617 Overhaul IFilesystemNode handling, to simplify tests and use POLA internally.
* stop using IURI as an adapter
* pass cap strings around instead of URI instances
* move filenode/dirnode creation duties from Client to new NodeMaker class
* move other Client duties to KeyGenerator, SecretHolder, History classes
* stop passing Client reference to dirnode/filenode constructors
  - pass less-powerful references instead, like StorageBroker or Uploader
* always create DirectoryNodes by wrapping a filenode (mutable for now)
* remove some specialized mock classes from unit tests

Detailed list of changes (done one at a time, then merged together)

always pass a string to create_node_from_uri(), not an IURI instance
always pass a string to IFilesystemNode constructors, not an IURI instance
stop using IURI() as an adapter, switch on cap prefix in create_node_from_uri()
client.py: move SecretHolder code out to a separate class
test_web.py: hush pyflakes
client.py: move NodeMaker functionality out into a separate object
LiteralFileNode: stop storing a Client reference
immutable Checker: remove Client reference, it only needs a SecretHolder
immutable Upload: remove Client reference, leave SecretHolder and StorageBroker
immutable Repairer: replace Client reference with StorageBroker and SecretHolder
immutable FileNode: remove Client reference
mutable.Publish: stop passing Client
mutable.ServermapUpdater: get StorageBroker in constructor, not by peeking into Client reference
MutableChecker: reference StorageBroker and History directly, not through Client
mutable.FileNode: removed unused indirection to checker classes
mutable.FileNode: remove Client reference
client.py: move RSA key generation into a separate class, so it can be passed to the nodemaker
move create_mutable_file() into NodeMaker
test_dirnode.py: stop using FakeClient mockups, use NoNetworkGrid instead. This simplifies the code, but takes longer to run (17s instead of 6s). This should come down later when other cleanups make it possible to use simpler (non-RSA) fake mutable files for dirnode tests.
test_mutable.py: clean up basedir names
client.py: move create_empty_dirnode() into NodeMaker
dirnode.py: get rid of DirectoryNode.create
remove DirectoryNode.init_from_uri, refactor NodeMaker for customization, simplify test_web's mock Client to match
stop passing Client to DirectoryNode, make DirectoryNode.create_with_mutablefile the normal DirectoryNode constructor, start removing client from NodeMaker
remove Client from NodeMaker
move helper status into History, pass History to web.Status instead of Client
test_mutable.py: fix minor typo
2009-08-15 04:28:46 -07:00
Brian Warner
e330abc3c3 mutable repairer: skip repair of readcaps instead of throwing an exception.
This should improve the behavior of #625 a bit: at least all the files will
get repaired.
2009-06-30 18:13:43 -07:00
Brian Warner
bce4a5385b add --add-lease to 'tahoe check', 'tahoe deep-check', and webapi. 2009-02-17 19:32:43 -07:00
Brian Warner
2fe099a0b3 mutable: move recent operation history management code (MutableWatcher) into history.py, have History provide stats 2009-01-14 17:36:20 -07:00
Zooko O'Whielacronx
ef60e85ec6 naming: finish renaming "CheckerResults" to "CheckResults" 2009-01-09 18:00:52 -07:00
Zooko O'Whielacronx
5e6f90a015 rename "checker results" to "check results", because it is more parallel to "check-and-repair results" 2009-01-06 13:37:03 -07:00
Zooko O'Whielacronx
471e1f1b9b try to tidy up uri-as-string vs. uri-as-object
I get confused about whether a given argument or return value is a uri-as-string or uri-as-object.  This patch adds a lot of assertions that it is one or the other, and also changes CheckerResults to take objects not strings.
In the future, I hope that we generally use Python objects except when importing into or exporting from the Python interpreter e.g. over the wire, the UI, or a stored file.
2008-12-19 08:39:24 -07:00
Brian Warner
dfa2408157 checker: add is_recoverable() to checker results, make our stub immutable-verifier not throw an exception on unrecoverable files, add tests 2008-11-06 22:35:47 -07:00
Brian Warner
b1db6d9ff2 web: add 'Repair' button to checker results when they indicate unhealthyness. Also add the object's uri to the CheckerResults instance. 2008-10-29 18:09:17 -07:00
Brian Warner
9f21f7cf65 mutable: call remove_advise_corrupt_share when we see share corruption in mapupdate/download/check, tolerate servers that do not implement it 2008-10-24 13:21:28 -07:00
Brian Warner
54c0ffd214 mutable/checker: announce the mapupdate op on the 'recent uploads+downloads' page 2008-10-23 16:03:19 -07:00
Brian Warner
977c6ac510 more #514: pass a Monitor to all checker operations, make mutable-checker honor the cancel flag 2008-10-22 01:38:18 -07:00
Brian Warner
1d2d6a35a6 checker results: add output=JSON to webapi, add tests, clean up APIs
to make the internal ones use binary strings (nodeid, storage index) and
the web/JSON ones use base32-encoded strings. The immutable verifier is
still incomplete (it returns imaginary healty results).
2008-09-09 19:45:17 -07:00
Brian Warner
7fb3308498 mutable checker: even more tests. Everything in ICheckerResults should be covered now, except for immutable-verify which is incomplete 2008-09-09 17:57:06 -07:00
Brian Warner
9676586732 mutable checker: oops, fix redefinition of 'healthy' (numshares < N, not numshares < k, which is 'recoverable' not 'healthy') 2008-09-09 17:28:53 -07:00
Brian Warner
f895e39d48 checker results: more tests, more results. immutable verifier tests are disabled until they emit more complete results 2008-09-09 17:15:46 -07:00
Brian Warner
3408d552cd checker: overhaul checker results, split check/check_and_repair into separate methods, improve web displays 2008-09-07 12:44:56 -07:00
Brian Warner
1668401c16 mutable: make mutable-repair work for non-verifier runs, add tests 2008-08-26 16:34:54 -07:00
Brian Warner
a94af879ff logging: add 'unique-message-ids' (or 'umids') to each WEIRD-or-higher log.msg call, to make it easier to correlate log message with source code 2008-08-25 18:57:59 -07:00
Brian Warner
eb695f1c58 mutable/checker: log a WEIRD-level event when we see a hash failure, to trigger an Incident 2008-08-12 20:50:20 -07:00
Brian Warner
79576cf75b checker_results.problems: don't str the whole Failure, just extract the reason string 2008-08-11 21:23:06 -07:00
Brian Warner
d106e411af checker: add information to results, add some deep-check tests, fix a bug in which unhealthy files were not counted 2008-08-11 21:03:26 -07:00
Brian Warner
17d7f7d983 mutable/checker: rearrange a bit, change checker-results to have a status_report string 2008-08-11 20:20:33 -07:00
Brian Warner
879fefe5f3 first pass at a mutable repairer. not tested at all yet, but of course all existing tests pass 2008-07-17 21:09:23 -07:00
Brian Warner
3e9322bcb6 checker: re-enable checker web results (although they just say 'Healthy' right now) 2008-07-16 15:42:56 -07:00
Brian Warner
94e619c1f6 overhaul checker invocation
Removed the Checker service, removed checker results storage (both in-memory
and the tiny stub of sqlite-based storage). Added ICheckable, all
check/verify is now done by calling the check() method on filenodes and
dirnodes (immutable files, literal files, mutable files, and directory
instances).

Checker results are returned in a Results instance, with an html() method for
display. Checker results have been temporarily removed from the wui directory
listing until we make some other fixes.

Also fixed client.create_node_from_uri() to create LiteralFileNodes properly,
since they have different checking behavior. Previously we were creating full
FileNodes with LIT uris inside, which were downloadable but not checkable.
2008-07-15 17:23:25 -07:00
Brian Warner
a7aa6f8686 implement a mutable checker+verifier. No repair yet. Part of #205. 2008-07-07 17:36:00 -07:00