ExternalVendorCode/tahoe-lafs
1
0
Fork 0
mirror of https://github.com/tahoe-lafs/tahoe-lafs.git synced 2024-12-19 21:17:54 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity
6,089 Commits 185 Branches 90 Tags 89 MiB
c2874a7073
Commit Graph

3541 Commits

Author SHA1 Message Date
david-sarah
adca74d9f8 Tests for ref #1669. Also refactor the existing tests to reduce duplicated code and to fix a cut-and-paste error that caused one case (successful SDMF repair) to go untested. 2012-06-21 20:46:27 +00:00
david-sarah
196bd583b6 Add assertions to make sure that set_default_encoding_parameters is always called, rather than using hardcoded 3/7/10 defaults. Also update affected tests. Note that this by itself cannot fix the bug mentioned in ticket:1212#comment:41, but it might make it easier to reproduce. refs #1212 2012-06-19 02:05:04 +00:00
david-sarah
323774698e Make the intent of the loop over servers in test_system.Connections.test_rref clearer, and able to be the same in 1.9.2 and trunk. Remove the now-unused node_key_s attribute of Client. refs #1636 2012-06-16 18:18:44 +00:00
david-sarah
635e87bd7b Fix a bug in mutable publish that could cause an IndexError when a writer is removed in Publish._connection_problem. This version uses DictOfSets as suggested by warner. fixes #1749 2012-06-15 03:44:37 +00:00
david-sarah
3ab0f33413 Skip allmydata.test.test_system.Connections.test_rref unless we have foolscap >= 0.6.4, because of http://foolscap.lothar.com/trac/ticket/196 . refs #1636 2012-06-15 23:20:28 +00:00
david-sarah
1b07d30761 After a server disconnects, make the IServer retain the dead RemoteReference, and continue to return it to anyone who calls get_rref(). This removes the need for callers to guard against receiving a None (as long as the server was connected at least once, which is always the case for servers returned by get_servers_for_psi(), which is how all upload/download code gets servers). Includes test. fixes #1636 2012-06-15 01:48:55 +00:00
david-sarah
2c4ea1df5f Since DeprecationWarning about twisted.internet.interfaces.IFinishableConsumer is suppressed globally, it doesn't need to be suppressed during import. refs #1295 2012-06-14 21:33:15 +00:00
david-sarah
854afc095f Suppress DeprecationWarning about twisted.internet.interfaces.IFinishableConsumer. This also unifies the handling of DeprecationWarnings that need to be suppressed globally. refs #1295 2012-06-14 21:23:08 +00:00
Brian Warner
b2dcbbb62d test/common.py: fix race condition waiting for the helper connection 
The wait_for_connections() method, which is used at the start of
test_system to make sure that all the clients are connected to all the
servers, did not also wait for clients to be connected to their Helpers.
Every once in a while, the helper connection would take a bit longer,
and then
test_system.SystemTest.test_filesystem._test_web._got_welcome_helper
would fail, because we'd check for a helper connection before it was
ready.

The fix is to modify wait_for_connections's polling predicate to look
for helper connections (if configured) as well as the regular
introducer- and server- connections.

Tested by temporarily adding a large (30s) delay to the connectTo() call
in Uploader.startService, simulating a long helper
connection-establishment delay. This makes the test fail consistently.
Then I fixed wait_for_connections(), and the test passed (slowly). Then
I removed the delay.

Closes #1467
 2012-06-14 12:18:35 -07:00
david-sarah
daa24bce8b Clarify documentation of RIStorageServer.slot_testv_and_readv_and_writev. fixes #1744 2012-06-13 16:51:35 +00:00
Brian Warner
ef2db56104 introweb: the Subscribed Clients list shows tubids, not serverids 
Improve the column headers to make it clear that this list shows Tub
IDs. (we can't show pubkey-based serverids because clients don't give
those to us: only servers provide pubkeys). This should be the only
place in the whole webapi that shows TubIDs for modern (V2-introducer)
nodes.
 2012-06-12 14:37:27 -07:00
Brian Warner
c073565ecc Display serverids consistently as 8-char pubkey, or 6-char tubid. 
This makes it easy to distinguish between old V1-Introducer
nodes (identified by their Foolscap TubID) and new V2 nodes (identified
by their ed25519 pubkey).

This fixes a few places where we used to display a tubid even if we had
a pubkey, making it hard to visually correlate servers in two different
displays. It also cleans up the way we pass serverids to the JS-based
download timeline.

The "introweb" subscribed-clients list still shows tubids.
 2012-06-12 14:30:34 -07:00
Brian Warner
51a5aaa639 test_system.py: wait for the Helper connection properly before uploading 2012-06-11 23:19:30 -07:00
Brian Warner
5e432b12a1 test_system.py: clean up control flow, reduce use of stall() 
The _upload_resumable() test interrupts a Helper upload partway
through (by shutting down the Helper), then restarts the Helper and
resumes the upload. The control flow is kind of tricky: to do anything
"partway through" requires adding a hook to the Uploadable. The previous
flow depended upon a (fragile) call to self.stall(), which waits a fixed
number of seconds.

This removes one of those stall() calls (the remainder is in
test/common.py and I'll try removing it in a subsequent revision). It
also removes some now-redundant wait_for_connections() calls, since
bounce_client() doesn't fire its Deferred until the client has finished
coming back up (and uses wait_for_connections() internally to do so).
 2012-06-11 18:22:35 -07:00
Brian Warner
82519ef93f test_system.py: fix minor typo 2012-06-11 18:16:36 -07:00
Brian Warner
a809e4caba offloaded.py: don't drop the Deferred 
There was one corner case (where the client disconnects at just the
wrong time) that could have dropped a Deferred, leading to an Unhandled
Error. Clean up the control flow to avoid this case.
 2012-06-11 18:16:02 -07:00
Brian Warner
e1093cbb33 introducer: add sequence-numbers to announcements, ignore replays 
This will support revocation of Accounting recommendation records,
assuming the gossip-based broadcast channel isn't easily jammed.
 2012-06-10 19:10:22 -07:00
Brian Warner
bf416af49e client.py: rename "server key" to "node key", use old name if present 
This prepares for invitation-based reciprocal-permission Accounting. In
the scheme I'm developing, nodes publish "I accept shares from Y"
messages, which are assembled into a graph, and server will accept
shares from any client node reachable in this graph. For this to work,
the serverX->clientY edge must be connectable to the serverY->clientZ
edge, which means "clientY" and "serverY" must be connected. If clientY
and serverY are two distinct keys, they must be cross-signed. Life is
easier if there's just one key "Y", rather than distinct client- and
server- keys. Calling this one key "server.privkey" would be confusing.
"node.privkey" and "node.pubkey" makes more sense.

One-server-per-node is a pretty easy restriction. Originally I was
thinking that the client.key should be provided in each webapi call,
just like a filecap is, making a single node useable by multiple users
(Accounting principals), and not providing any ambient storage
authority. But I've been unable to think of a comfortable WUI for
that (at least without requiring javascript), nor a friendly way to
transfer account authority (e.g. writecaps that include storage
authority). So I'm more willing to have one-client-per-node these days.

(and note that this rename doesn't seriously preclude
many-clients-per-node or zero-clients-per-node anyways, it just makes
one-client-per-node less awkward)
 2012-06-10 18:14:55 -07:00
Brian Warner
26d3869076 node.py: add get_private_config() 
Also add tests for this and the pre-existing private-config methods.
 2012-06-10 17:46:38 -07:00
Brian Warner
518e4cec98 Fix text in Publish Status results. Closes #1762. 2012-06-08 15:21:46 -07:00
Brian Warner
188c7fecf5 CheckResults corrupt/incompatible shares now return IServers 
DeepResultsBase also has a get_corrupt_shares(), and it is populated
from CheckResults.get_corrupt_shares(). It has been updated too, along
with get_remaining_corrupt_shares().

Remove temporary get_new_corrupt_shares() and
get_new_incompatible_shares().
 2012-06-02 11:39:12 -07:00
Brian Warner
da9ac55294 CheckResults.get_servers_responding() now returns IServers 
Remove temporary get_new_servers_responding().
 2012-06-02 11:39:12 -07:00
Brian Warner
957a5315aa CheckResults.get_sharemap() now returns IServers 
Remove temporary get_new_sharemap().
 2012-06-02 11:39:11 -07:00
Brian Warner
76fca000df CheckResults: pass IServer to corrupt/incompatible share locators 
Getters still return serverid. Adds temporary get_new_corrupt_shares()
and get_new_incompatible_shares().
 2012-06-02 11:39:11 -07:00
Brian Warner
dd8178ee6d CheckResults: pass IServer to servers_responding=, getter returns serverid 
Add temporary get_new_servers_responding().
 2012-06-02 11:39:11 -07:00
Brian Warner
a4c95609c7 CheckResults: pass IServer to sharemap=, but get_sharemap() returns serverids 
This changes all code which feeds CheckResults(sharemap=) to provide
IServer instances, but CheckResults converts these to old-style
serverids during output, so downstream code doesn't have to change yet.

It adds a temporary get_new_sharemap(), which *does* return IServer
instances, so the immutable repairer can build new CheckResults from an
old one. This will go away when get_sharemap() is updated to return
IServer (and downstream code is updated too).
 2012-06-02 11:39:11 -07:00
Brian Warner
c03b6aff97 CheckResults: internal cleanup 
replace the one-big-dictionary with normal private attributes
 2012-06-02 11:39:11 -07:00
Brian Warner
437de4340b CheckResults: privatize remaining attributes 2012-06-02 11:39:10 -07:00
Brian Warner
0fcc054a61 CheckResults: use fat init, add type-checking assertions 
Added assertions for sharemap, servermap, servers_responding,
list_corrupt_shares, and list_incompatible_shares.
 2012-06-02 11:39:10 -07:00
Brian Warner
fba26d7bae mutable/checker: refactor to make CheckResults easier to change 2012-06-02 11:39:10 -07:00
Brian Warner
8daacbcf69 CheckResults: replace get_data() with as_dict(), use getters in web status 2012-06-02 11:39:10 -07:00
Brian Warner
4867dca3f0 use the new CheckResult getters almost everywhere 
The remaining get_data() calls are either in
web.check_results.json_check_results(), or functioning as repr()s in
various unit test failure cases.
 2012-06-02 11:39:10 -07:00
Brian Warner
1883d393c6 CheckResults: replace get_data() with a bunch of individual getters 2012-06-02 11:39:10 -07:00
Brian Warner
ccfcd4de37 change CheckResults to use a fat set_data() 
i.e. change set_data() to accept lots of parameters, instead of taking
a single dictionary with lots of keys. Also Convert all CheckResults
creators to use it.
 2012-06-02 11:39:10 -07:00
Brian Warner
d446897282 CheckResults: simplify self._data 2012-06-02 11:39:09 -07:00
Brian Warner
e313cf6406 CheckResults: start hiding .data, first step to clean it up 
The goal is to make CheckResults more strongly typed, and remove the
ambiguous ".data" field in favor of a bunch of specific counters and
sharelists, so I can changes .sharemap and .servermap to use IServer
instances instead of string serverids. By cleaning this up first, I hope
to get that task done with less debugging.
 2012-06-02 11:39:09 -07:00
Brian Warner
17c5384f79 immutable.CiphertextFileNode.check_and_repair: simplify for refactoring 
There were too many nested functions here, making some upcoming changes
too difficult, so let's refactor it first.
 2012-06-02 11:39:09 -07:00
david-sarah
2ee1bc7148 Catch exceptions from CLI in order to prevent the Ubuntu crash monolog from triggering. refs #1746 2012-05-20 15:35:29 +00:00
Brian Warner
3ba77925d9 node.py: stop stripping whitespace in write_private_config() 
It's nice to add newlines to the saved file, so 'cat' is easy to use. We
still strip on the input side, in get_or_create_private_config().
 2012-05-30 00:17:55 -07:00
Brian Warner
bfee999e20 test_web.py: fix memory leak when run with --until-failure 
The Fake*Node classes in test/common.py were accumulating share data in
a class-level dictionary, which persisted from one test run to the next.
As a result, running test_web.py over and over (with trial's
--until-failure feature) made this dictionary grow without bound,
eventually running out of memory.

This fix moves that dictionary into the FakeClient built fresh for each
test, so it doesn't build up. It does the same thing for "file_types",
which was much smaller but still lived at the class level.

Closes #1729
 2012-05-22 15:39:49 -07:00
Brian Warner
bcdfb5802e test/check_memory.py: oops, fix one last ur.uri -> ur.get_uri() 2012-05-22 08:50:36 -07:00
Brian Warner
3a1c02cfdf change UploadResults to return IServers, update users to match 
This finally changes all callers of get_servermap()/get_sharemap() to
accept IServers, and changes UploadResults to provide them.
 2012-05-21 21:18:37 -07:00
Brian Warner
843739486a UploadResults: store IServers internally, but still return serverids 
This stores IDisplayableServer-providing instances (StubServers or
NativeStorageServers) in the .servermap and .sharemap dictionaries. But
get_servermap()/get_sharemap() still return data structures with
serverids, not IServers, by translating their data on the way out. This
lets us put off changing the callers for a little bit longer.
 2012-05-21 21:18:25 -07:00
Brian Warner
97a1eb6ebf split IDisplayableServer from IServer, add sb.get_stub_server() 
IDisplayableServer includes just enough functionality to call
.get_name() and friends, which is all that the UploadResults really
need. IServer is a superset that includes actual share-manipulation
methods. StubServer instances provide only IDisplayableServer, while
actual NativeStorageServer instances provide the full IServer interface.

When the Helper sends a serverid (so we know what to call the server but
nothing else about it, and have no corresponding NativeStorageServer
object to reference), but we want to store an IDisplayableServer in the
UploadResults, we create a synthetic StubServer "server" and store that
instead.
 2012-05-21 21:17:27 -07:00
Brian Warner
3d771132a8 switch UploadResults to use get_uri(), hide internal ._uri 
Complete the getter-based transformation, by hiding ".uri" and updating
callers to use get_uri(). Also don't set a dummy self._uri, leave it
undefined until someone calls set_uri().
 2012-05-21 21:14:44 -07:00
Brian Warner
29b11531b5 switch UploadResults to use getters, hide internal data, for all but .uri 
This hides attributes with e.g. _sharemap, and creates getters like
get_sharemap() to access them, for every field except .uri . This will
make it easier to modify the internal representation of .sharemap
without requiring callers to adjust quite yet.

".uri" has so many users that it seemed better to update it in a
subsequent patch.
 2012-05-21 21:14:28 -07:00
Brian Warner
08f5bc8e2f convert UploadResults to a fat init 
Populate most of UploadResults (except .uri, which is learned later when
using a Helper) in the constructor, instead of allowing creators to
write to attributes later. This will help isolate the fields that we
want to change to use IServers.
 2012-05-21 21:14:14 -07:00
Brian Warner
b71234c538 add HelperUploadResults 
This splits the pb.Copyable on-wire object (HelperUploadResults) out
from the local results object (UploadResults). To maintain compatibility
with older Helpers, we have to leave pb.Copyable classes alone and
unmodified, but we want to change UploadResults to use IServers instead
of serverids. So by using a different class on the wire, and translating
to/from it on either end, we can accomplish both.
 2012-05-21 21:14:00 -07:00
Brian Warner
b3af012b13 Uploader cleanup: create results at end, not beginning 
This will make it easier to populate the UploadResults during __init__,
instead of doing it one-field-at-a-time later.
 2012-05-21 21:13:47 -07:00
Brian Warner
0df833eac9 clean up Helper to make later changes easier 
Fix up control flow inside the Helper, to make it more friendly for
later refactoring.
 2012-05-21 21:13:32 -07:00
Brian Warner
e60982c851 helper: remove timings["existence_check"], aka "Already-In-Grid Check" 
This measured how long the Helper took to do a filecheck before asking
for ciphertext. The "Contacting Helper" report includes both
existence_check and the client-helper RTT.

For non-overlapping uploads, it was being returned correctly. But when
multiple upload requests overlapped, and the file was not already in the
grid, the filecheck would only run once, and its existence_check time
would be reported for all uploaders (even if they didn't have to wait
for that time). Cleaning that up proved too difficult: the only correct
place to report this time is from the initial remote_upload_chk() call,
but the return value of that is too constrained to accomodate it in the
needs-upload case.

So I'm removing it altogether. Eventually I plan to add a proper
events/times field and record more data, including this check, in a form
that can be drawn on a nice zoomable timeline view.

Old clients talking to a new Helper (which doesn't supply the value)
will tolerate the loss (they'll just display an empty field on the web
view).
 2012-05-21 21:13:11 -07:00
Brian Warner
393c0729de test_checker: minor improvement in fake-server setup 
This prepares for testing the differences between tubid and pubkey-based
name/longname.
 2012-05-21 19:49:36 -07:00
david-sarah
4ddcde3094 Since we now require Python 2.5, we can use os.SEEK_END. 2012-05-16 21:39:48 +00:00
david-sarah
a1a1b5bf8a Simplifications resulting from requiring Python 2.5 and therefore being able to use sqlite3 from the standard library. This also drops sqlite3 from the set of versions and paths we report. 2012-05-16 02:47:25 +00:00
david-sarah
0fc196ea5f Require Python 2.5. 2012-05-16 02:41:49 +00:00
Brian Warner
cc366903ce dictutil.DictOfSets: remove .union() method, it was misleading 
Unlike set.union(), which returns a new set, DictOfSets.union() modified
the DictOfSets in-place. The name collision bit me when I changed some
code from using DictOfSets to a normal set, and expected that
set.union() would modify the set in-place. Since there was only one user
of DictOfSets.union, I figured it was safer to just get rid of it.
 2012-05-16 16:55:09 -07:00
Brian Warner
9acf5beebd immutable repairer: populate servers-responding properly 
If a server did not respond to the pre-repair filecheck, but did respond
to the repair, that server was not correctly added to the
RepairResults.data["servers-responding"] list. (This resulted from a
buggy usage of DictOfSets.union() in filenode.py).

In addition, servers to which filecheck queries were sent, but did not
respond, were incorrectly added to the servers-responding list
anyawys. (This resulted from code in the checker.py not paying attention
to the 'responded' flag).

The first bug was neatly masked by the second: it's pretty rare to have
a server suddenly start responding in the one-second window between a
filecheck and a subsequent repair, and if the server was around for the
filecheck, you'd never notice the problem. I only spotted the smelly
code while I was changing it for IServer cleanup purposes.

I added coverage to test_repairer.py for this. Trying to get that test
to fail before fixing the first bug is what led me to discover the
second bug. I also had to update test_corrupt_file_verno, since it was
incorrectly asserting that 10 servers responded, when in fact one of
them throws an error (but the second bug was causing it to be reported
anyways).
 2012-05-16 16:55:09 -07:00
david-sarah
3738c3e2d1 fileutil.py: use try/finally to close file in write_atomically. 2012-05-16 23:08:39 +00:00
Brian Warner
c4d7b7b109 write node.url and portnum files atomically, to fix race in test_runner 
Previously, test_runner sometimes fails because the _node_has_started()
poller fires after the portnum file has been opened, but before it has
actually been filled, allowing the test process to observe an empty file,
which flunks the test.

This adds a new fileutil.write_atomically() function (using the usual
write-to-.tmp-then-rename approach), and uses it for both node.url and
client.port . These files are written a bit before the node is really up and
running, but they're late enough for test_runner's purposes, which is to know
when it's safe to read client.port and use 'tahoe restart' (and therefore
SIGINT) to restart the node.

The current node/client code doesn't offer any better "are you really done
with startup" indicator.. the ideal approach would be to either watch the
logfile, or connect to its flogport, but both are a hassle. Changing the node
to write out a new "all done" file would be intrusive for regular
operations.
 2012-05-14 15:03:14 -07:00
david-sarah
bf7e351464 Improve a comment in __init__.py. 2012-05-14 16:34:31 +00:00
david-sarah
4b80299fdd Suppress the PowmInsecureWarning from PyCrypto. refs #1586 2012-05-14 03:23:52 +00:00
Brian Warner
fcc7e64759 Doc updates and cosmetic fixes for #1115 patch. 
Removes the caveat from webapi.txt about count-good-share-hosts being wrong.

This series should close #1115.
 2012-05-13 01:15:50 -07:00
Andrew Miller
04eb6086ad Fixed an error in previous commit where an empty servermap would throw an exception in 'count-good-share-hosts'. Augmented unit test. 
Signed-off-by: Andrew Miller <amiller@dappervision.com>
 2012-05-13 00:59:30 -07:00
Andrew Miller
4b7f34d179 Added tests for count-good-share-hosts under check and repair conditions. Patched the incorrect computation in immutable/filenode.py 
Signed-off-by: Andrew Miller <amiller@dappervision.com>

Fixed missing import statements

Signed-off-by: Andrew Miller <amiller@dappervision.com>
 2012-05-13 00:59:30 -07:00
Brian Warner
8f630a76fe test_web: fix use of headers= that's been wrong for a while 2012-05-13 00:45:12 -07:00
Brian Warner
5d404db898 webapi: don't allow ETags in t=info or t=rename-form, both are variable 
t=info contains randomly-generated ophandles, and t=rename-form contains the
name of the child being renamed, so neither is eligible for a
short-circuiting ETag. Enhanced test_web to exercise this. Had to improve
FakeCHKFileNode slightly to let it participate. Refs #443.
 2012-05-13 00:45:11 -07:00
Brian Warner
d437a9b33e test_web: improve ETag tests, add If-None-Match test 2012-05-13 00:45:11 -07:00
Andrew Miller
16eddc35e8 Added unit tests covering #466:comment-15. Refactored the 'etag' behavior for immutable files to respond to all GET '?t=' flags, not just t=None 
Signed-off-by: Andrew Miller <amiller@dappervision.com>
 2012-05-13 00:45:11 -07:00
Jeremy Fitzhardinge
4f1bc1b387 Short circuit GET on ETags match 
When client does a conditional GET/HEAD with If-none-match:, if the condition
fails (ie, the client's ETag matches the file's) then we can short-circuit
the whole process and immediately return an empty body.
 2012-05-13 00:45:11 -07:00
Jeremy Fitzhardinge
518ef25bdb Add ETags for immutable directories 
Like immutable files, the ETag is based on the storage index. However, since
a directory is a special interpretation of a file, it is distinguished from
the file by prepending "DIR:" onto the start of the ETag, and adding
-representation on the end (where -representation is the ?t= argument, json,
info, etc).

It also checks the return of setETag and avoids generating a representation
if the client already has it.
 2012-05-13 00:45:11 -07:00
david-sarah
379901bf8f Add 'tahoe debug flogtool' command, test for --help, and docs. This version gets the help synopses more correct, and changes the doc to say that this command is added in 1.10.0 rather than 1.9.2. fixes #1693 2012-03-31 22:41:22 +00:00
Brian Warner
eb2a4ef246 webapi: remove undocumented t=mkdir-p operation 
Closes #380
 2012-05-12 19:19:43 -07:00
Brian Warner
8417579147 webui: merge 'move' form with 'rename' form 2012-05-09 14:21:37 -07:00
Brian Warner
81c30b1cd1 test_web: improve shouldFail2() error reporting 2012-05-09 14:18:37 -07:00
Brian Warner
a92385b54b webapi 'move'-button cleanups 
test_web.py: use shouldFail2(), safer than old shouldFail()
directory.py: forbid slashes in from_name=, return BAD_REQUEST instead of
              GONE when trying to move into a non-directory
 2012-05-09 14:18:27 -07:00
Marcus Wanner
e89bce21a5 Add unit test for moving a directory 
My gut tells me this case sould be tested. The rename suite tests it, so
move's will too.
 2012-05-09 13:07:14 -07:00
Marcus Wanner
e58a01270b Change the arbitrary URI support from implied to explicit 
The move webapi function now takes a target_type argument which lets it
know whether the target is a subdirectory name or URI. This is an
improvement over the old system in which the move handler tried to guess
whether the target was a name or a URI. Also fixed a little docs
copypaste problem and tweaked some line wrapping.
 2012-05-09 13:07:14 -07:00
Marcus Wanner
b29d0920d3 Adding 'move' button to web UI, closes #1579 
This adds "move file" capability to the web UI's directory display. The
support and test framework is heavily based on the similar "rename file"
feature. Unit tests and documentation are included. Multiple in-progress
versions of this patch may be found in ticket 1579. This version
includes arbitrary URI target support and is compatible with the change
from tahoe_css to tahoe.css.
 2012-05-09 13:07:13 -07:00
david-sarah
a555336910 Make sure that foolscap.logging.log.setLogDir is called with a str (not unicode) path, v2. Includes test. fixes #1725 2012-04-29 02:28:44 +00:00
Brian Warner
842a3a132b introweb announcements: show serverid, not tubid 
'serverid' is the pubkey (for V2 clients), falling back to the tubid (for V1
clients). This also required cleaning up the way the index is created for the
old V1 introducer.
 2012-04-24 01:37:28 -04:00
Brian Warner
84c9f3bfb4 Fix introweb display for mixed V1/V2 clients. Closes #1721. 
This significantly cleans up the IntroducerServer web-status renderers.
Instead of poking around in the introducer's internals, now the web-status
renderers get clean AnnouncementDescriptor and SubscriberDescriptor
objects. They are still somewhat foolscap-centric, but will provide a clean
abstraction boundary for future improvements.

The specific #1721 bug was that old (V1) subscribers were handled by
wrapping their RemoteReference in a special WrapV1SubscriberInV2Interface
object, but the web-status display was trying to peek inside the object to
learn what host+port it was associated with, and the wrapper did not proxy
those extra attributes.

A test was added to test_introducer to make sure the introweb page renders
properly and at least contains the nicknames of both the V1 and V2 clients.
 2012-04-23 18:30:53 -04:00
david-sarah
9d87f8ddd5 Change capitalization of WUI and introducer welcome page headings; add test for introducer welcome page. Also fix a typo in a CSS class name. fixes #1708 2012-04-05 23:57:23 +00:00
Brian Warner
4e93f77289 test/common.py: remove ununsed 'is_bad' mechanism 
This was a premature feature addition to the mock filenode, and gets in the
way of the IServer refactoring I'm trying to do. Best to remove it now and
re-introduce it in a better form later when it's actually needed.
 2012-04-04 12:11:03 -07:00
Brian Warner
4b8876c5da checker.py: minor simplifications 2012-04-04 12:05:31 -07:00
Brian Warner
55973e61e5 make IServer instances retain identity in copy() and deepcopy() 2012-04-04 11:14:09 -07:00
Brian Warner
b38cfd0235 move IServer from storage_client.py to interfaces.py 2012-04-04 11:13:59 -07:00
Brian Warner
bf390cca4b Rename web CheckResults to -Renderer, to avoid confusion. Closes #1705. 
This avoids the name collision between the actual results
objects (defined in allmydata.check_results) and the code that renders
these objects into HTML (defined in allmydata.web.check_results). Only
the web-side objects were renamed.
 2012-04-02 20:04:51 -07:00
Brian Warner
470acbf1e1 servermap.py: oops, fix _done() condition, good catch by davidsarah 2012-04-01 15:10:34 -07:00
david-sarah
93e50f5e16 interfaces.py: ensure that NoSuchChildError can be converted to str even when it is for a non-ASCII name. fixes #1483 2011-08-14 22:59:59 +00:00
david-sarah
122f63aa64 Spelling error in a comment. 2012-04-01 01:36:55 +00:00
david-sarah
972beffb25 test_node.py: test that we tolerate a UTF-8 BOM at the start of tahoe.cfg, and can read UTF-8 option values. refs #1470 2011-08-08 18:05:52 +00:00
david-sarah
ba0df23927 node.py: tolerate a UTF-8 BOM at the start of tahoe.cfg. fixes #1470 2011-08-08 18:02:04 +00:00
david-sarah
87ca4fc705 mutable/layout.py: improve confusing documentation of layout. fixes #1534 2011-09-14 14:39:47 +00:00
Brian Warner
5bae4a1bd2 Mutable repair: use new MODE_REPAIR to query all servers *and* get privkey 
This fixes bug #1689. Repair was using MODE_READ to build the servermap,
which doesn't try hard enough to grab the privkey, and also doesn't guarantee
sending queries to all servers. This patch adds a new MODE_REPAIR which does
both, and does a separate, distinct mapupdate to start wth repair cycle,
instead of relying upon the (MODE_CHECK) mapupdate leftover from the
filecheck that triggered the repair.
 2012-03-31 11:39:02 -07:00
Brian Warner
2b8a312cb7 Add test for bug #1689: repairing empty file hits no-privkey assertion 2012-03-31 11:39:02 -07:00
david-sarah
74cfa65f0d test_ftp.py: fix a couple of unused imports. refs #1668 2012-03-31 02:17:25 +00:00
Peter Le Bek
7f6ee7e918 ftpd file `size' attribute must be an integer 2012-03-22 13:18:06 +00:00
Peter Le Bek
be1fd9d2b5 unit test for ftpd LIST 2012-03-30 23:41:39 +00:00
Peter Le Bek
14a50f258a fix ftpd mtime retrieval 2012-03-30 23:41:19 +00:00
Brian Warner
24812905a1 test_introducer.SystemTest: fix race condition 
SystemTest has a couple of different phases, separated by a poller which
waits for everything to be idle (all messages delivered, none in flight). It
does this by watching some internal "_debug_outstanding" counters in the
server and in each client, and waiting for them to hit zero.

Just before the last phase, we replace the server with a new one (to make
sure clients re-send their messages properly). Unfortunately, the polling
function closed over the variable holding the original server, and didn't see
the replacement. It kept polling the old server, and failed to notice the
outstanding messages for the new server. The last phase of the test (check3)
was started too early, which failed (since some messages had not yet been
delivered), and then exploded in a flurry of dirty-reactor errors (because
some messages were delivered after test shutdown).

This replaces the closed-over-variable with a "self.the_introducer", which
seems to fix the race.

One additional place to look at in the future: the client
announcement-receive path (remote_announce) uses an eventually(). If the
message has been received and the eventual-send posted (but not yet executed)
when the poller sees it, the poller might erroneously conclude that the
client is idle and cause the same problem as above. To fix this, the poller
(probably all pollers) could be enhanced to do a flushEventualQueue before
querying the are-we-done-yet predicate function.
 2012-03-30 17:29:06 -07:00
Brian Warner
a56e639346 Fix mutable status (mapupdate/retrieve/publish) to use serverids, not tubids 
This still leaves immutable-publish results incorrectly using tubids instead
of serverids. That will need some more work, since it might change the Helper
interface.
 2012-03-17 17:01:35 -07:00
Brian Warner
890449801e IServer.get_name(): remove v0- prefix from displayed server names 
Don't remove the prefix if it isn't there: that avoids the need to fix tests
which use a bogus key (usually all-zeros).
 2012-03-17 17:01:35 -07:00
david-sarah
cef646c4a7 Fix a missing comma in the last patch. refs #1295 2012-03-14 23:50:40 +00:00
david-sarah
421b8b9369 Temporarily suppress the DeprecationWarning about IFinishableConsumer; it's irritating, but not in a way that is likely to make me fix the underlying issue (#1525) any sooner :-). refs #1295 2012-03-14 23:47:29 +00:00
Brian Warner
a133c08732 minor: hush pyflakes, move pycryptopp dep to unconditional section 
Also change Makefile's "pyflakes" rule to emit less output, so buildbot will
count errors properly.
 2012-03-13 23:20:35 -07:00
david-sarah
d9c7dfa4a0 Update more links from http: to https: in documentation and comments. 2012-03-13 20:26:54 +00:00
Brian Warner
bc21726dfd new introducer: signed extensible dictionary-based messages! refs #466 
This introduces new client and server halves to the Introducer (renaming the
old one with a _V1 suffix). Both have fallbacks to accomodate talking to a
different version: the publishing client switches on whether the server's
.get_version() advertises V2 support, the server switches on which
subscription method was invoked by the subscribing client.

The V2 protocol sends a three-tuple of (serialized announcement dictionary,
signature, pubkey) for each announcement. The V2 server dispatches messages
to subscribers according to the service-name, and throws errors for invalid
signatures, but does not otherwise examine the messages. The V2 receiver's
subscription callback will receive a (serverid, ann_dict) pair. The
'serverid' will be equal to the pubkey if all of the following are true:

  the originating client is V2, and was told a privkey to use
  the announcement went through a V2 server
  the signature is valid

If not, 'serverid' will be equal to the tubid portion of the announced FURL,
as was the case for V1 receivers.

Servers will create a keypair if one does not exist yet, stored in
private/server.privkey .

The signed announcement dictionary puts the server FURL in a key named
"anonymous-storage-FURL", which anticipates upcoming Accounting-related
changes in the server advertisements. It also provides a key named
"permutation-seed-base32" to tell clients what permutation seed to use. This
is computed at startup, using tubid if there are existing shares, otherwise
the pubkey, to retain share-order compatibility for existing servers.
 2012-03-13 18:24:32 -07:00
Brian Warner
5ea8b698a5 'tahoe admin generate-keypair/derive-pubkey': add Ed25519 keypair commands 
Also add parse_privkey/parse_pubkey tools to util.keyutil
 2012-03-13 18:24:32 -07:00
Brian Warner
0e60920baf bump pycryptopp dependency to >=0.6.0, to get ed25519 signatures 
This is for the upcoming #466 signed-introducer code.
 2012-03-13 18:24:32 -07:00
david-sarah
aaee828abb Make the link on the Welcome page to 'https://tahoe-lafs.org/', not 'http:'. Includes a test. fixes #1682 2012-03-08 23:17:58 +00:00
david-sarah
aebaa838da Update various references to allmydata.org or http://tahoe-lafs.org in comments, to https://tahoe-lafs.org. refs #1682 2012-03-08 23:17:19 +00:00
david-sarah
916d26e710 Suppress a warning from win32eventreactor on Windows (patch v2). fixes #1681 2012-02-27 19:03:17 +00:00
Brian Warner
880af4e1fd Add nickname/nodeid to storage-status web page. Closes #1204. 
Also add tahoe.css to the page, to make it look slightly prettier.
 2012-03-12 19:57:36 -07:00
Brian Warner
dc28e3144a add some quick tests of the introducer/web improvements 2012-03-12 12:35:36 -07:00
Brian Warner
e27423e4a9 introducer web page: add CSS styling, roughly match client Welcome page 
Also add /static and the top-level /tahoe.css -type stuff to the introducer's
web server.
 2012-03-06 18:25:05 -08:00
Brian Warner
ec7bc05c04 tahoe.css: fix #section typo, update welcome.xhtml to match 
The "#section" declaration (which matches id="section") should have been
".section" (which matches class="section").

The welcome page has a feature that I actually liked: the little "This
Client" sidebar sits just to the right of the start of the Controls block.
Fixing .section broke that (the clear:both introduces a gap, forcing the
Controls block to start strictly below the bottom of the This Client block).
So I also removed class="section" from the Controls block to allow them to
share the horizontal space again.
 2012-03-06 18:22:41 -08:00
Brian Warner
11f9394704 remove 'provisioning'/'reliability' from WUI, add to misc/operations_helpers 
Also remove docs related to reliability/provisioning pages
 2012-02-16 22:29:05 +00:00
Brian Warner
e0f409c681 provisioning.py: update disk sizes and usage numbers 2012-02-13 15:57:08 +00:00
Kevan
1469b1b9f9 Ensure that verification proceeds and stops when appropriate. 
The removed assertions are appropriate for a download that seeks to
return plaintext to a caller; if we don't have at least k active remote
shares, then we can't hope to do that. They're not appropriate for a
verification operation; a user can try to verify a file that has fewer
than k shares available, so that shouldn't be treated as an error.
Instead, we proceed with fewer than k shares, and ensure that we
terminate the download if we have no shares at all and we're verifying.
 2012-01-24 12:52:09 -08:00
Kevan
5e1e58d4a8 Add test_verify_mdmf_all_bad_sharedata 
test_verify_mdmf_all_bad_sharedata tests for the regression described
in ticket 1648. In particular, it will trigger the misplaced assertion
in the share activation code. It also tests to make sure that
verification continues with fewer than k shares.
 2012-01-24 12:52:09 -08:00
Brian Warner
d10f260f36 retrieve.py: unconditionally check share-hash-tree. Fixes #1654. 
Add Kevan's unit test, update known_issues.rst
 2012-01-12 13:35:53 -08:00
david-sarah
76e7f0ad4b _auto_deps.py: don't allow pycrypto 2.0.1. fixes #1631 2012-01-10 19:57:58 +00:00
Brian Warner
893eea849b mutable/retrieve.py: clean up control flow to avoid dropping errors 
* replace DeferredList with gatherResults, simplify result handling
* use BadShareError to signal recoverable problems in either fetch or
  validate, catch after _validate_block
* _validate_block is thus not responsible for noticing fetch problems
* rename _validation_or_decoding_failed() to _handle_bad_share()
* _get_needed_hashes() returns two Deferreds, instead of a hard-to-unpack
  DeferredList
 2012-01-08 14:12:48 -08:00
Brian Warner
c56839478e mutable/layout.py: raise BadShareError instead of assert() 2012-01-08 14:12:47 -08:00
Brian Warner
ca78e6b146 mutable: don't tell server about corruption unless it's really CorruptShareError 2012-01-08 14:12:45 -08:00
Brian Warner
f1752f54c0 mutable: simplify Retrieve._process_segment() to use a gatherDeferred 2012-01-08 14:12:44 -08:00
Brian Warner
7f0bc64325 Retrieve.decode(): simplify setup of DeferredList-like argument 
make it more obviously match the expectations of _decode_blocks() and
_maybe_decode_and_decrypt_segment()
 2012-01-08 14:12:40 -08:00
Brian Warner
341b8c5715 mutable: add comments about the tricky DeferredList structures in retrieve 2012-01-08 14:12:38 -08:00
david-sarah
7f8bbcc155 Use a private/drop_upload_dircap file instead of the [drop_upload]upload.dircap option in tahoe.cfg. Fail if the upload.dircap option is used, or options are missing. Also updates tests and docs. fixes #1593 2011-11-20 23:24:26 +00:00
Brian Warner
7989fe21cc test_mutable: don't use 75 shares (slow), now that the bug is fixed 
I missed this part of Kevan's fix-1628.darcs.2.patch .
 2011-12-28 14:38:19 -08:00
Kevan Carstensen
147670fd89 mutable publish: fix not-enough-shares detection. Refs #1628. 
This should match the "fix-1628.darcs.2.patch" attachment on that ticket.
 2011-12-27 21:50:18 -08:00
Kevan Carstensen
e29323f68f mutable publish: track multiple servers-per-share. Fixes some of #1628. 
The remaining work is to write additional tests.

src/allmydata/test/no_network.py:

 This supports tests in which servers leave the grid only to return with
 their shares intact at a later time.

src/allmydata/test/test_mutable.py:

 The UCWEs in the incident reports associated with #1628 all seem to be
 associated with shares that the servermap knows about, but which aren't
 accounted for during the publish process for whatever reason. Specifically,
 it looks like the publisher is only capable of keeping track of a single
 storage server for a given share. This makes the repair process worse than
 it was pre-MDMF at updating all of the shares of a particular file to the
 newest version, and can also cause spurious UCWEs. This test simulates such
 a layout and fails if an UCWE is thrown. We need to write another test to
 ensure that all copies of a share are updated to the latest version (or
 alter this test to do that), so that the test suite doesn't pass unless both
 regressions are fixed.

 We want the publisher to follow the existing share placement when uploading
 a new version of a mutable file, and we don't want this test to pass unless
 it does.

src/allmydata/mutable/publish.py:

 Before this commit, the publisher only kept track of a single writer for
 each share. This is insufficient to handle updates in which a single share
 may live on multiple servers. In the best case, an update will only update
 one of the existing shares instead of all of them. In some cases, the update
 will encounter the existing shares when publishing some other share,
 interpret it as a sign of an uncoordinated update, and fail. Keeping track
 of all of the writers helps ensure that all existing shares are updated, and
 helps avoid spurious uncoordinated write errors.
 2011-12-27 21:33:58 -08:00
david-sarah
531f562fcf Tests for ref #1592. 2011-12-17 04:31:30 +00:00
david-sarah
4618571389 test_web.py cleanup: use failUnlessIn/failIfIn in preference to 'in' operator. 2011-12-17 04:27:10 +00:00
david-sarah
054374400e Marcus Wanner's favicon patch. fixes #1592 2011-12-17 03:32:01 +00:00
Brian Warner
7d5ca407a1 setup.py: get version from git or darcs 
This replaces the setup.cfg aliases that run "darcsver" before each major
command with the new "update_version". update_version is defined in setup.py,
and tries to get a version string from either darcs or git (or leaves the
existing _version.py alone if neither VC metadata is available).

Also clean up a tiny typo in verlib.py that messed up syntax hilighting.
 2011-12-04 20:40:01 -08:00
Brian Warner
ce8d40f31b more tiny buildbot-testing whitespace changes 2011-11-18 00:20:41 +00:00
Brian Warner
d6925dcc01 more tiny buildbot-testing whitespace changes 2011-11-18 00:18:28 +00:00
Brian Warner
990f294eea remove remaining uses of nevow's "formless" module 
We're slowly moving away from Nevow, and marcusw's previous patch removed
uses of the formless CSS file, so now we can stop testing that nevow can find
that file, and remove the lingering unused "import formless" call.
 2011-11-17 14:54:23 -08:00
Marcus Wanner
0ed6260a3c 1585-webui.darcs.patch 2011-11-17 21:49:23 +00:00
Brian Warner
1445953adc Remove duplicate tahoe_css links from manifest.xhtml and rename-form.xhtml 
They were probably meant to be links to webform_css, but we aren't really
using Nevow's form-generation code anyways, so they can just be removed.
Thanks to 'marcusw' for the catch.
 2011-11-16 14:42:25 -08:00
Brian Warner
1b84612fdf iputil: handle openbsd5 (just like openbsd4) 
Patch by 'sickness'. Closes #1584
 2011-11-15 14:04:23 -08:00
david-sarah
5af3bfda61 _auto_deps.py: blacklist PyCrypto 2.4. 2011-11-05 02:24:57 +00:00
Brian Warner
7da112ccef Rewrite download-status-timeline visualizer ('viz') with d3.js 
* use d3.js v2.4.6
* add a "toggle misc events" button, to get hash/bitmap-checking details
* only draw data that's on screen, for speed
* add fragment-arg to fetch timeline data.json from somewhere else
 2011-10-31 23:18:21 -07:00
Brian Warner
5bf1ffbc87 IServer refactoring: pass IServer instances around, instead of peerids 
refs #1363

This collapses 88 small incremental changes (each of which passes all tests)
into one big patch. The development process for the long path started with
adding some temporary scaffolding, changing one method at a time, then
removing the scaffolding. The individual pieces are as follows, in reverse
chronological order (the first patch is at the end of this comment):

 commit 9bbe4174fd0d98a6cf47a8ef96e85d9ef34b2f9a
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 16:05:00 2011 -0400

     immutable/downloader/status.py: correct comment

  src/allmydata/immutable/downloader/status.py |    2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)

 commit 72146a7c7c91eac2f7c3ceb801eb7a1721376889
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 15:46:20 2011 -0400

     remove temporary ServerMap._storage_broker

  src/allmydata/mutable/checker.py   |    2 +-
  src/allmydata/mutable/filenode.py  |    2 +-
  src/allmydata/mutable/publish.py   |    2 +-
  src/allmydata/mutable/servermap.py |    5 ++---
  src/allmydata/test/test_mutable.py |    8 ++++----
  5 files changed, 9 insertions(+), 10 deletions(-)

 commit d703096b41632c47d76414b12672e076a422ff5c
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 15:37:05 2011 -0400

     remove temporary storage_broker.get_server_for_id()

  src/allmydata/storage_client.py  |    3 ---
  src/allmydata/test/no_network.py |   13 -------------
  2 files changed, 0 insertions(+), 16 deletions(-)

 commit 620cc5d80882ef6f7decfd26af8a6c7c1ddf80d1
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 12:50:06 2011 -0400

     API of Retrieve._try_to_validate_privkey(), trying to remove reader.server

  src/allmydata/mutable/retrieve.py |   10 +++++-----
  1 files changed, 5 insertions(+), 5 deletions(-)

 commit 92f43f856f4a8b36c207d1b190ed8699b5a4ecb4
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 12:48:08 2011 -0400

     API of Retrieve._validate_block(), trying to remove reader.server

  src/allmydata/mutable/retrieve.py |   14 +++++++-------
  1 files changed, 7 insertions(+), 7 deletions(-)

 commit 572d5070761861a2190349d1ed8d85dbc25698a5
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 12:36:58 2011 -0400

     API of Retrieve._mark_bad_share(), trying to remove reader.server

  src/allmydata/mutable/retrieve.py |   21 +++++++++------------
  1 files changed, 9 insertions(+), 12 deletions(-)

 commit a793ff00c0de1e2eec7b46288fdf388c7a2bec89
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 12:06:13 2011 -0400

     remove now-unused get_rref_for_serverid()

  src/allmydata/mutable/servermap.py |    3 ---
  1 files changed, 0 insertions(+), 3 deletions(-)

 commit 1b9827cc9366bf90b93297fdd6832f2ad0480ce7
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 12:03:09 2011 -0400

     Retrieve: stop adding .serverid attributes to readers

  src/allmydata/mutable/retrieve.py |    1 -
  1 files changed, 0 insertions(+), 1 deletions(-)

 commit 5d4e9d491b19e49d2e443a1dfff2c672842c36ef
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 12:03:34 2011 -0400

     return value of Retrieve(verify=True)

  src/allmydata/mutable/checker.py  |   11 ++++++-----
  src/allmydata/mutable/retrieve.py |    3 +--
  2 files changed, 7 insertions(+), 7 deletions(-)

 commit e9ab7978c384e1f677cb7779dc449b1044face82
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 11:54:23 2011 -0400

     Retrieve._bad_shares (but not return value, used by Verifier)

  src/allmydata/mutable/retrieve.py |    7 ++++---
  1 files changed, 4 insertions(+), 3 deletions(-)

 commit 2d91926de233ec5c881f30e36b4a30ad92ab42a9
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 11:51:23 2011 -0400

     Publish: stop adding .serverid attributes to writers

  src/allmydata/mutable/publish.py |    9 ++-------
  1 files changed, 2 insertions(+), 7 deletions(-)

 commit 47c7a0105dec7cbf4f7e0a3ce800bbb85b15df4a
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 11:56:33 2011 -0400

     API of get_write_enabler()

  src/allmydata/mutable/filenode.py |    7 ++++---
  src/allmydata/mutable/publish.py  |    4 ++--
  src/allmydata/test/no_network.py  |    3 +++
  3 files changed, 9 insertions(+), 5 deletions(-)

 commit 9196a5c6590fdbfd660325ea8358b345887d3db0
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 11:46:24 2011 -0400

     API of get_(renewal|cancel)_secret()

  src/allmydata/mutable/filenode.py  |   14 ++++++++------
  src/allmydata/mutable/publish.py   |    8 ++++----
  src/allmydata/mutable/servermap.py |    5 ++---
  3 files changed, 14 insertions(+), 13 deletions(-)

 commit de7c1552f8c163eff5b6d820b5fb3b21c1b47cb5
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 11:41:52 2011 -0400

     API of CorruptShareError. Also comment out some related+unused test_web.py code

  src/allmydata/mutable/common.py    |   13 +++++--------
  src/allmydata/mutable/retrieve.py  |   10 +++++-----
  src/allmydata/mutable/servermap.py |    8 +++-----
  src/allmydata/test/common.py       |   13 ++++++++-----
  4 files changed, 21 insertions(+), 23 deletions(-)

 commit 2c1c314046b620c16f1e66d030c150d768b7d01e
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 12:01:46 2011 -0400

     API of ServerMap.mark_bad_share()

  src/allmydata/mutable/publish.py   |    2 +-
  src/allmydata/mutable/retrieve.py  |    6 +++---
  src/allmydata/mutable/servermap.py |    6 ++----
  src/allmydata/test/test_mutable.py |    3 +--
  4 files changed, 7 insertions(+), 10 deletions(-)

 commit 1bed349030779fd0c378ae4e821384f953c6f6ff
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 11:11:17 2011 -0400

     API+name of ServerMap.shares_on_server() : only for tests, so debug_ prefix

  src/allmydata/mutable/servermap.py |    7 ++-----
  src/allmydata/test/test_mutable.py |    6 +++---
  2 files changed, 5 insertions(+), 8 deletions(-)

 commit 2d32e448677d6b818692e801045d4115b29abf21
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 11:07:10 2011 -0400

     API of ServerMap.all_servers_for_version()

  src/allmydata/mutable/servermap.py |    4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)

 commit 48f3204d1889c3e7179578125c4bdef515af3d6a
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 11:04:50 2011 -0400

     internals of ServerMap methods that use make_versionmap(), remove temp copy

  src/allmydata/mutable/servermap.py |   28 +++++++++----------------
  1 files changed, 10 insertions(+), 18 deletions(-)

 commit 5c3da77b6c777a145bd5ddfaa4db849dc9495548
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 11:01:28 2011 -0400

     API of ServerMap.make_versionmap()

  src/allmydata/mutable/checker.py   |    4 ++--
  src/allmydata/mutable/retrieve.py  |    5 ++---
  src/allmydata/mutable/servermap.py |    4 ++--
  src/allmydata/test/test_mutable.py |    7 ++++---
  4 files changed, 10 insertions(+), 10 deletions(-)

 commit b6882ece49afb4c507d118af2db346fa329209dc
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 10:53:38 2011 -0400

     make a copy of ServerMap.make_versionmap() (_make_versionmap2) for internal use

  src/allmydata/mutable/servermap.py |   18 +++++++++++++-----
  1 files changed, 13 insertions(+), 5 deletions(-)

 commit 963f8e63faf32b950eb1b8103cd2ff16fe8f0151
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 00:45:58 2011 -0400

     API of RetrieveStatus.add_problem()

  src/allmydata/mutable/retrieve.py |    5 +++--
  1 files changed, 3 insertions(+), 2 deletions(-)

 commit 4976d29ffae565a048851601c29013bbae2976d8
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 00:45:05 2011 -0400

     API of RetrieveStatus.add_fetch_timing()

  src/allmydata/mutable/retrieve.py |    5 +++--
  1 files changed, 3 insertions(+), 2 deletions(-)

 commit d057d3bbba72663ee148a8b916bc2d52be2e3982
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 00:44:04 2011 -0400

     API of Retrieve.notify_server_corruption()

  src/allmydata/mutable/retrieve.py |    6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)

 commit 8a2a81e46671c860610e0e96d6add1a57551f22d
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 00:42:32 2011 -0400

     remove unused _outstanding_queries

  src/allmydata/mutable/retrieve.py |    1 -
  1 files changed, 0 insertions(+), 1 deletions(-)

 commit 56d12cc9968d03ccd53764455c671122c4f391d1
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 00:40:57 2011 -0400

     change Retrieve.remaining_sharemap

  src/allmydata/mutable/retrieve.py |    4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)

 commit 4f0b7af4821f43290bfc70f2b1fc30149ad81281
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 10:40:18 2011 -0400

     accessor for PublishStatus._problems

  src/allmydata/mutable/publish.py |    4 +++-
  src/allmydata/web/status.py      |    2 +-
  2 files changed, 4 insertions(+), 2 deletions(-)

 commit 627087cf66d0b8cc519f4d551a967a7bd9b6a741
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 10:36:39 2011 -0400

     accessor for RetrieveStatus._problems

  src/allmydata/mutable/retrieve.py |    8 ++++++--
  src/allmydata/web/status.py       |    2 +-
  2 files changed, 7 insertions(+), 3 deletions(-)

 commit ca7dea81f03801b1c7353fc00ecba689268109cf
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 00:35:32 2011 -0400

     add .server to "reader", so we can get at it later

  src/allmydata/mutable/retrieve.py |    5 +++--
  1 files changed, 3 insertions(+), 2 deletions(-)

 commit 6ef516e24908ec195af084a7550d1921a5e983b0
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 00:32:32 2011 -0400

     temporarily give Retrieve a _storage_broker, so it can map serverids to servers

  src/allmydata/mutable/checker.py   |    3 ++-
  src/allmydata/mutable/filenode.py  |    6 ++++--
  src/allmydata/mutable/retrieve.py  |    5 +++--
  src/allmydata/test/test_mutable.py |    4 ++--
  4 files changed, 11 insertions(+), 7 deletions(-)

 commit afe08e4dd3f4ff9ff7e8a2a8d28b181e3625bcc9
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 00:21:51 2011 -0400

     mutable/retrieve.py: s/peer/server/

  src/allmydata/mutable/retrieve.py  |   82 +++++++++++++-------------
  src/allmydata/test/test_mutable.py |    6 +-
  2 files changed, 44 insertions(+), 44 deletions(-)

 commit 910afcb5d7f274880f68dd6cdb5b05f2bbc29adc
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 00:16:01 2011 -0400

     web.status.PublishStatusPage: add comment, I think .problems isn't exercised

  src/allmydata/web/status.py |    2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)

 commit 311466dd8c931bbba40d590ade867704282e7f1a
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:48:16 2011 -0400

     API of PublishStatus.add_per_server_time()

  src/allmydata/mutable/publish.py |    5 +++--
  1 files changed, 3 insertions(+), 2 deletions(-)

 commit 2df5faa1b6cbfbaded520d2320305a62fe961118
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:46:37 2011 -0400

     more simplifications

  src/allmydata/mutable/publish.py |    4 +---
  1 files changed, 1 insertions(+), 3 deletions(-)

 commit 6ac4544a3da385f2aad9392f906b90192f4f919a
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:44:08 2011 -0400

     API of ServerMap.version_on_server()

  src/allmydata/mutable/publish.py   |    2 +-
  src/allmydata/mutable/servermap.py |    4 ++--
  src/allmydata/test/test_mutable.py |    5 ++---
  3 files changed, 5 insertions(+), 6 deletions(-)

 commit 3e187e322511072e4683329df6b2c6c733a66dba
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 00:16:32 2011 -0400

     API of ServerMap.make_sharemap()

  src/allmydata/mutable/servermap.py |    4 ++--
  src/allmydata/test/test_mutable.py |    7 ++++---
  src/allmydata/web/status.py        |    4 ++--
  3 files changed, 8 insertions(+), 7 deletions(-)

 commit 318feed8437bdd8d4943c6569d38f7b54b6313cc
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:36:19 2011 -0400

     small cleanups

  src/allmydata/mutable/publish.py |    4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)

 commit bd459ed5714e1db5a7163935c54b7b0b56db8349
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:33:39 2011 -0400

     API of ServerMap.add_new_share()

  src/allmydata/mutable/publish.py   |    4 ++--
  src/allmydata/mutable/servermap.py |    6 ++----
  2 files changed, 4 insertions(+), 6 deletions(-)

 commit f2804fb6ed11d80088e0da8ed48e6c2922f2ffef
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:30:26 2011 -0400

     API of ServerMap.get_bad_shares()

  src/allmydata/mutable/publish.py   |    3 +--
  src/allmydata/mutable/servermap.py |    9 ++++-----
  2 files changed, 5 insertions(+), 7 deletions(-)

 commit 965074a47b3ce1431cb46d9a233840afcf9105f5
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:26:58 2011 -0400

     more small cleanups

  src/allmydata/mutable/publish.py |    6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)

 commit 38020da34f034f8889947dd3dc05e087ffff7106
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:18:47 2011 -0400

     change Publish.bad_share_checkstrings

  src/allmydata/mutable/publish.py |    6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)

 commit 5efebcbd2ee0c2f299ea86f7591d856c0f265304
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:16:31 2011 -0400

     change internals of Publish.update_goal()

  src/allmydata/mutable/publish.py |    8 +++-----
  1 files changed, 3 insertions(+), 5 deletions(-)

 commit e91b55ff4c2a69165b71f2c7b217ac319ff4c527
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:11:42 2011 -0400

     get rid of Publish.connections

  src/allmydata/mutable/publish.py |   27 +++++----------------------
  1 files changed, 5 insertions(+), 22 deletions(-)

 commit 64e9a53b3229ebe2f9ebf7ed502d539311d0e037
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:05:32 2011 -0400

     change Publish.bad_servers

  src/allmydata/mutable/publish.py |   10 +++++-----
  1 files changed, 5 insertions(+), 5 deletions(-)

 commit b85a934bef315a06bcfe00c9c12a3627fed2b918
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:03:07 2011 -0400

     Publish.bad_servers: fix bug, this should be a set of serverids, not writers

  src/allmydata/mutable/publish.py |    2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)

 commit 605ea15ec15ed671513819003ccd211cdb9761e0
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:00:21 2011 -0400

     change .placed

  src/allmydata/mutable/publish.py |    6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)

 commit f7aba37b1b345d5b6d5cb16e3b3f6f3c1afb658e
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 22:59:22 2011 -0400

     temporarily stash IServer as .server on the "writer" object

  src/allmydata/mutable/publish.py |    2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)

 commit f9b551d788e7db1f187fce5ab98ab5d5fe4e1c36
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 22:48:18 2011 -0400

     change Publish.goal and API of log_goal() to use IServer, not serverid

  src/allmydata/mutable/publish.py |   48 ++++++++++++++--------------
  1 files changed, 24 insertions(+), 24 deletions(-)

 commit 75f20616558e4900b8b1f685dd99aa838de6d452
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 15:27:02 2011 -0400

     API of ServerMap.get_known_shares()

  src/allmydata/mutable/publish.py   |   16 ++++++++++------
  src/allmydata/mutable/servermap.py |    7 ++-----
  2 files changed, 12 insertions(+), 11 deletions(-)

 commit 1c38c9d37bb08221b4418762234b1a62397b3b4b
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 15:20:29 2011 -0400

     Publish.full_serverlist

  src/allmydata/mutable/publish.py |   10 +++++-----
  1 files changed, 5 insertions(+), 5 deletions(-)

 commit b6cbd215a04b9cde31a7d92a97a7f048622b16f1
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 15:12:31 2011 -0400

     API of ServerMap.all_servers()

  src/allmydata/mutable/servermap.py |   19 ++++++-------------
  1 files changed, 6 insertions(+), 13 deletions(-)

 commit e63cd0315fae65357b1727ec6d5ff3c6e0d27c98
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 15:10:18 2011 -0400

     remove ServerMap.connections, set_rref_for_serverid()

  src/allmydata/mutable/servermap.py |   11 +----------
  1 files changed, 1 insertions(+), 10 deletions(-)

 commit 4df52db2f80eb12eefa5d57103c24893cde89553
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 15:04:06 2011 -0400

     API of ServerMap.mark_server_reachable()

  src/allmydata/mutable/servermap.py |    7 ++-----
  1 files changed, 2 insertions(+), 5 deletions(-)

 commit 69c715bde77944dc25181b3dbbeb042c816f9a1b
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 15:03:21 2011 -0400

     API of ServerMap.mark_server_unreachable()

  src/allmydata/mutable/servermap.py |    9 +++------
  1 files changed, 3 insertions(+), 6 deletions(-)

 commit 3d784d60eec1c508858e3a617e4411ffbcc3c1fa
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 15:02:03 2011 -0400

     API of status.set_privkey_from()

  src/allmydata/mutable/servermap.py |    7 +++----
  1 files changed, 3 insertions(+), 4 deletions(-)

 commit 544ed3ea29bed7e66da7fd29ca3f6f076f27a9e6
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 15:01:15 2011 -0400

     API of status.add_per_server_time()

  src/allmydata/mutable/servermap.py |    7 ++++---
  1 files changed, 4 insertions(+), 3 deletions(-)

 commit fffe5008b6320bd1e04c3c68389a2bf2ee383fa8
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:59:02 2011 -0400

     remove unused .versionmap

  src/allmydata/mutable/servermap.py |    7 -------
  1 files changed, 0 insertions(+), 7 deletions(-)

 commit 2816562e090d2294179db3588dafcca18de1bc2b
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:57:51 2011 -0400

     remove serverid from all log messages. Also one unused lambda.

  src/allmydata/mutable/servermap.py |   30 +++++++++++++-------------
  1 files changed, 15 insertions(+), 15 deletions(-)

 commit 28fa6b1a2738fa98c1f1dbd3d0e01ae98912d11f
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:54:30 2011 -0400

     removed unused _readers

  src/allmydata/mutable/servermap.py |    3 ---
  1 files changed, 0 insertions(+), 3 deletions(-)

 commit a8e4ed3d645ab592d1add6a1e69b6d1ebfb77817
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:54:16 2011 -0400

     remove unused _sharemap

  src/allmydata/mutable/servermap.py |    1 -
  1 files changed, 0 insertions(+), 1 deletions(-)

 commit 3f072e55cf1d0700f9fffe23f8f3a475725df588
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:49:03 2011 -0400

     _must_query

  src/allmydata/mutable/servermap.py |    8 ++++----
  1 files changed, 4 insertions(+), 4 deletions(-)

 commit c599a059b8df3f5785e4bf89fb6ecc6d8dcd708b
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:48:05 2011 -0400

     _queries_outstanding

  src/allmydata/mutable/servermap.py |   16 +++++++---------
  1 files changed, 7 insertions(+), 9 deletions(-)

 commit 7743759f98ac2c07926b2fdbd80bf52dfab33085
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:46:17 2011 -0400

     _empty_servers

  src/allmydata/mutable/servermap.py |    5 ++---
  1 files changed, 2 insertions(+), 3 deletions(-)

 commit 6bb1825916828a713a32cdf7f7411fa3ea2e1e5d
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:45:39 2011 -0400

     _good_servers

  src/allmydata/mutable/servermap.py |    4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)

 commit 1768fab1b51d8dd93ecabbaaabfadfa20cf6c3d4
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:44:59 2011 -0400

     _bad_servers

  src/allmydata/mutable/servermap.py |   14 +++++++-------
  1 files changed, 7 insertions(+), 7 deletions(-)

 commit dccbaef30f0ba714c746bf6d4a1a803c36e17b65
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:41:54 2011 -0400

     API of _try_to_set_pubkey()

  src/allmydata/mutable/servermap.py |    7 ++++---
  1 files changed, 4 insertions(+), 3 deletions(-)

 commit 0481ea70042ba3575f15eac7fd0780f8ece580cc
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:35:02 2011 -0400

     API of notify_server_corruption()

  src/allmydata/mutable/servermap.py |    6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)

 commit bea9cba18fb3b9c11bb22f18356a263ecec7351e
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:34:09 2011 -0400

     API of _got_signature_one_share()

  src/allmydata/mutable/servermap.py |    9 +++++----
  1 files changed, 5 insertions(+), 4 deletions(-)

 commit 1520123583cf78650706e114b15bb5b0ac1f4a14
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:32:33 2011 -0400

     API of _try_to_validate_privkey()

  src/allmydata/mutable/servermap.py |    9 +++++----
  1 files changed, 5 insertions(+), 4 deletions(-)

 commit 938852c9c8519c7a078f58a9b1f4dd8ec8b6715e
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:31:48 2011 -0400

     API and internals of _add_lease_failed()

  src/allmydata/mutable/servermap.py |    8 ++++----
  1 files changed, 4 insertions(+), 4 deletions(-)

 commit 3843dba367e3c19e176a622ab853cb51d2472ddf
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:30:37 2011 -0400

     API of _privkey_query_failed()

  src/allmydata/mutable/servermap.py |    5 +++--
  1 files changed, 3 insertions(+), 2 deletions(-)

 commit 2219a710e1633cd57d0ca0786490de87b3e19ba7
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:29:43 2011 -0400

     fix bug in call to _privkey_query_failed, unrelated to refactoring

  src/allmydata/mutable/servermap.py |    2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)

 commit ae615bec7d0d1b269710b6902797b12f9592ad62
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:27:17 2011 -0400

     API of _got_corrupt_share()

  src/allmydata/mutable/servermap.py |   17 +++++++++--------
  1 files changed, 9 insertions(+), 8 deletions(-)

 commit cb51c95a6f4e077278157a77dab060c8c1ad7a81
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:23:16 2011 -0400

     API of _got_results()

  src/allmydata/mutable/servermap.py |    9 +++++----
  1 files changed, 5 insertions(+), 4 deletions(-)

 commit bac9154fe0af18f226999a58ffc2362d8cf4b802
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:19:19 2011 -0400

     API of _query_failed()

  src/allmydata/mutable/servermap.py |    5 +++--
  1 files changed, 3 insertions(+), 2 deletions(-)

 commit fdc29a8ca95d4b5c503e5382b9e5d4d02141ba12
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:17:20 2011 -0400

     API of _do_read()

  src/allmydata/mutable/servermap.py |    6 ++++--
  1 files changed, 4 insertions(+), 2 deletions(-)

 commit e7e9e338f28d004aa4d423d11c65f1e271ac7322
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:20:21 2011 -0400

     API of _do_query()

  src/allmydata/mutable/servermap.py |   15 +++++++--------
  1 files changed, 7 insertions(+), 8 deletions(-)

 commit 330625b9dac4cdbe72a11464a893065b9aeed453
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:43:05 2011 -0400

     next step: first batch of updates to ServermapUpdater

     updates:
      most method-local variables in update()
      API of _build_initial_querylist()
      API of _send_initial_requests()
      .full_serverlist
      .extra_servers

  src/allmydata/mutable/servermap.py |   39 ++++++++++++++------------
  1 files changed, 21 insertions(+), 18 deletions(-)

 commit 4aadc584fa7dcb2daa86b048c81dee0049ba26d9
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 15:07:00 2011 -0400

     internal change: index _bad_shares with IServer

  src/allmydata/mutable/servermap.py |   20 ++++++++++----------
  1 files changed, 10 insertions(+), 10 deletions(-)

 commit 16d4e6fa82a9907dbdc92094213387c6a4164e41
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 18:20:47 2011 +0100

     internal change: index _known_shares with IServer instead of serverid

     callers are unchanged

  src/allmydata/mutable/servermap.py |   42 +++++++++++++++----------
  1 files changed, 25 insertions(+), 17 deletions(-)

 commit ceeb5f4938cc814a0c75d1b8f4018aed965c2176
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 18:11:43 2011 +0100

     accessors and name cleanup for servermap.Servermap.last_update_mode/time

  src/allmydata/mutable/filenode.py  |    6 +++---
  src/allmydata/mutable/publish.py   |    4 ++--
  src/allmydata/mutable/servermap.py |   17 +++++++++++------
  3 files changed, 16 insertions(+), 11 deletions(-)

 commit 8d3cbda82661c0a7e5c3d3b65cf7a5d5ab7e32c0
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 18:11:14 2011 +0100

     accessors and name cleanup for servermap.Servermap.problems

  src/allmydata/mutable/servermap.py |   21 +++++++++++++--------
  src/allmydata/test/test_mutable.py |    6 +++---
  2 files changed, 16 insertions(+), 11 deletions(-)

 commit 348f57988f79389db0aab7672e6eaa9a6d8e3219
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 18:10:41 2011 +0100

     accessors and name cleanup for servermap.Servermap.bad_shares

  src/allmydata/mutable/publish.py   |    2 +-
  src/allmydata/mutable/servermap.py |   30 ++++++++++++++-----------
  2 files changed, 18 insertions(+), 14 deletions(-)

 commit 520c9368134673cdf76c653c5e1bb91c2ab5d51e
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 18:10:05 2011 +0100

     accessors and name cleanup for servermap.Servermap.servermap .

  src/allmydata/mutable/publish.py   |   14 +++++----
  src/allmydata/mutable/servermap.py |   38 ++++++++++++++-----------
  2 files changed, 29 insertions(+), 23 deletions(-)

 commit b8b8dc38287a91dbdf494426ac801d9381ce5841
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 18:08:02 2011 +0100

     fix reachable_servers

  src/allmydata/mutable/checker.py   |    3 ++-
  src/allmydata/mutable/publish.py   |    4 +++-
  src/allmydata/mutable/servermap.py |   12 ++++++++++--
  3 files changed, 15 insertions(+), 4 deletions(-)

 commit cb0cfd1adfefad357c187aaaf690c3df68b622bc
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 18:06:03 2011 +0100

     fix Servermap.unreachable_servers

  src/allmydata/mutable/servermap.py |   11 ++++++++---
  1 files changed, 8 insertions(+), 3 deletions(-)

 commit 2d9ea79b94bd4db674d40386fda90825785ac495
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 18:03:48 2011 +0100

     give ServerMap a StorageFarmBroker, temporary

     this makes it possible for the ServerMap to accept bare serverids and still
     build data structures with IServers

  src/allmydata/mutable/checker.py   |    2 +-
  src/allmydata/mutable/filenode.py  |    2 +-
  src/allmydata/mutable/publish.py   |    2 +-
  src/allmydata/mutable/servermap.py |    5 +++--
  src/allmydata/test/test_mutable.py |    8 ++++----
  5 files changed, 10 insertions(+), 9 deletions(-)

 commit 718d1aeff6fded893f65397806d22ece928b0dd4
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 13:43:30 2011 -0400

     add StorageFarmBroker.get_server_for_id(), temporary helper

     This will go away once we're passing IServers everywhere.

  src/allmydata/storage_client.py  |    2 ++
  src/allmydata/test/no_network.py |   13 +++++++++++++
  2 files changed, 15 insertions(+), 0 deletions(-)

 commit ece20231d7fda0d503704842a4aa068dfbc2e54e
 Author: Brian Warner <warner@lothar.com>
 Date:   Sun Oct 2 01:11:50 2011 +0100

     add proper accessors for Servermap.connections, to make refactoring easier

  src/allmydata/mutable/publish.py   |    6 +++---
  src/allmydata/mutable/retrieve.py  |   10 +++++-----
  src/allmydata/mutable/servermap.py |   17 +++++++++++------
  3 files changed, 19 insertions(+), 14 deletions(-)

 commit 3b943d6bf302ff702668081a612fc4fe2604cf9c
 Author: Brian Warner <warner@lothar.com>
 Date:   Fri Sep 23 10:34:30 2011 -0700

     mutable/servermap.py and neighbors: s/peer/server/

  src/allmydata/mutable/checker.py   |   22 +-
  src/allmydata/mutable/publish.py   |  204 +++++++-------
  src/allmydata/mutable/servermap.py |  402 +++++++++++++-------------
  src/allmydata/test/test_mutable.py |   18 +-
  4 files changed, 323 insertions(+), 323 deletions(-)
IServer refactoring: pass IServer instances around, instead of peerids

refs #1363

This collapses 88 small incremental changes (each of which passes all tests)
into one big patch. The development process for the long path started with
adding some temporary scaffolding, changing one method at a time, then
removing the scaffolding. The individual pieces are as follows, in reverse
chronological order (the first patch is at the end of this comment):

 commit 9bbe4174fd0d98a6cf47a8ef96e85d9ef34b2f9a
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 16:05:00 2011 -0400

     immutable/downloader/status.py: correct comment

  src/allmydata/immutable/downloader/status.py |    2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)

 commit 72146a7c7c91eac2f7c3ceb801eb7a1721376889
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 15:46:20 2011 -0400

     remove temporary ServerMap._storage_broker

  src/allmydata/mutable/checker.py   |    2 +-
  src/allmydata/mutable/filenode.py  |    2 +-
  src/allmydata/mutable/publish.py   |    2 +-
  src/allmydata/mutable/servermap.py |    5 ++---
  src/allmydata/test/test_mutable.py |    8 ++++----
  5 files changed, 9 insertions(+), 10 deletions(-)

 commit d703096b41632c47d76414b12672e076a422ff5c
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 15:37:05 2011 -0400

     remove temporary storage_broker.get_server_for_id()

  src/allmydata/storage_client.py  |    3 ---
  src/allmydata/test/no_network.py |   13 -------------
  2 files changed, 0 insertions(+), 16 deletions(-)

 commit 620cc5d80882ef6f7decfd26af8a6c7c1ddf80d1
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 12:50:06 2011 -0400

     API of Retrieve._try_to_validate_privkey(), trying to remove reader.server

  src/allmydata/mutable/retrieve.py |   10 +++++-----
  1 files changed, 5 insertions(+), 5 deletions(-)

 commit 92f43f856f4a8b36c207d1b190ed8699b5a4ecb4
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 12:48:08 2011 -0400

     API of Retrieve._validate_block(), trying to remove reader.server

  src/allmydata/mutable/retrieve.py |   14 +++++++-------
  1 files changed, 7 insertions(+), 7 deletions(-)

 commit 572d5070761861a2190349d1ed8d85dbc25698a5
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 12:36:58 2011 -0400

     API of Retrieve._mark_bad_share(), trying to remove reader.server

  src/allmydata/mutable/retrieve.py |   21 +++++++++------------
  1 files changed, 9 insertions(+), 12 deletions(-)

 commit a793ff00c0de1e2eec7b46288fdf388c7a2bec89
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 12:06:13 2011 -0400

     remove now-unused get_rref_for_serverid()

  src/allmydata/mutable/servermap.py |    3 ---
  1 files changed, 0 insertions(+), 3 deletions(-)

 commit 1b9827cc9366bf90b93297fdd6832f2ad0480ce7
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 12:03:09 2011 -0400

     Retrieve: stop adding .serverid attributes to readers

  src/allmydata/mutable/retrieve.py |    1 -
  1 files changed, 0 insertions(+), 1 deletions(-)

 commit 5d4e9d491b19e49d2e443a1dfff2c672842c36ef
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 12:03:34 2011 -0400

     return value of Retrieve(verify=True)

  src/allmydata/mutable/checker.py  |   11 ++++++-----
  src/allmydata/mutable/retrieve.py |    3 +--
  2 files changed, 7 insertions(+), 7 deletions(-)

 commit e9ab7978c384e1f677cb7779dc449b1044face82
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 11:54:23 2011 -0400

     Retrieve._bad_shares (but not return value, used by Verifier)

  src/allmydata/mutable/retrieve.py |    7 ++++---
  1 files changed, 4 insertions(+), 3 deletions(-)

 commit 2d91926de233ec5c881f30e36b4a30ad92ab42a9
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 11:51:23 2011 -0400

     Publish: stop adding .serverid attributes to writers

  src/allmydata/mutable/publish.py |    9 ++-------
  1 files changed, 2 insertions(+), 7 deletions(-)

 commit 47c7a0105dec7cbf4f7e0a3ce800bbb85b15df4a
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 11:56:33 2011 -0400

     API of get_write_enabler()

  src/allmydata/mutable/filenode.py |    7 ++++---
  src/allmydata/mutable/publish.py  |    4 ++--
  src/allmydata/test/no_network.py  |    3 +++
  3 files changed, 9 insertions(+), 5 deletions(-)

 commit 9196a5c6590fdbfd660325ea8358b345887d3db0
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 11:46:24 2011 -0400

     API of get_(renewal|cancel)_secret()

  src/allmydata/mutable/filenode.py  |   14 ++++++++------
  src/allmydata/mutable/publish.py   |    8 ++++----
  src/allmydata/mutable/servermap.py |    5 ++---
  3 files changed, 14 insertions(+), 13 deletions(-)

 commit de7c1552f8c163eff5b6d820b5fb3b21c1b47cb5
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 11:41:52 2011 -0400

     API of CorruptShareError. Also comment out some related+unused test_web.py code

  src/allmydata/mutable/common.py    |   13 +++++--------
  src/allmydata/mutable/retrieve.py  |   10 +++++-----
  src/allmydata/mutable/servermap.py |    8 +++-----
  src/allmydata/test/common.py       |   13 ++++++++-----
  4 files changed, 21 insertions(+), 23 deletions(-)

 commit 2c1c314046b620c16f1e66d030c150d768b7d01e
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 12:01:46 2011 -0400

     API of ServerMap.mark_bad_share()

  src/allmydata/mutable/publish.py   |    2 +-
  src/allmydata/mutable/retrieve.py  |    6 +++---
  src/allmydata/mutable/servermap.py |    6 ++----
  src/allmydata/test/test_mutable.py |    3 +--
  4 files changed, 7 insertions(+), 10 deletions(-)

 commit 1bed349030779fd0c378ae4e821384f953c6f6ff
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 11:11:17 2011 -0400

     API+name of ServerMap.shares_on_server() : only for tests, so debug_ prefix

  src/allmydata/mutable/servermap.py |    7 ++-----
  src/allmydata/test/test_mutable.py |    6 +++---
  2 files changed, 5 insertions(+), 8 deletions(-)

 commit 2d32e448677d6b818692e801045d4115b29abf21
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 11:07:10 2011 -0400

     API of ServerMap.all_servers_for_version()

  src/allmydata/mutable/servermap.py |    4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)

 commit 48f3204d1889c3e7179578125c4bdef515af3d6a
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 11:04:50 2011 -0400

     internals of ServerMap methods that use make_versionmap(), remove temp copy

  src/allmydata/mutable/servermap.py |   28 +++++++++----------------
  1 files changed, 10 insertions(+), 18 deletions(-)

 commit 5c3da77b6c777a145bd5ddfaa4db849dc9495548
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 11:01:28 2011 -0400

     API of ServerMap.make_versionmap()

  src/allmydata/mutable/checker.py   |    4 ++--
  src/allmydata/mutable/retrieve.py  |    5 ++---
  src/allmydata/mutable/servermap.py |    4 ++--
  src/allmydata/test/test_mutable.py |    7 ++++---
  4 files changed, 10 insertions(+), 10 deletions(-)

 commit b6882ece49afb4c507d118af2db346fa329209dc
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 10:53:38 2011 -0400

     make a copy of ServerMap.make_versionmap() (_make_versionmap2) for internal use

  src/allmydata/mutable/servermap.py |   18 +++++++++++++-----
  1 files changed, 13 insertions(+), 5 deletions(-)

 commit 963f8e63faf32b950eb1b8103cd2ff16fe8f0151
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 00:45:58 2011 -0400

     API of RetrieveStatus.add_problem()

  src/allmydata/mutable/retrieve.py |    5 +++--
  1 files changed, 3 insertions(+), 2 deletions(-)

 commit 4976d29ffae565a048851601c29013bbae2976d8
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 00:45:05 2011 -0400

     API of RetrieveStatus.add_fetch_timing()

  src/allmydata/mutable/retrieve.py |    5 +++--
  1 files changed, 3 insertions(+), 2 deletions(-)

 commit d057d3bbba72663ee148a8b916bc2d52be2e3982
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 00:44:04 2011 -0400

     API of Retrieve.notify_server_corruption()

  src/allmydata/mutable/retrieve.py |    6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)

 commit 8a2a81e46671c860610e0e96d6add1a57551f22d
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 00:42:32 2011 -0400

     remove unused _outstanding_queries

  src/allmydata/mutable/retrieve.py |    1 -
  1 files changed, 0 insertions(+), 1 deletions(-)

 commit 56d12cc9968d03ccd53764455c671122c4f391d1
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 00:40:57 2011 -0400

     change Retrieve.remaining_sharemap

  src/allmydata/mutable/retrieve.py |    4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)

 commit 4f0b7af4821f43290bfc70f2b1fc30149ad81281
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 10:40:18 2011 -0400

     accessor for PublishStatus._problems

  src/allmydata/mutable/publish.py |    4 +++-
  src/allmydata/web/status.py      |    2 +-
  2 files changed, 4 insertions(+), 2 deletions(-)

 commit 627087cf66d0b8cc519f4d551a967a7bd9b6a741
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 10:36:39 2011 -0400

     accessor for RetrieveStatus._problems

  src/allmydata/mutable/retrieve.py |    8 ++++++--
  src/allmydata/web/status.py       |    2 +-
  2 files changed, 7 insertions(+), 3 deletions(-)

 commit ca7dea81f03801b1c7353fc00ecba689268109cf
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 00:35:32 2011 -0400

     add .server to "reader", so we can get at it later

  src/allmydata/mutable/retrieve.py |    5 +++--
  1 files changed, 3 insertions(+), 2 deletions(-)

 commit 6ef516e24908ec195af084a7550d1921a5e983b0
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 00:32:32 2011 -0400

     temporarily give Retrieve a _storage_broker, so it can map serverids to servers

  src/allmydata/mutable/checker.py   |    3 ++-
  src/allmydata/mutable/filenode.py  |    6 ++++--
  src/allmydata/mutable/retrieve.py  |    5 +++--
  src/allmydata/test/test_mutable.py |    4 ++--
  4 files changed, 11 insertions(+), 7 deletions(-)

 commit afe08e4dd3f4ff9ff7e8a2a8d28b181e3625bcc9
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 00:21:51 2011 -0400

     mutable/retrieve.py: s/peer/server/

  src/allmydata/mutable/retrieve.py  |   82 +++++++++++++-------------
  src/allmydata/test/test_mutable.py |    6 +-
  2 files changed, 44 insertions(+), 44 deletions(-)

 commit 910afcb5d7f274880f68dd6cdb5b05f2bbc29adc
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 00:16:01 2011 -0400

     web.status.PublishStatusPage: add comment, I think .problems isn't exercised

  src/allmydata/web/status.py |    2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)

 commit 311466dd8c931bbba40d590ade867704282e7f1a
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:48:16 2011 -0400

     API of PublishStatus.add_per_server_time()

  src/allmydata/mutable/publish.py |    5 +++--
  1 files changed, 3 insertions(+), 2 deletions(-)

 commit 2df5faa1b6cbfbaded520d2320305a62fe961118
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:46:37 2011 -0400

     more simplifications

  src/allmydata/mutable/publish.py |    4 +---
  1 files changed, 1 insertions(+), 3 deletions(-)

 commit 6ac4544a3da385f2aad9392f906b90192f4f919a
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:44:08 2011 -0400

     API of ServerMap.version_on_server()

  src/allmydata/mutable/publish.py   |    2 +-
  src/allmydata/mutable/servermap.py |    4 ++--
  src/allmydata/test/test_mutable.py |    5 ++---
  3 files changed, 5 insertions(+), 6 deletions(-)

 commit 3e187e322511072e4683329df6b2c6c733a66dba
 Author: Brian Warner <warner@lothar.com>
 Date:   Tue Oct 4 00:16:32 2011 -0400

     API of ServerMap.make_sharemap()

  src/allmydata/mutable/servermap.py |    4 ++--
  src/allmydata/test/test_mutable.py |    7 ++++---
  src/allmydata/web/status.py        |    4 ++--
  3 files changed, 8 insertions(+), 7 deletions(-)

 commit 318feed8437bdd8d4943c6569d38f7b54b6313cc
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:36:19 2011 -0400

     small cleanups

  src/allmydata/mutable/publish.py |    4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)

 commit bd459ed5714e1db5a7163935c54b7b0b56db8349
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:33:39 2011 -0400

     API of ServerMap.add_new_share()

  src/allmydata/mutable/publish.py   |    4 ++--
  src/allmydata/mutable/servermap.py |    6 ++----
  2 files changed, 4 insertions(+), 6 deletions(-)

 commit f2804fb6ed11d80088e0da8ed48e6c2922f2ffef
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:30:26 2011 -0400

     API of ServerMap.get_bad_shares()

  src/allmydata/mutable/publish.py   |    3 +--
  src/allmydata/mutable/servermap.py |    9 ++++-----
  2 files changed, 5 insertions(+), 7 deletions(-)

 commit 965074a47b3ce1431cb46d9a233840afcf9105f5
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:26:58 2011 -0400

     more small cleanups

  src/allmydata/mutable/publish.py |    6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)

 commit 38020da34f034f8889947dd3dc05e087ffff7106
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:18:47 2011 -0400

     change Publish.bad_share_checkstrings

  src/allmydata/mutable/publish.py |    6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)

 commit 5efebcbd2ee0c2f299ea86f7591d856c0f265304
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:16:31 2011 -0400

     change internals of Publish.update_goal()

  src/allmydata/mutable/publish.py |    8 +++-----
  1 files changed, 3 insertions(+), 5 deletions(-)

 commit e91b55ff4c2a69165b71f2c7b217ac319ff4c527
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:11:42 2011 -0400

     get rid of Publish.connections

  src/allmydata/mutable/publish.py |   27 +++++----------------------
  1 files changed, 5 insertions(+), 22 deletions(-)

 commit 64e9a53b3229ebe2f9ebf7ed502d539311d0e037
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:05:32 2011 -0400

     change Publish.bad_servers

  src/allmydata/mutable/publish.py |   10 +++++-----
  1 files changed, 5 insertions(+), 5 deletions(-)

 commit b85a934bef315a06bcfe00c9c12a3627fed2b918
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:03:07 2011 -0400

     Publish.bad_servers: fix bug, this should be a set of serverids, not writers

  src/allmydata/mutable/publish.py |    2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)

 commit 605ea15ec15ed671513819003ccd211cdb9761e0
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 23:00:21 2011 -0400

     change .placed

  src/allmydata/mutable/publish.py |    6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)

 commit f7aba37b1b345d5b6d5cb16e3b3f6f3c1afb658e
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 22:59:22 2011 -0400

     temporarily stash IServer as .server on the "writer" object

  src/allmydata/mutable/publish.py |    2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)

 commit f9b551d788e7db1f187fce5ab98ab5d5fe4e1c36
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 22:48:18 2011 -0400

     change Publish.goal and API of log_goal() to use IServer, not serverid

  src/allmydata/mutable/publish.py |   48 ++++++++++++++--------------
  1 files changed, 24 insertions(+), 24 deletions(-)

 commit 75f20616558e4900b8b1f685dd99aa838de6d452
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 15:27:02 2011 -0400

     API of ServerMap.get_known_shares()

  src/allmydata/mutable/publish.py   |   16 ++++++++++------
  src/allmydata/mutable/servermap.py |    7 ++-----
  2 files changed, 12 insertions(+), 11 deletions(-)

 commit 1c38c9d37bb08221b4418762234b1a62397b3b4b
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 15:20:29 2011 -0400

     Publish.full_serverlist

  src/allmydata/mutable/publish.py |   10 +++++-----
  1 files changed, 5 insertions(+), 5 deletions(-)

 commit b6cbd215a04b9cde31a7d92a97a7f048622b16f1
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 15:12:31 2011 -0400

     API of ServerMap.all_servers()

  src/allmydata/mutable/servermap.py |   19 ++++++-------------
  1 files changed, 6 insertions(+), 13 deletions(-)

 commit e63cd0315fae65357b1727ec6d5ff3c6e0d27c98
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 15:10:18 2011 -0400

     remove ServerMap.connections, set_rref_for_serverid()

  src/allmydata/mutable/servermap.py |   11 +----------
  1 files changed, 1 insertions(+), 10 deletions(-)

 commit 4df52db2f80eb12eefa5d57103c24893cde89553
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 15:04:06 2011 -0400

     API of ServerMap.mark_server_reachable()

  src/allmydata/mutable/servermap.py |    7 ++-----
  1 files changed, 2 insertions(+), 5 deletions(-)

 commit 69c715bde77944dc25181b3dbbeb042c816f9a1b
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 15:03:21 2011 -0400

     API of ServerMap.mark_server_unreachable()

  src/allmydata/mutable/servermap.py |    9 +++------
  1 files changed, 3 insertions(+), 6 deletions(-)

 commit 3d784d60eec1c508858e3a617e4411ffbcc3c1fa
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 15:02:03 2011 -0400

     API of status.set_privkey_from()

  src/allmydata/mutable/servermap.py |    7 +++----
  1 files changed, 3 insertions(+), 4 deletions(-)

 commit 544ed3ea29bed7e66da7fd29ca3f6f076f27a9e6
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 15:01:15 2011 -0400

     API of status.add_per_server_time()

  src/allmydata/mutable/servermap.py |    7 ++++---
  1 files changed, 4 insertions(+), 3 deletions(-)

 commit fffe5008b6320bd1e04c3c68389a2bf2ee383fa8
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:59:02 2011 -0400

     remove unused .versionmap

  src/allmydata/mutable/servermap.py |    7 -------
  1 files changed, 0 insertions(+), 7 deletions(-)

 commit 2816562e090d2294179db3588dafcca18de1bc2b
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:57:51 2011 -0400

     remove serverid from all log messages. Also one unused lambda.

  src/allmydata/mutable/servermap.py |   30 +++++++++++++-------------
  1 files changed, 15 insertions(+), 15 deletions(-)

 commit 28fa6b1a2738fa98c1f1dbd3d0e01ae98912d11f
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:54:30 2011 -0400

     removed unused _readers

  src/allmydata/mutable/servermap.py |    3 ---
  1 files changed, 0 insertions(+), 3 deletions(-)

 commit a8e4ed3d645ab592d1add6a1e69b6d1ebfb77817
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:54:16 2011 -0400

     remove unused _sharemap

  src/allmydata/mutable/servermap.py |    1 -
  1 files changed, 0 insertions(+), 1 deletions(-)

 commit 3f072e55cf1d0700f9fffe23f8f3a475725df588
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:49:03 2011 -0400

     _must_query

  src/allmydata/mutable/servermap.py |    8 ++++----
  1 files changed, 4 insertions(+), 4 deletions(-)

 commit c599a059b8df3f5785e4bf89fb6ecc6d8dcd708b
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:48:05 2011 -0400

     _queries_outstanding

  src/allmydata/mutable/servermap.py |   16 +++++++---------
  1 files changed, 7 insertions(+), 9 deletions(-)

 commit 7743759f98ac2c07926b2fdbd80bf52dfab33085
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:46:17 2011 -0400

     _empty_servers

  src/allmydata/mutable/servermap.py |    5 ++---
  1 files changed, 2 insertions(+), 3 deletions(-)

 commit 6bb1825916828a713a32cdf7f7411fa3ea2e1e5d
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:45:39 2011 -0400

     _good_servers

  src/allmydata/mutable/servermap.py |    4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)

 commit 1768fab1b51d8dd93ecabbaaabfadfa20cf6c3d4
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:44:59 2011 -0400

     _bad_servers

  src/allmydata/mutable/servermap.py |   14 +++++++-------
  1 files changed, 7 insertions(+), 7 deletions(-)

 commit dccbaef30f0ba714c746bf6d4a1a803c36e17b65
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:41:54 2011 -0400

     API of _try_to_set_pubkey()

  src/allmydata/mutable/servermap.py |    7 ++++---
  1 files changed, 4 insertions(+), 3 deletions(-)

 commit 0481ea70042ba3575f15eac7fd0780f8ece580cc
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:35:02 2011 -0400

     API of notify_server_corruption()

  src/allmydata/mutable/servermap.py |    6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)

 commit bea9cba18fb3b9c11bb22f18356a263ecec7351e
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:34:09 2011 -0400

     API of _got_signature_one_share()

  src/allmydata/mutable/servermap.py |    9 +++++----
  1 files changed, 5 insertions(+), 4 deletions(-)

 commit 1520123583cf78650706e114b15bb5b0ac1f4a14
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:32:33 2011 -0400

     API of _try_to_validate_privkey()

  src/allmydata/mutable/servermap.py |    9 +++++----
  1 files changed, 5 insertions(+), 4 deletions(-)

 commit 938852c9c8519c7a078f58a9b1f4dd8ec8b6715e
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:31:48 2011 -0400

     API and internals of _add_lease_failed()

  src/allmydata/mutable/servermap.py |    8 ++++----
  1 files changed, 4 insertions(+), 4 deletions(-)

 commit 3843dba367e3c19e176a622ab853cb51d2472ddf
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:30:37 2011 -0400

     API of _privkey_query_failed()

  src/allmydata/mutable/servermap.py |    5 +++--
  1 files changed, 3 insertions(+), 2 deletions(-)

 commit 2219a710e1633cd57d0ca0786490de87b3e19ba7
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:29:43 2011 -0400

     fix bug in call to _privkey_query_failed, unrelated to refactoring

  src/allmydata/mutable/servermap.py |    2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)

 commit ae615bec7d0d1b269710b6902797b12f9592ad62
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:27:17 2011 -0400

     API of _got_corrupt_share()

  src/allmydata/mutable/servermap.py |   17 +++++++++--------
  1 files changed, 9 insertions(+), 8 deletions(-)

 commit cb51c95a6f4e077278157a77dab060c8c1ad7a81
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:23:16 2011 -0400

     API of _got_results()

  src/allmydata/mutable/servermap.py |    9 +++++----
  1 files changed, 5 insertions(+), 4 deletions(-)

 commit bac9154fe0af18f226999a58ffc2362d8cf4b802
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:19:19 2011 -0400

     API of _query_failed()

  src/allmydata/mutable/servermap.py |    5 +++--
  1 files changed, 3 insertions(+), 2 deletions(-)

 commit fdc29a8ca95d4b5c503e5382b9e5d4d02141ba12
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:17:20 2011 -0400

     API of _do_read()

  src/allmydata/mutable/servermap.py |    6 ++++--
  1 files changed, 4 insertions(+), 2 deletions(-)

 commit e7e9e338f28d004aa4d423d11c65f1e271ac7322
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:20:21 2011 -0400

     API of _do_query()

  src/allmydata/mutable/servermap.py |   15 +++++++--------
  1 files changed, 7 insertions(+), 8 deletions(-)

 commit 330625b9dac4cdbe72a11464a893065b9aeed453
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 14:43:05 2011 -0400

     next step: first batch of updates to ServermapUpdater

     updates:
      most method-local variables in update()
      API of _build_initial_querylist()
      API of _send_initial_requests()
      .full_serverlist
      .extra_servers

  src/allmydata/mutable/servermap.py |   39 ++++++++++++++------------
  1 files changed, 21 insertions(+), 18 deletions(-)

 commit 4aadc584fa7dcb2daa86b048c81dee0049ba26d9
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 15:07:00 2011 -0400

     internal change: index _bad_shares with IServer

  src/allmydata/mutable/servermap.py |   20 ++++++++++----------
  1 files changed, 10 insertions(+), 10 deletions(-)

 commit 16d4e6fa82a9907dbdc92094213387c6a4164e41
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 18:20:47 2011 +0100

     internal change: index _known_shares with IServer instead of serverid

     callers are unchanged

  src/allmydata/mutable/servermap.py |   42 +++++++++++++++----------
  1 files changed, 25 insertions(+), 17 deletions(-)

 commit ceeb5f4938cc814a0c75d1b8f4018aed965c2176
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 18:11:43 2011 +0100

     accessors and name cleanup for servermap.Servermap.last_update_mode/time

  src/allmydata/mutable/filenode.py  |    6 +++---
  src/allmydata/mutable/publish.py   |    4 ++--
  src/allmydata/mutable/servermap.py |   17 +++++++++++------
  3 files changed, 16 insertions(+), 11 deletions(-)

 commit 8d3cbda82661c0a7e5c3d3b65cf7a5d5ab7e32c0
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 18:11:14 2011 +0100

     accessors and name cleanup for servermap.Servermap.problems

  src/allmydata/mutable/servermap.py |   21 +++++++++++++--------
  src/allmydata/test/test_mutable.py |    6 +++---
  2 files changed, 16 insertions(+), 11 deletions(-)

 commit 348f57988f79389db0aab7672e6eaa9a6d8e3219
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 18:10:41 2011 +0100

     accessors and name cleanup for servermap.Servermap.bad_shares

  src/allmydata/mutable/publish.py   |    2 +-
  src/allmydata/mutable/servermap.py |   30 ++++++++++++++-----------
  2 files changed, 18 insertions(+), 14 deletions(-)

 commit 520c9368134673cdf76c653c5e1bb91c2ab5d51e
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 18:10:05 2011 +0100

     accessors and name cleanup for servermap.Servermap.servermap .

  src/allmydata/mutable/publish.py   |   14 +++++----
  src/allmydata/mutable/servermap.py |   38 ++++++++++++++-----------
  2 files changed, 29 insertions(+), 23 deletions(-)

 commit b8b8dc38287a91dbdf494426ac801d9381ce5841
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 18:08:02 2011 +0100

     fix reachable_servers

  src/allmydata/mutable/checker.py   |    3 ++-
  src/allmydata/mutable/publish.py   |    4 +++-
  src/allmydata/mutable/servermap.py |   12 ++++++++++--
  3 files changed, 15 insertions(+), 4 deletions(-)

 commit cb0cfd1adfefad357c187aaaf690c3df68b622bc
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 18:06:03 2011 +0100

     fix Servermap.unreachable_servers

  src/allmydata/mutable/servermap.py |   11 ++++++++---
  1 files changed, 8 insertions(+), 3 deletions(-)

 commit 2d9ea79b94bd4db674d40386fda90825785ac495
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 18:03:48 2011 +0100

     give ServerMap a StorageFarmBroker, temporary

     this makes it possible for the ServerMap to accept bare serverids and still
     build data structures with IServers

  src/allmydata/mutable/checker.py   |    2 +-
  src/allmydata/mutable/filenode.py  |    2 +-
  src/allmydata/mutable/publish.py   |    2 +-
  src/allmydata/mutable/servermap.py |    5 +++--
  src/allmydata/test/test_mutable.py |    8 ++++----
  5 files changed, 10 insertions(+), 9 deletions(-)

 commit 718d1aeff6fded893f65397806d22ece928b0dd4
 Author: Brian Warner <warner@lothar.com>
 Date:   Mon Oct 3 13:43:30 2011 -0400

     add StorageFarmBroker.get_server_for_id(), temporary helper

     This will go away once we're passing IServers everywhere.

  src/allmydata/storage_client.py  |    2 ++
  src/allmydata/test/no_network.py |   13 +++++++++++++
  2 files changed, 15 insertions(+), 0 deletions(-)

 commit ece20231d7fda0d503704842a4aa068dfbc2e54e
 Author: Brian Warner <warner@lothar.com>
 Date:   Sun Oct 2 01:11:50 2011 +0100

     add proper accessors for Servermap.connections, to make refactoring easier

  src/allmydata/mutable/publish.py   |    6 +++---
  src/allmydata/mutable/retrieve.py  |   10 +++++-----
  src/allmydata/mutable/servermap.py |   17 +++++++++++------
  3 files changed, 19 insertions(+), 14 deletions(-)

 commit 3b943d6bf302ff702668081a612fc4fe2604cf9c
 Author: Brian Warner <warner@lothar.com>
 Date:   Fri Sep 23 10:34:30 2011 -0700

     mutable/servermap.py and neighbors: s/peer/server/

  src/allmydata/mutable/checker.py   |   22 +-
  src/allmydata/mutable/publish.py   |  204 +++++++-------
  src/allmydata/mutable/servermap.py |  402 +++++++++++++-------------
  src/allmydata/test/test_mutable.py |   18 +-
  4 files changed, 323 insertions(+), 323 deletions(-)
 2011-10-31 21:03:19 -07:00
Brian Warner
05dbb1cf86 update project-home URLs: /trac/tahoe/ was replaced by /trac/tahoe-lafs/ 2011-10-30 12:14:28 -07:00
Zooko O'Whielacronx
d9c1064d42 undo the effects of a patch I hadn't intended to commit, named "debugprint the values of blocks and hashes thereof; make the test data and the seg size small in order to make the debugprints easy to look at" 
rolling back:

Thu Sep 29 23:46:28 MDT 2011  zooko@zooko.com
  * debugprint the values of blocks and hashes thereof; make the test data and the seg size small in order to make the debugprints easy to look at

    M ./src/allmydata/mutable/publish.py -1 +2
    M ./src/allmydata/mutable/retrieve.py +3
    M ./src/allmydata/test/test_mutable.py -2 +2
 2011-10-28 22:03:49 +00:00
Zooko O'Whielacronx
416701e404 debugprint the values of blocks and hashes thereof; make the test data and the seg size small in order to make the debugprints easy to look at 2011-09-30 05:46:28 +00:00
Brian Warner
633641174a mutable/retrieve: don't write() after we've been pauseProducer'ed 
This fixes a test failure found against current Twisted trunk in
test_mutable.Filenode.test_retrieve_producer_mdmf (when it uses
PausingAndStoppingConsumer). There must be some sort of race: I could
make it fail against Twisted-11.0 if I just increased the 0.5s delay in
test_download.PausingAndStoppingConsumer to about 0.6s, and could make
Twisted-trunk pass by reducing it to about 0.3s .

I fixed the test (as opposed to the bug) by replacing the delay with a
simple reliable eventually(), and adding extra asserts to fail the test
if the consumer's write() method is called while the producer is
supposed to be paused

The bug itself was that mutable.retrieve.Retrieve wasn't checking the
"stopped" flag after resuming from a pause, and thus delivered one
segment to a consumer that wasn't expecting it. I split out
stopped-flag-checking to separate function, which is now called
immediately after _check_for_paused(). I also cleaned up some Deferred
usage and whitespace.
 2011-10-16 17:24:00 -07:00
Brian Warner
fff237be9a remove interpreter shbang lines from non-executables 
thanks to Greg Troxel for the catch
 2011-10-14 10:23:01 -07:00
Brian Warner
d28041fbe4 CLI: don't deprecate --mutable, small docs fixes. refs #1561 
Also don't accept 'tahoe mkdir --format=chk'.
 2011-10-13 21:00:02 -07:00
Brian Warner
57ee56533f add --format= to 'tahoe put'/'mkdir', remove --mutable-type. Closes #1561 2011-10-13 20:15:00 -07:00
Brian Warner
3eb60ff986 web/filenode.py: rely on Request.notifyFinish. Closes #1366. 
This is safe now that tahoe depends upon Twisted>=10.1, since notifyFinish
first appeared in Twisted-9.0
 2011-10-13 13:12:19 -07:00
david-sarah
3dc491758d util/happinessutil.py: suppress a warning from check-miscaptures. (It is not a bug because the capturing function is only used by a 'map' in the same iteration.) refs #1556 2011-10-08 22:21:06 -07:00
david-sarah
22da015dd1 Fix some more potential bugs in test code exposed by check-miscaptures.py. refs #1556 2011-10-06 20:38:47 -07:00
david-sarah
a31a701e42 Fix some potential bugs (in non-test code) exposed by check-miscaptures.py. refs #1556 2011-10-06 20:24:44 -07:00
david-sarah
bdfa7b3772 Fix some potential bugs in test code exposed by check-miscaptures.py. refs #1556 2011-10-06 19:34:43 -07:00
Brian Warner
11ce612589 oops, missed a test failure 2011-10-13 09:37:13 -07:00
Brian Warner
210761cd49 misc mutable-type fixes: 
* fix tahoe.cfg control of default mutable type
* tolerate arbitrary case in [client]mutable.format value
* small docs improvements
* use get_mutable_type() as a format-is-mutable predicate
* tighten up error message
 2011-10-13 09:32:29 -07:00
Brian Warner
01b00dc7dc webapi: use all-caps "SDMF"/"MDMF" acronyms in t=json response 
docs: upcase examples of t=json output and format= input
 2011-10-13 09:31:43 -07:00
Brian Warner
dad354b275 webapi: handle format=, remove mutable-type= 
* fix CLI commands (put, mkdir) to send format=, not mutable-type=
* fix tests
* test_cli: fix tests that observe t=json output, don't ignore failures in
  'tahoe put'
* fix handling of version= to make it easier to use the default
* interpret ?mutable=true&format=MDMF as MDMF, not SDMF
 2011-10-13 09:29:51 -07:00
david-sarah
2acc0419b9 Tests for ref #1547 2011-10-01 20:53:16 -07:00
david-sarah
978fd83856 Change the file upload forms on directory and welcome pages to use a 3-way radio button to select immutable, SDMF, or MDMF. Add '(experimental)' to the label for creating an MDMF directory. Also improve the spacing of form elements. refs #1547 2011-10-01 20:45:03 -07:00
david-sarah
ac780a4148 test_web.py: minor cleanups, mainly to make the first argument to shouldFail tests consistent 2011-10-01 21:03:32 -07:00
david-sarah
38d819ef44 Tests for ref #1552 2011-10-01 21:00:36 -07:00
david-sarah
5ae446fbd2 test/common.py: in shouldFail and shouldHTTPError, when the raised exception does not include the expected substring (or, for shouldHTTPError, when the status code is wrong), mention which test that happened in. 2011-10-10 17:22:27 -07:00
david-sarah
cf9bf2ea10 interfaces.py: remove get_extension_params and set_extension_params methods from IMutableFileURI. refs #393, #1526 2011-10-10 12:48:42 -07:00
david-sarah
de00b277cc interfaces.py: fix a typo in the name of IMutableSlotWriter.put_encprivkey. refs #393 2011-10-10 12:46:42 -07:00
david-sarah
bc0d9b682e immutable/literal.py: add pauseProducing method to LiteralProducer. refs #1537 2011-10-03 12:52:39 -07:00
david-sarah
bc50edc86e no_network.py: Clean up whitespace around code changed by previous patch. 2011-10-03 18:04:07 -07:00
david-sarah
ee25dcd2fc no_network.py: Fix potential bugs in some tests due to capture of slots in for loops. 2011-10-03 18:02:31 -07:00
Brian Warner
0716c496c8 MDMF: remove extension fields from caps, tolerate arbitrary ones. Fixes #1526 
The filecaps used to be produced with hints for 'k' and segsize, but they
weren't actually used, and doing so had the potential to limit how we change
those filecaps in the future. Also the parsing code had some problems dealing
with other numbers of extensions. Removing the existing fields and making the
parser tolerate (and ignore) extra ones makes MDMF more future-proof.
 2011-10-02 00:35:53 +01:00
david-sarah
5ba0529b87 test/test_runner.py: BinTahoe.test_path has rare nondeterministic failures; this patch probably fixes a problem where the actual cause of failure is masked by a string conversion error. 2011-09-27 15:53:36 -07:00
david-sarah
f72cba50ba mutable/publish.py: fix an unused import. refs #1542 2011-09-24 22:22:06 -07:00
david-sarah
c88adf0ac0 mutable/layout.py: make unpack_sdmf_checkstring and unpack_mdmf_checkstring more similar, and change an assert to give a more useful message if it fails. refs #1540 2011-09-24 19:36:51 -07:00
kevan
70d27cfa7c mutable/publish: handle unknown mutable share formats when handling errors 2011-09-24 17:43:05 -07:00
kevan
e8afe84813 mutable/layout: break unpack_checkstring into unpack_mdmf_checkstring and unpack_sdmf_checkstring, add distinguisher function for checkstrings 2011-09-24 17:41:34 -07:00
kevan
2187f27ad0 test/test_mutable: reenable mdmf publish surprise test 2011-09-24 16:54:15 -07:00
kevan
a911e15783 mutable/publish: use unpack_mdmf_checkstring and unpack_sdmf_checkstring instead of unpack_checkstring. fixes #1540 2011-09-24 16:51:37 -07:00
david-sarah
1fa5c729b7 mutable/publish.py: copy the self.writers dict before iterating over it, since we remove elements from it during the iteration. refs #393 2011-09-24 14:12:08 -07:00
david-sarah
f94eb86fc9 mutable/publish.py: simplify by refactoring self.outstanding to self.num_outstanding. refs #393 2011-09-24 13:50:04 -07:00
david-sarah
4af626a798 test_mutable.py: update SkipTest message for test_publish_surprise_mdmf to reference the right ticket number. refs #1540. 2011-09-23 14:16:22 -07:00
Brian Warner
f61bdbffd7 control.py: unbreak speed-test: overwrite() wants a MutableData, not str 
Really, all the upload/modify APIs should take a string or a filehandle, and
internally wrap it as needed. Callers should not need to be aware of
Uploadable() or MutableData() classes.
 2011-09-23 00:37:48 -07:00
david-sarah
5d3d0dc336 test_mutable.py: skip test_publish_surprise_mdmf, which is causing an error. refs #1534, #393 2011-09-20 11:33:19 -07:00
kevan
d93b2965c3 test/test_mutable: write publish surprise test for MDMF, rename existing test_publish_surprise to clarify that it is for SDMF 2011-09-17 17:36:57 -07:00
kevan
19cf58f2aa test/test_mutable: refactor publish surprise test into common test fixture, rewrite test_publish_surprise to use test fixture 2011-09-17 17:35:33 -07:00
kevan
8c7406508a mutable/publish: add errback immediately after write, don't consume errors from other parts of the publisher 2011-09-17 16:47:08 -07:00
david-sarah
87a936564e uri.py: fix two interface violations in verifier URI classes. refs #1474 2011-09-19 20:01:56 -07:00
Brian Warner
8e69b94588 Make platform-detection code tolerate linux-3.0, patch by zooko. 
Otherwise address-autodetection can't find ifconfig. refs #1536
 2011-09-15 13:26:20 -07:00
david-sarah
3f756e9429 test_web.py: fix a bug in _count_leases that was causing us to check only the lease count of one share file, not of all share files as intended. 2011-09-15 11:51:26 -07:00
Zooko O'Whielacronx
2025e42ec2 tests: bump up the timeout in this test that fails on FreeStorm's CentOS in order to see if it is just very slow 2011-09-12 19:42:55 -07:00
david-sarah
c10099f982 interfaces: document that the 'fills-holes-with-zero-bytes' key should be used to detect whether a storage server has that behavior. refs #1528 2011-09-12 17:28:43 -07:00
Zooko O'Whielacronx
32f80625c9 storage: more paranoid handling of bounds and palimpsests in mutable share files 
* storage server ignores requests to extend shares by sending a new_length
* storage server fills exposed holes (created by sending a write vector whose offset begins after the end of the current data) with 0 to avoid "palimpsest" exposure of previous contents
* storage server zeroes out lease info at the old location when moving it to a new location
ref. #1528
 2011-09-12 15:26:55 -07:00
Zooko O'Whielacronx
942c5e5162 storage: test that the storage server ignores requests to extend shares by sending a new_length, and that the storage server fills exposed holes with 0 to avoid "palimpsest" exposure of previous contents 
ref. #1528
 2011-09-12 15:25:54 -07:00
Zooko O'Whielacronx
20e2910c61 immutable: prevent clients from reading past the end of share data, which would allow them to learn the cancellation secret 
Declare explicitly that we prevent this problem in the server's version dict.
fixes #1528 (there are two patches that are each a sufficient fix to #1528 and this is one of them)
 2011-09-12 15:24:58 -07:00
Zooko O'Whielacronx
5476f67dc1 storage: remove the storage server's "remote_cancel_lease" function 
We're removing this function because it is currently unused, because it is dangerous, and because the bug described in #1528 leaks the cancellation secret, which allows anyone who knows a file's storage index to abuse this function to delete shares of that file.
fixes #1528 (there are two patches that are each a sufficient fix to #1528 and this is one of them)
 2011-09-12 15:23:31 -07:00
Zooko O'Whielacronx
65de17245d storage: test that the storage server does *not* have a "remote_cancel_lease" function 
We're removing this function because it is currently unused, because it is dangerous, and because the bug described in #1528 leaks the cancellation secret, which allows anyone who knows a file's storage index to abuse this function to delete shares of that file.
ref. #1528
 2011-09-12 15:23:24 -07:00
Zooko O'Whielacronx
cffc987804 immutable: test whether the server allows clients to read past the end of share data, which would allow them to learn the cancellation secret 
Also test whether the server explicitly declares that it prevents this problem.
ref #1528
 2011-09-12 15:12:01 -07:00
Brian Warner
bd709c4833 Retrieve._activate_enough_peers: rewrite Verify logic 2011-09-09 11:11:50 -07:00
Brian Warner
a15ce96846 Retrieve: implement/test stopProducing 2011-09-09 11:11:50 -07:00
Brian Warner
748e419a9b move DownloadStopped from download.common to interfaces 2011-09-09 11:11:50 -07:00
Brian Warner
425152c34e retrieve.py: remove vestigal self._validated_readers 2011-09-09 11:11:50 -07:00
Brian Warner
df07060f93 Retrieve: rewrite flow-control: use a top-level loop() to catch all errors 
This ought to close the potential for dropped errors and hanging downloads.
Verify needs to be examined, I may have broken it, although all tests pass.
 2011-09-09 11:11:50 -07:00
Brian Warner
53bbc1d06a Retrieve: merge _validate_active_prefixes into _add_active_peers 2011-09-09 11:11:50 -07:00
Brian Warner
2edfb1a334 Retrieve: remove the initial prefix-is-still-good check 
This check needs to be done with each fetch from the storage server, to
detect when someone has changed the share (i.e. our servermap goes stale).
Doing it just once at the beginning of retrieve isn't enough: a write might
occur after the first segment but before the second, etc.

_try_to_validate_prefix() was not removed: it will be used by the future
check-with-each-fetch code.

test_mutable.Roundtrip.test_corrupt_all_seqnum_late was disabled, since it
fails until this check is brought back. (the corruption it applies only
touches the prefix, not the block data, so the check-less retrieve actually
tolerates it). Don't forget to re-enable it once the check is brought back.
 2011-09-09 11:11:50 -07:00
Brian Warner
2b4f2b7fa3 MDMFSlotReadProxy: remove the queue 
This is a neat trick to reduce Foolscap overhead, but the need for an
explicit flush() complicates the Retrieve path and makes it prone to
lost-progress bugs.

Also change test_mutable.FakeStorageServer to tolerate multiple reads of the
same share in a row, a limitation exposed by turning off the queue.
 2011-09-09 11:11:50 -07:00
Brian Warner
1597aafea1 rearrange Retrieve: first step, shouldn't change order of execution 2011-09-09 11:11:49 -07:00
david-sarah
6b97e6f407 CLI: test_cli.py -- remove an unnecessary call in test_mkdir_mutable_type. refs #1527 2011-09-06 11:37:30 -07:00
david-sarah
1e59930fda CLI: improve test for 'tahoe mkdir --mutable-type='. refs #1527 2011-09-06 11:30:20 -07:00
david-sarah
23f46b758e CLI: make the --mutable-type option value for 'tahoe put' and 'tahoe mkdir' case-insensitive, and change --help for these commands accordingly. fixes #1527 2011-09-04 19:09:22 -07:00
Kevan Carstensen
64996a913d cli: make --mutable-type imply --mutable in 'tahoe put' 2011-09-03 12:09:20 -07:00
david-sarah
1b7b3bd0d4 SFTP: add a comment about a subtle interaction between OverwriteableFileConsumer and GeneralSFTPFile, and test the case it is commenting on. 2011-09-03 15:23:04 -07:00
Brian Warner
01b5124d0a improve the storage/mutable.py asserts even more 2011-09-01 09:05:43 -07:00
wilcoxjg
d2e5de3b69 storage/mutable.py: special characters in struct.foo arguments indicate standard as opposed to native sizes, we should be using these characters in these asserts 2011-09-01 01:41:44 -07:00
Brian Warner
7191b0b32e test_mutable.Version: consolidate some tests, reduce runtime from 19s to 15s 2011-08-30 22:04:51 -07:00
Kevan Carstensen
32e30c9023 mutable/retrieve: handle the case where self._read_length is 0. 
Note that the downloader will still fetch a segment for a zero-length
read, which is wasteful. Fixing that isn't specifically required to fix
#1512, but it should probably be fixed before 1.9.
 2011-08-30 14:01:41 -07:00
Brian Warner
980eb778c1 test_mutable.Update: only upload the files needed for each test. refs #1500 
This first step shaves 15% off the runtime: from 139s to 119s on my laptop.
It also fixes a couple of places where a Deferred was being dropped, which
would cause two tests to run in parallel and also confuse error reporting.
 2011-08-29 00:27:17 -07:00
Brian Warner
fd676a5846 Let Uploader retain History instead of passing it into upload(). Fixes #1079. 
This consistently records all immutable uploads in the Recent Uploads And
Downloads page, regardless of code path. Previously, certain webapi upload
operations (like PUT /uri/$DIRCAP/newchildname) failed to pass the History
object and were left out.
 2011-08-28 23:32:46 -07:00
Brian Warner
9756146d61 Fix mutable publish/retrieve timing status displays. Fixes #1505. 
publish:
* encrypt and encode times are cumulative, not just current-segment

retrieve:
* same for decrypt and decode times
* update "current status" to include segment number
* set status to Finished/Failed when download is complete
* set progress to 1.0 when complete

More improvements to consider:
* progress is currently 0% or 100%: should calculate how many segments are
  involved (remembering retrieve can be less than the whole file) and set it
  to a fraction
* "fetch" time is fuzzy: what we want is to know how much of the delay is not
  our own fault, but since we do decode/decrypt work while waiting for more
  shares, it's not straightforward
 2011-08-28 16:22:21 -07:00
Brian Warner
d575ccba28 Teach 'tahoe debug catalog-shares about MDMF. Closes #1507. 2011-08-28 01:09:31 -07:00
Brian Warner
97b601f75a debug.py: remove some dead comments 2011-08-28 00:45:56 -07:00
Brian Warner
9f827ffa7b hush pyflakes 2011-08-28 00:42:54 -07:00
Brian Warner
0f1d674d76 MutableFileNode.set_downloader_hints: never depend upon order of dict.values() 
The old code was calculating the "extension parameters" (a list) from the
downloader hints (a dictionary) with hints.values(), which is not stable, and
would result in corrupted filecaps (with the 'k' and 'segsize' hints
occasionally swapped). The new code always uses [k,segsize].
 2011-08-28 00:41:03 -07:00
Brian Warner
4b0078dc49 layout.py: fix MDMF share layout documentation 2011-08-28 00:39:21 -07:00
Brian Warner
a3d6fd4992 teach 'tahoe debug dump-share' about MDMF and offsets. refs #1507 2011-08-28 00:38:34 -07:00
Brian Warner
101ca8dfe4 test_mutable.Version.test_debug: use splitlines() to fix buildslaves 
Any slave running in a directory with spaces in the name was miscounting
shares, causing the test to fail.
 2011-08-27 23:47:28 -07:00
Brian Warner
7d48f902fc test_mutable.Version: exercise 'tahoe debug find-shares' on MDMF. refs #1507 
Also changes NoNetworkGrid to put shares in storage/shares/ .
 2011-08-27 17:55:42 -07:00
Brian Warner
80bdc0f5b7 test_mutable.py: oops, missed a .todo 2011-08-27 17:21:18 -07:00
Brian Warner
e0eaeff7f5 test_mutable: merge davidsarah's patch with my Version refactorings 2011-08-27 16:57:07 -07:00
david-sarah
3c92b832f2 Make the immutable/read-only constraint checking for MDMF URIs identical to that for SSK URIs. refs #393 2011-08-22 18:27:20 -07:00
david-sarah
88989a4ea2 Additional tests for MDMF URIs and for zero-length files. refs #393 2011-08-22 18:15:32 -07:00
david-sarah
ac7b8400d4 Additional tests for zero-length partial reads and updates to mutable versions. refs #393 2011-08-21 18:41:11 -07:00
Brian Warner
03ba8065ce test_mutable.Version: factor out some expensive uploads, save 25% runtime 2011-08-27 16:27:37 -07:00
Brian Warner
370e6f271e SDMF: update filenode with correct k/N after Retrieve. Fixes #1510. 
Without this, we get a regression when modifying a mutable file that was
created with more shares (larger N) than our current tahoe.cfg . The
modification attempt creates new versions of the (0,1,..,newN-1) shares, but
leaves the old versions of the (newN,..,oldN-1) shares alone (and throws a
assertion error in SDMFSlotWriteProxy.finish_publishing in the process).

The mixed versions that result (some shares with e.g. N=10, some with N=20,
such that both versions are recoverable) cause problems for the Publish code,
even before MDMF landed. Might be related to refs #1390 and refs #1042.
 2011-08-27 15:50:31 -07:00
Brian Warner
b8c90d24fc layout.py: annotate assertion to figure out 'tahoe backup' failure 2011-08-27 12:52:53 -07:00
Brian Warner
482a7dd3f1 Add 'tahoe debug dump-cap' support for MDMF, DIR2-CHK, DIR2-MDMF. refs #1507. 
This also adds tests for all those cases, and fixes an omission in uri.py
that broke parsing of DIR2-MDMF-Verifier and DIR2-CHK-Verifier.
 2011-08-27 12:50:48 -07:00
Brian Warner
522868d5e9 MDMF: more writable/writeable consistentifications 2011-08-27 12:06:02 -07:00
Brian Warner
48544a251d MDMF: s/Writable/Writeable/g, for consistency with existing SDMF code 2011-08-27 11:33:57 -07:00
Brian Warner
349093fa2e test_mutable.Update: increase timeout from 120s to 400s, slaves are failing 2011-08-25 16:01:40 -07:00
Zooko O'Whielacronx
5bf5404ba1 tests: fix check_memory test 
fixes #1503
 2011-08-25 13:11:16 -07:00
Brian Warner
9e613ee9b3 test_mutable.Update: remove .timeout overrides, otherwise tests ERROR 2011-08-24 19:24:55 -07:00
Brian Warner
445e9eaf37 blacklist.py: add read() method too, for completeness 2011-08-24 19:19:02 -07:00
david-sarah
3d7a32647c Implementation, tests and docs for blacklists. This version allows listing directories containing a blacklisted child. Inclusion of blacklist.py fixed. fixes #1425 2011-08-24 08:59:28 -07:00
david-sarah
505ebeb353 mutable/layout.py: fix unused import. refs #393 2011-08-16 15:50:43 -07:00
david-sarah
4b62b53cd6 mutable/retrieve.py: cosmetics and remove a stale comment. refs #393 2011-08-16 14:46:12 -07:00
Kevan Carstensen
62c72881bb mutable/filenode.py: don't fetch more segments than necesasry to update the file 2011-08-13 14:00:05 -07:00
Kevan Carstensen
b49986f801 test/test_mutable: test for incorrect div_ceil equations 2011-08-13 11:39:36 -07:00
Kevan Carstensen
3423b7d6f7 mutable/retrieve.py: use floor division to calculate segment boundaries, don't fetch more segments than necessary 2011-08-13 11:38:33 -07:00
Kevan Carstensen
a1a334e1a0 mdmf: clean up boolean expressions, correct typos, remove self._paused, and don't unconditionally initialize block hash trees, asll as suggested by davidsarahs' review comments 2011-08-13 11:37:10 -07:00
Brian Warner
fa6a768bc1 now that tests pass with full-size keys, return test-keys to normal (522bit) 2011-08-11 10:54:18 -07:00
Brian Warner
eec428ba5f fix SHARE_HASH_CHAIN_SIZE computation 2011-08-11 10:53:50 -07:00
david-sarah
dd23f265bf More idiomatic resolution of the conflict between ticket393-MDMF-2 and trunk. refs #393 2011-08-10 13:29:42 -07:00
david-sarah
c55c44e70a Replace the hard-coded 522-bit RSA key size used for tests with a TEST_RSA_KEY_SIZE constant defined in test/common.py (part 2). refs #393 2011-08-10 13:23:10 -07:00
david-sarah
fbe0e42bbf Replace the hard-coded 522-bit RSA key size used for tests with a TEST_RSA_KEY_SIZE constant defined in test/common.py (part 1). refs #393 2011-08-10 13:22:43 -07:00
Zooko O'Whielacronx
95a7551586 merge some minor conflicts in test code from the 393-2 branch and trunk 2011-08-10 10:21:39 -07:00
david-sarah
b60506b0ba test_web.py: fix a test failure dependent on whether simplejson.loads returns a unicode or str object. 2011-08-08 14:39:25 -07:00
Kevan Carstensen
c413a8fae1 immutable/filenode: fix pyflakes warnings 2011-08-06 17:45:14 -07:00
Kevan Carstensen
47eab03c6c test: fix assorted tests broken by MDMF changes 2011-08-06 17:44:59 -07:00
Kevan Carstensen
a7fb55f986 uri: add MDMF and MDMF directory caps, add extension hint support 2011-08-06 17:44:36 -07:00
Kevan Carstensen
52e9512b49 test/test_mutable: tests for MDMF 
These are their own patch because they cut across a lot of the changes
I've made in implementing MDMF in such a way as to make it difficult to
split them up into the other patches.
 2011-08-06 17:44:14 -07:00
Kevan Carstensen
a9cada2e03 webapi changes for MDMF 
- Learn how to create MDMF files and directories through the
      mutable-type argument.
    - Operate with the interface changes associated with MDMF and #993.
    - Learn how to do partial updates of mutable files.
 2011-08-06 17:43:48 -07:00
Kevan Carstensen
bb10d685ed mutable/servermap: Rework the servermap to work with MDMF mutable files 2011-08-06 17:42:59 -07:00
Kevan Carstensen
cde17ec041 dirnode: teach dirnode to make MDMF directories 2011-08-06 17:42:24 -07:00
david-sarah
50c692d1c5 Fix some test failures caused by #393 patch. 2011-08-01 20:28:10 -07:00
Kevan Carstensen
4d09ef3d83 cli: teach CLI how to create MDMF mutable files 
Specifically, 'tahoe mkdir' and 'tahoe put' now take a --mutable-type
argument.
 2011-08-01 19:16:13 -07:00
Kevan Carstensen
a87fc8ffab frontends/sftpd: Resolve incompatibilities between SFTP frontend and MDMF changes 2011-08-01 19:12:07 -07:00
Kevan Carstensen
b1b77d3b89 mutable/layout: Define MDMF share format, write tools for working with MDMF share format 
The changes in layout.py are mostly concerned with the MDMF share
format. In particular, we define read and write proxy objects used by
retrieval, publishing, and other code to write and read the MDMF share
format. We create equivalent proxies for SDMF objects so that these
objects can be suitably general.
 2011-08-01 19:11:20 -07:00
Kevan Carstensen
edf9858fb8 immutable/filenode: implement unified filenode interface 2011-08-01 19:09:05 -07:00
Kevan Carstensen
3e996e0c94 immutable/literal.py: Implement interface changes in literal nodes. 2011-08-01 19:08:14 -07:00
Kevan Carstensen
8ae564d8ad test/common: Alter common test code to work with MDMF. 
This mostly has to do with making the test code implement the new
unified filenode interfaces.
 2011-08-01 18:56:43 -07:00
Kevan Carstensen
f80a7fdf18 mutable: train checker and repairer to work with MDMF mutable files 2011-08-01 18:51:40 -07:00
Kevan Carstensen
893a9028f0 nodemaker: teach nodemaker about MDMF caps 2011-08-01 18:49:26 -07:00
Kevan Carstensen
d3f98498c0 client: teach client how to create and work with MDMF files 2011-08-01 18:48:11 -07:00
Kevan Carstensen
e4f2b27305 mutable/filenode: Modify mutable filenodes for use with MDMF 
In particular:
    - Break MutableFileNode and MutableFileVersion into distinct classes.
    - Implement the interface modifications made for MDMF.
    - Be aware of MDMF caps.
    - Learn how to create and work with MDMF files.
 2011-08-01 18:45:01 -07:00
Kevan Carstensen
4c48e2e1b6 nodemaker: teach nodemaker how to create MDMF mutable files 2011-08-01 18:42:58 -07:00
Kevan Carstensen
126d1ad010 interfaces: change interfaces to work with MDMF 
A lot of this work concerns #993, in that it unifies (to an extent) the
interfaces of mutable and immutable files.
 2011-08-01 18:41:19 -07:00
Kevan Carstensen
1576c35d38 mutable/publish: teach the publisher how to publish MDMF mutable files 
Like the downloader, the publisher needs some substantial changes to handle multiple segment mutable files.
 2011-08-01 18:39:31 -07:00
Kevan Carstensen
ac3b2647dd mutable/retrieve: rework the mutable downloader to handle multiple-segment files 
The downloader needs substantial reworking to handle multiple segment
mutable files, which it needs to handle for MDMF.
 2011-08-01 18:35:24 -07:00
david-sarah
c41bf14029 Cleanups for suppression of UserWarnings. refs #1435 2011-08-17 21:07:49 -07:00
Zooko O'Whielacronx
f627af3b9f suppress warning emitted by newer zope.interface with Nevow 0.10 
refs #1435
 2011-08-17 13:31:34 -07:00
david-sarah
a98a4ec13f _auto_deps.py: change the requirement for zope.interface to <= 3.6.2, >= 3.6.6. fixes #1435 2011-08-14 19:53:47 -07:00
david-sarah
a48aee11b3 allmydata/__init__.py, test_version.py: make version parsing understand '<=', with test. refs #1435 2011-08-14 20:51:53 -07:00
Brian Warner
5aad81f84d replace tabs with spaces in the #1441 'tahoe debug' synopsis 2011-08-11 10:37:04 -07:00
david-sarah
0dc917af08 Correct the information printed by '/usr/bin/tahoe debug --help' on Debian/Ubuntu. fixes #1441 2011-07-24 09:25:30 -07:00
david-sarah
6c2670641e Add test for webopen. fixes #1149 2011-07-24 14:16:59 -07:00
david-sarah
ab9eb12f70 test_client.py: relax a check in test_create_drop_uploader so that it should pass on Python 2.4.x. refs #1429 2011-08-09 22:25:04 -07:00
david-sarah
0dc56daaf9 test/common_util.py: correct fix to mkdir_nonascii. refs #1472 2011-08-09 22:19:06 -07:00
david-sarah
2deba3319c test/common_util.py: fix a typo. refs #1472 2011-08-09 21:42:35 -07:00
david-sarah
0104dbab92 test_client.py, test_drop_upload.py: fix pyflakes warnings. 2011-08-09 20:45:05 -07:00
david-sarah
db22fdc20d Factor out methods dealing with non-ASCII directories and filenames from test_drop_upload.py into common_util.py. refs #1429, #1472 2011-08-09 20:15:58 -07:00
david-sarah
c102056ac1 test_client.py: add a test that the drop-uploader is initialized correctly by client.py. Also give the DropUploader service a name, which is necessary for the test. refs #1429 2011-08-09 20:05:38 -07:00
david-sarah
10ee22f50e drop-upload: rename 'start' method to 'startService', which is what you're supposed to use to start a Service. refs #1429 2011-08-09 20:03:45 -07:00
david-sarah
f157b73367 test_drop_upload.py: add comment explaining why we don't use FilePath.setContent. refs #1429 2011-08-09 19:59:42 -07:00
david-sarah
369e30b1df test_drop_upload.py: fix some grammatical and spelling nits. refs #1429 2011-08-09 15:12:31 -07:00
Zooko O'Whielacronx
612abca271 drop-upload: report the configured local directory being absent differently from it being a file 
refs #1429
 2011-08-09 15:09:30 -07:00
Zooko O'Whielacronx
b7683d9b83 drop-upload: rename the 'upload.uri' parameter to 'upload.dircap', and a couple of cleanups to error messages. refs #1429 
I rerecorded this patch, originally by David-Sarah, to use "darcs replace" instead of editing to do the renames. This uncovered one missed rename in Client.init_drop_uploader. (Which also means that code isn't exercised by the current unit tests.)
refs #1429
 2011-08-09 15:05:08 -07:00
Zooko O'Whielacronx
5633375d26 drop-upload test for non-existent local dir separately from test for non-directory local dir 
A candidate patch for #1429 has a bug when it is using FilePath.is_dir() to detect whether the configured local dir exists and is a directory. FilePath.is_dir() raises exception, instead of returning False, if the thing doesn't exist. This test is to make sure that DropUploader.__init__ raise different exceptions for those two cases.
refs #1429
 2011-08-09 15:01:15 -07:00
Zooko O'Whielacronx
720bc2433b drop-upload: unit tests for the configuration options being named "cap" instead of "uri" 
This is a subset of a patch that David-Sarah attached to #1429. This is just the unit-tests part of that patch, and uses darcs record instead of hunks to change the names.
refs #1429
 2011-08-09 14:59:13 -07:00
david-sarah
c2972e22cb src/allmydata/storage/server.py: use the filesystem of storage/shares/, rather than storage/, to calculate remaining space. fixes #1384 2011-07-18 19:27:52 -07:00
david-sarah
4c592f1505 test_storage.py: test that we are using the filesystem of storage/shares/, rather than storage/, to calculate remaining space, and that the HTML status output reflects the values returned by fileutil.get_disk_stats. This version works with older versions of the mock library. refs #1384 2011-08-09 12:07:22 -07:00